Savegame format changing ordeal

2008-07-25 (Friday)

HI! Tried changing the Cine engine’s savegame format from plain data without any markers to a chunked format akin to IFF. Got lots of ideas, and some code together too, but eventually gave up as making it really work would’ve required some major objectifying of the engine and it’s still very C style in many places currently.

So I’m not going to make use of a finegrained totally new chunked format now, but I’ll make a compromise and use a new chunked format but dump most of the savegame data into a single chunk. The format can be improved later.

I looked into how the SCUMM engine has implemented its savegame handling and its approach looks to be quite an interesting one. Take a look at scumm/saveload.h and scumm/saveload.cpp.

P.S. My IRC attendance on #scummvm will suffer in the near future as the server on which I ran irssi in a screen moves town starting today. I’ll try to get back on IRC during Sunday or Monday the latest.

Back to working with Operation Stealth

2008-07-22 (Tuesday)

Hi, just got the release critical Future Wars old savegame loading bug fixed (My thanks to wjp on #scummvm for his help with nailing this bug). Getting the Operation Stealth’s savegame stuff working is now next…

Bughunting season

2008-07-21 (Monday)

Hi! Last week went mostly in bughunting as the next ScummVM release is being prepared and bugs are being fixed.

  • Fixed a bug that made Amiga and Atari ST versions of Future Wars crash right in the beginning. Seems like the script writer for those Future Wars versions made a typo (He tested local variable 251 in the scripts when he meant to test *global* variable 251). So it was a bug in the original game’s scripts.
  • Worked on reducing the Cine engine’s BSS size (i.e. reducing the amount of global variables that are allocated statically and increase the size of the executable. Here’s a patch for it). Got a nice 64kB reduction in the BSS size.
  • Tried to fix a release critical bug that makes the current ScummVM code not be able to load some Future Wars savegames that were made with 0.11.0 or 0.11.1 releases of ScummVM. Haven’t yet been able to fix it completely, but I’ve got an idea what’s making it bug at the moment (Thanks to wjp for his help with this bug).

So now I’ll try to get that release critical savegame loading bug fixed first and then it’s onto fixing Operation Stealth’s savegame support (Which doesn’t look *so* intimidating anymore after all this looking into Cine engine’s savegame format).

Making Operation Stealth completable

2008-07-13 (Sunday)

Hi! With the mid-term goals done and the bathroom door crash fixed I’m now setting out to make the PC version of Operation Stealth completable. My plan is to try to play through the PC version of Operation Stealth and fix any showstopper bugs along the way (i.e. bugs that keep me from completing the game). In addition I’ll also try to fix any seriously annoying bugs like e.g. the current mouse cursor pointing problem (You can’t just point to an object onscreen to access it but you have to point a bit to the right of it).

After I get the PC version completable it’s time for all the non-crucial stuff like fixing the music, sound effects, graphic glitches etc. And then there’s also the matter of making sure that the Amiga and Atari ST versions are completable too. So all in all a lot to do! But let us not lose hope but push ever forwards ;-).

Ah, and as the saving and restoring of games in Operation Stealth is broken at the moment, that’s the first thing to be fixed before the completion journey begins.

P.S. I’ll be making blog posts along the way as I get further towards completing the game.

Mid-term goals done

2008-07-4 (Friday)

Hi! The last of my mid-term goals (The actor animation regression) got nailed this week.

Here are my recent achievements:

  • Implemented resetGfxEntityEntry function in r32848
  • Fixed addAni function in r32850
  • Fixed and implemented the rest of processSeqListElement function in r32874

And I’ve been documenting the Operation Stealth’s disassembly generally along the way a bit too.

What next?

I’ll probably try to nail the sprite related problem when trying to enter the bathroom in the airport (In the very first scene of Operation Stealth, that is). There’s a bad memory allocation happening when one walks to the bathroom door and the door should open. It happens because of trying to allocate some humongously – and very erronously so – large array.

Also saving games in Operation Stealth is broken at the moment, so that’s one quite integral thing that could use some fixing too.

Got into reverse engineering mode

2008-06-27 (Friday)

Hi! Got the Operation Stealth versions’ opcode comparison done after all (It took an awful lot of time with all the procrastination and indecision thrown in and it wasn’t very inspiring work) and now I’ve been mostly reverse engineering Operation Stealth’s missing opcodes or fixing existing ones’ implementations. Feels a lot better now :-). Reverse engineering is fun!

IDA‘s graph view has proven extremely useful. It helps a lot in understanding what a function does and how it does it by giving a flow chart view of the function. It wouldn’t be easy to go back to using IDA without it, I’ll say! The graph view is one good reason for getting a license for IDA Pro as the freeware version 4.9 of IDA doesn’t have it.

Mid-term goals

Talked with Sev and he urged me onwards as I was still fiddling with the opcode comparison on Monday evening. So we set a new goal for the mid-term: All unimplemented opcodes that are used in Operation Stealth’s intro have to be implemented by mid-term (And fix an animation related regression). Here’s the goal in parts:

Implement unimplemented opcodes used in intro:

  • 40h: Undefined opcode (Was o1_closePart in Future Wars)
  • A1h: o2_removeGfxElementA0
  • A2h: o2_opA2
  • A3h: o2_opA3

And fix the animation regression in Operation Stealth’s very first room (The actor just slides around unanimated).

Recent achievements

So here’s what I’ve got done recently:

Fixed already existing opcodes:

  • 08h: o1_checkCollision in r32790
  • 83h: o2_isSeqRunning in r32804
  • 9Ah: o2_wasZoneChecked in r32790
  • A0h: o2_addGfxElementType20 (Renamed opcode) in r32769

Implemented missing opcodes:

  • 82h: o2_modifySeqListElement (Renamed opcode) in r32786
  • 8Dh: o2_op8D (No good name yet) in r32785
  • A1h: o2_removeGfxElementType20 (Renamed opcode) in r32769
  • A2h: o2_addGfxElementType21 (Renamed opcode) in r32769
  • A3h: o2_removeGfxElementType21 (Renamed opcode) in r32769

Added Operation Stealth support for:

  • zoneQuery in r32790 (Only used in Operation Stealth)
  • addOverlay in r32816 (Was previously Future Wars specific)

What’s still left of the mid-term goals:

As the opcode 40h turned out to be a no-operation in Operation Stealth’s PC version (It was o1_closePart in Future Wars) it only leaves me an actor animation regression to fix (Previously actor animation worked in the very first room in Operation Stealth, but now it bugs, the actor isn’t animated, he just slides around like on wheels). So that’s to go and then I’ve got my mid-term goals done.

Delphine compression format deciphering

2008-06-16 (Monday)

Spent most of my work time last week deciphering Delphine’s compression format. Now I feel satisfied by having made the implementation in ScummVM’s Cine-engine quite robust and hopefully understandable too (I documented it and really made myself understand what the darn thing does).

The decompression code had been initially reverse engineered from disassembly by someone else and there were no comments to speak of so that’s why it wasn’t very understandable to me in the beginning. Just compare the older version that I started with (Header file & code file) with the version that’s now in place (Header file & code file).

I also wrote documentation for the compression format to ScummVM’s wiki (Look there if you want to intimately know how the compression format works).

So in the end it turned out the compression algorithm used by Delphine’s adventure games uses sliding window compression (Quite like LZ77) combined with a fixed non-adaptive entropy coding scheme (Not of any type I could recognize).

Now to use the decompressor and unpack all those scripts and do the comparison…

P.S. Oh, and I also tried IDA’s graph view for the first time and worked with it on Delphine’s unpacking routine’s disassembly. Here’s an image of the abstracted graph of the unpacking function.

Some procrastination and preparation for opcode comparison

2008-06-9 (Monday)

Hello everyone. There’s been some procrastination in the air so to speak. I’ve been working on making an opcode comparison between different Operation Stealth versions in order to discover which opcodes are platform specific (Amiga/PC/ST), which are extensively used etc. The idea is that once I know that then I’ll know which opcodes to try to implement first. Sev pointed out to me a standalone resource file unpacker and script decompiler for Cinématique.

My first problem was what files to decompress i.e. knowing which files are resource files. Well, I looked at ‘vol.1’ – ‘vol.9’ files. They seemed to be simply text files containing lists of at least some of the resource files (Maybe all of them, don’t know). Then I got interested in how Cine currently figures out which resources files to unpack and it seemed it doesn’t use the ‘vol.?’ files at all but instead it uses a ‘vol.cnf’ file. Well, I tried to figure out the code that deciphers the ‘vol.cnf’ file and now I understand it. I started a page about Cinématique‘s internals, file formats etc in ScummVM’s wiki at the Cine Specifications page and wrote some documentation about the ‘vol.cnf’ file format there.

The standalone resource file unpacker and script decompiler didn’t have any support for wildcards or directory recursion so I wanted to try to use Python to do the directory tree walking and call the external unpacker or script decompiler when necessary. Hadn’t done directory tree walking or cleanly done command line options parsing before in Python so it took some time to learn to use those. Eventually after some tryings I used getopt.getopt for command line parsing and os.walk for directory tree walking.

I also was somehow torn with what to do in Python and what to do in C. The resource file unpacker and script decompiler were in C. If I were to call them from Python what about error handling? I couldn’t find a nice way to pass info back to the Python script from the external programs – I tried popen3 a bit but maybe the stdin/stdout/stderr passing only works on Unix, dunno, or maybe I just didn’t try hard enough :-).

Eventually after some procrastination and different ways of trying things I’m now quite convinced it’s probably a good compromise and use of resources to let low-level stuff be done in the C code (Bit manipulation, unpacking etc) and the higher level stuff (Calculating scripts’ opcode statistics etc) be done in Python because that way I’m using both of the languages’ strengths (i.e. use them for what they’re best for) instead of trying to do most or all of the stuff in one language only.

So now I’m at the point that I’ll just combine the Python and C programs to do the unpacking of all the resource files for the different game versions and then modify the script decompiler to output in .CSV format so that it’s easy to read the C program’s output in with Python or Open Office. Once I have the data parsed I can do the opcode statistics calculations. There’s a question though about how to detect what files are script files and what are not so I know which files to call the script decompiler on. Well, we shall see…

BTW sometimes understanding C code that deals with pointers and C library calls can be a bit of a PITA (And no, I don’t mean the eatable kind). Just compare fixVolCnfName()’s old C code version and the more C++ style version I did of it with some comments. Now I can understand what it does :-).

First steps: French language and dead code

2008-05-29 (Thursday)

Got the disassembly of the French version of Operation Stealth (There was already some work done on it by another person so I’m not starting totally from scratch here) and also the disassembly of the French version of Future Wars (Les Voyageurs du Temps) so I’ve got another version of the Cinématique engine to compare functions with. Luckily enough I know at least a little bit of French (I studied French as a fourth language in elementary school and gymnasium) – although I’ve forgotten most of it ;-). Learned that échec in French means failure btw.

Found where in the Operation Stealth’s disassembly the opcode list was and named the functions according to the current namings in the ScummVM’s cinE-engine. Also commented a couple of small functions in the disassembly (readScriptU8 and readScriptU16) and found out that Turbo C wasn’t so hot on dead code elimination back in the day. Look at this code from readScriptU16:

seg002:0035 mov al, es:[bx]
seg002:0038 mov ah, 0
seg002:003A mov si, ax
seg002:003C mov ax, si
seg002:003E mov cl, 8
seg002:0040 shl ax, cl
seg002:0042 mov si, ax

Now isn’t that beautiful?-). I mean, bouncing stuff around without any meaning. They could’ve just done:

mov ah, es:[bx]
xor al, al
mov si, ax

7 instructions boiled down to 3. That’s quite a reduction I’d say. But well, don’t know what compiler optimization settings they used for compiling Operation Stealth and to be fair computer horsepower back when Turbo C was hot stuff was mildly different than today :-D.

And so it begins!

2008-05-26 (Monday)

Hi! It’s time for GSoC to start! Just installed my licensed copy of IDA Pro and getting ready for some reading up on the disassembly and on the cinE-engine characteristics. Let the games begin!