Full graph of Operation Stealth’s executePlayerInput-function in IDA (2916×2994 image i.e. LARGE!)

I tried a semi-quick hack to put it into ScummVM and it did seem to help with my previous problem of borking the player input handling with my incomplete fix in r33872 (See a forum post about the problem or my previous blog post for more info). Feels good getting things done Now just to clean it up a bit and see that I didn’t break anything else with it… and yes, also committing it

Status update

The problem I was having with player movement last time really was just that, the player movement routine (executePlayerInput) was incomplete. Once I reverse engineered Operation Stealth’s version of it and modified ScummVM’s current implementation (r33872) of that function a bit I got the swimming working and I could get to the girl and free her:

Freeing the girl in ScummVM

I might as well tell a bit about what happens in the game after that:

The player character and the girl swim to the surface and are rescued by a boat:

Being rescued by boat

The story goes on and they meet The Movement for The Liberation of Santa Paragua for the first time:

Meeting the movement

Then they leave to infiltrate a palace:

Leaving for the palace

And enroute to the palace some masking bugs

Masking bugging while enroute to the palace

And then after a cutscene at the palace there’s another arcade sequence: A labyrinth.

The labyrinth

Okay, so far I’ve gotten to the second arcade sequence which is a labyrinth:

First labyrinth

But there are some showstopping problems here. The player can be controlled but the doors that should revolve when walked against don’t revolve and so the arcade sequence is not yet completable.

I played Operation Stealth using DOSBox to the labyrinth and used DOSBox’s debugger to look for the collision data page (And I found it! Yay!) and dumped it to a binary file. What I wanted to do was to compare the original’s collision data page to the ScummVM’s version in the labyrinth to see if there were any differences that could be breaking the gameplay. And voila, there were! Here are some pictures:

First labyrinth's collision data from DOSBox

First labyrinth's collision data from ScummVM

By the way the collision data page can currently be viewed in the Cine engine by pressing F11 (I added that in r33877).

I tried loading the collision data I dumped from DOSBox to Cine’s collision data page and yes, I got the doors to revolve! But alas, they didn’t yet work totally correctly, sometimes you could just simply walk through a door like it was thin air and sometimes a door would shake between two frames of animation a bit weirdly.

So it’s looking like at least there’s something wrong with how Cine currently manages the collision data or how it’s loaded. I checked the collision data loading routine loadCtOS and the data conversion routine gfxConvertSpriteToRaw that it uses. It looks like the gfxConvertSpriteToRaw is indeed correct, I reverse engineered a version of it myself from Operation Stealth’s disassembly, tried using my own version of it but it didn’t seem to make a difference at all. So that particular routine doesn’t look to be the culprit here.

Release testing related bugfixes

With the next stable version of ScummVM being prepared and release testing in progress I got some Future Wars related bug reports and did some bugfixing:

• Adlib sound related bugfix to fix a crash when teleporting to the swamp (r33935 trunk, r33937 0.12.0)
• Fix for overparanoid monks beating player unconscious (r33950 trunk, r33951 0.12.0)
• Fix for crashing when examing rubble above a manhole (r33967 trunk, r33966 0.12.0)

And with those fixes in place Future Wars’s English DOS floppy version should now be completable using the 0.12.0 branch.

Reported Future Wars bugs still unfixed

There are some regressions and bugs still left in the Cine engine though:

• Bug #2057656: FW: Assert during demo (regression)
• Bug #2057637: FW: Spaceship glitches in demo (regression)
• Bug #2057619: FW: Glitches in title display of demo (regression)
• Bug #2055912: FW: incrustSprite fails on savegame loading
• Bug #2055836: FW: Music is not restarted when loading a saved game
• Bug #2019344: FW: crash with Amiga Italian version (photocopy room)

What I learned during this summer

Ok, enough about the status update, here are some points I’d like to make about what I learned during this summer of code:

• A lot about reverse engineering – I feel a lot more competent in it now than I did before this summer – and in particular that IDA’s graph view is simply essential, I wouldn’t dream about going back to not using it when reverse engineering something. It’s just absolutely essential, no doubt about that.
• Tried Valgrind for the first time and it looks like a good tool to have around when fixing memory corruption problems. I recommend giving it a go if you have memory corruption problems in a software you’re developing!
• Tried DOSBox’s debugger for the first time and it looks also to be a useful piece of software, although I probably don’t know half of what can be done with it yet.
• That I’m not always as productive as I might like to, my effort changes with the tasks I’m doing. So knowing how to focus and get things done even though the task might not be to my current liking is something I haven’t yet truly mastered. I’ve heard that if you are completely present in a situation and totally focused on something then that something *becomes* interesting, whatever it may be. Now that’d be a lifesaving thing to know how to do – maybe some meditation (e.g. Zazen) would be in order?
• That I probably take my work too seriously at times: Getting something working felt great at times and not getting something working felt awful at other times. Or maybe it’s simply human nature to feel like that? Dunno. Anyway, I should probably know when to relax and let go, simply do and not measure my worth by what I get done but by the effort I put in regardless of the outcome (Yeah, quite the Gandhi talking ).

What’s so good about this reverse engineering thing anyway?

All in all this summer was a learning experience for me. What I found particularly interesting was the reverse engineering bit, taking a function in disassembly, looking at how big and intractable it looks and then starting on it, bit by bit, part by part making it more and more understandable until what you have in front of you is something you actually are able to understand. An example:

This is the graph of executePlayerInput function without any work done on it yet.

Here's how the function looks after quite a bit of abstracting done on it.

And here's the highest abstraction level graph that I made of the function.

It’s quite a bit of work, I tell you, but it’s quite riveting (There’s that focus thing again. I mean, just take a look at what Merriam-Webster says about the word riveting: “having the power to fix the attention”). Yeah!

What about the future?

I’m still committed to making Operation Stealth completable although it’ll be outside this GSoC. I’ll be fixing some of the Cine engine related bugs for the upcoming 0.12.0 release of ScummVM too.

Closing words

I’d like to congratulate everyone who made it to the end of Google Summer of Code 2008! It was an interesting ride, glad so many people with such diverse projects could get along for it. Thanks Google!

And last but not least thanks to my mentor Eugene for guiding me to a more productive direction in my work at times during this summer. His sane voice was the voice of practicality and, I imagine, experience.

So I recommend trying out Valgrind if you’ve got memory corruption problems with a program you’re developing even if you don’t run Linux natively, it can be used through a virtual machine like VirtualBox, QEMU or VMWare. I already got some memory corruption problems fixed with Valgrind’s help.

Stuff done since last update:

Bugfixes:

• Fixed popup boxes blocking animation (r33686)
• Partially fixed Adlib volume setting (r33700 trunk, r33701 0.12.0)
• Fixed ‘text hard to read’ problems (r33792)
• Fixed inventory showing (r33795)
• Fixed verb handling bug (r33805)

New functionality:

• Implemented game speed changing by pressing – or + (r33689)
• Added moving using keyboard (r33698 & r33710)
• Basic support for CD Future Wars (r33703 trunk, r33705 0.12.0)
• Added drawing of type 22 overlays as filled rectangles (r33713)
• Preliminary drawing of type 21 overlays (r33721 & r33722). WIP!
• Implemented transparent cutscene textboxes (r33790)

Cutscene textboxes working in ScummVM

Memory corruption and out of bounds access related changes:

• Fixed rectangle drawing (r33687 & r33811 trunk, r33814 0.12.0)
• Added sanity tests to background scrolling (r33688)
• Converted objectTable to Common::Array (r33725)
• Converted animDataTable to Common::Array (r33726)
• Converted zoneData & zoneQuery to Common::Array (r33727)
• Converted partBuffer to Common::Array (r33783)
• Converted palPtr to Common::Array (r33785)
• Fixed fontParamTable freeing (r33786)
• Converted commandBuffer to Common::String (r33793)
• Fixed handling of incorrect input data in loadMsg (r33810)
• Fixed initialization of instrument data (r33812 trunk, r33813 0.12.0)

Cleanup:

• Removed unnecessary textDataPtr (r33784)

And as Valgrind noticed some memory problems in the AGI engine’s sound engine I committed some fixes related to them (r33816 & r33822 trunk, r33820 & r33823 0.12.0). I had previously noticed that the AGI engine’s sound engine sometimes crashed when starting the first sound in an AGI game, so hopefully that’s fixed now.

What next?

Well, I got to the part where the player character is dropped into the sea with a woman, but I’m currently stuck there. After sorting out the memory corruption problems and testing the original game’s behaviour in this particular part it seems the problem is somehow related to incorrect character moving. The character should first swim near the girl and then try “Operate Girl” to untie her ropes (It works that way in the original). Here’s a picture of the thing working in DOSBox:

Freeing the girl in DOSBox

But alas in ScummVM currently the player character can only be moved to the left and he automatically moves upwards almost all the time. So I can’t get to the girl to untie her and they both drown. With the memory corruption problems out of the way I’m looking into fixing this character moving problem next.

• First arcade sequence was practically impossible to complete (Oxygen forced to maximum by a hack in r33667 for now so it’s possible to complete it)
• Player teleported to a wrong location when getting out of the water after the first arcade sequence (Fixed by fixing checkCollision function’s boundary checking in r33668)
• Player got completely stuck after getting out of the water after the first arcade sequence (Introduced a hack in r33673 to nudge the player off his position by one pixel to get him free)

Here's where one could get stuck after the first arcade sequence

So now the first arcade sequence is completable (Although by a hack) and one can get out of the water after it and continue playing.

BTW while playing I’ve noticed that although savegames work good enough to allow continuing playing the graphics aren’t always correct, see for yourself:

Beach scene bugging after loading a savegame

What next?

Carrying onwards with the game! Let’s see how far I’ll get this time without encountering any showstopping bugs…

Bugs found on first completion run try so far

#1: Verb-handling bug:

If you open the briefcase with “Operate briefcase” the operate-verb will stay chosen and then you can choose “Operate calculator” again and again etc and it becomes “Operate calculator calculator calculator…” and finally crashes.

Verb handling bugs

#2: Background handling bug:

If you load a savegame that has the briefcase open the palette and the background will be wrong (If you opened the briefcase in the bathroom stall the background will be the bathroom stall but with wrong colors. Pen, passport and calculator in the briefcase are shown on top with correct colors though.

Briefcase's background bugs after loading

Briefcase's background works after not loading

#3: Airport bag conveyor belt masking bug:

If you go to the airport’s bag picking up scene where bags run in a circle on a conveyor belt then you’ll see that the scene looks pretty weird. Some masks are probably off… (Things go in front of stuff that they shouldn’t or something similar).

Conveyor belt masking bugs

Conveyor belt masking working in DOSBox

#4: Fading bug:

If for an example you go to the airport’s bathroom stall and open and close the briefcase you’ll get a double fading effect.

#5: Inventory showing doesn’t work unless using the USE verb:

If you right click to get the menu and choose INVENTORY nothing happens. It should show the inventory. If you choose the USE verb you’ll get to choose an item to use from the inventory though.

#6: Game crashes or player is invisible when exiting the airport:

First the game crashed when trying to leave the airport when it didn’t find some files JOHN01.ANI and JOHN02.ANI. Fixed the resource loading to handle missing files gracefully in r33643. After that the player character was completely invisible when walking behind the glass in the next scene. Looked into that and made a workaround by loading the corresponding SET files instead (It worked and looked fine) in r33644.

Player visible behind glass after workaround

#7: Music may hang on a note when scene changes:

If for an example you leave the airport by a taxi and go downtown the music playing outside the airport may hang on a note and continue playing that single sound. The sound is quite annoying.

#8: Text hard to read:

For an example in the Santa Paragua’s bank when examining the stained glass door on the back the background of the popup box is light yellow and the text is white so it’s very hard to read what it says.

Text hard to read

#9: Text box wraps around the screen:

A “WATCH OOOOOUT!!!” text box wraps around the screen from the right to the left side in the Las Mimosas park during the drive-by shooting.

Text box wraps around screen

#10: Masking bugs at the hotel’s beach stairs:

This might very well be just like the next bug but when walking to the stairs that go down to the hotel’s beach in Santa Paragua the masking bugs and the player character can seemingly walk through the railing.

Masking bugs at the hotel's beach stairs

#11: Masking bugs in the bank vault in Santa Paragua’s bank:

When going to open the safe case in the bank’s vault in Santa Paragua the player character is seen in front of the bars although he’s walking behind them. The masking works later though… weird.

Vault bar masking bugs

Vault bar masking works

#12: Popup box blocks animation when it shouldn’t:

After getting caught by the enemy agents in Santa Paragua’s bank there’s a scene where it reads “Later, after a hard trip…” and the enemy agents and the protagonist walk from left to right on-screen. They don’t walk at all if one doesn’t click with the mouse (They should be animated even though there’s a popup box on-screen). So you can only get through the scene by clicking with the mouse many, many times but still, it’s possible.

Popup box blocking animation

#13: Game jams after failing first arcade sequence:

If you die by being underwater for too long in the first arcade sequence the game jams. Nothing happens, music just keeps playing and the “It isn’t recommended to hang around underwater. This is the end of your exploits” box is shown and the loading menu can’t be accessed.

#14: Red bar that shows the oxygen left isn’t updated:

In the first arcade sequence the bar should show the amount of oxygen left but it doesn’t, it simply stays the same long red bar all the time (Still you can get killed by being underwater for too long).

First arcade sequence

#15: It’s very hard to move in the first arcade sequence:

The player character doesn’t move much if you just click once somewhere on-screen and keep the mouse button down. Neither does using the keyboard do any good (Movement using keyboard isn’t implemented yet, I think). So you just have to click furiously with the mouse to where you want to swim in order to get there. This may make the first arcade sequence practically impossible to complete.

What next?

Alright, I got to the first arcade sequence but it’s hard to play it. I’ll fix that so it can be played through and then carry on!

• Object selection bug in Operation Stealth (Fixed in r33620 in trunk)
• Text prompt updating regression (Fixed in r33637 in trunk, backported to branch-0-12-0 in r33638)

So, savegames and the interface in Operation Stealth are working good enough now to start the game completion run. So, Operation Stealth, here I come! Wish me luck

The problem was that the animation data table entries were loaded using a different and less complete code when loading a savegame than when running the game and loading the entries using script opcodes like loadAbs, loadAnim etc. So what I did was I made it so that the savegame loading routine uses the exact same functions that are used by the resource loading related script opcodes (The changes I made are r33529 and r33530).

Yay! Next up: Fixing the mouse cursor pointing problem in Operation Stealth (If you point at an object on-screen it’s recognized a bit to the right of its correct location. I think the problem most likely is in getObjectUnderCursor and/or executePlayerInput functions).

Still, I have been doing some testing with the bug, not very much, but some. Now I know what data seems to not be up to date after loading. The next step is getting systematic about it. Identify the functions that manipulate the data that’s making things break currently, understand why and where they are called and see where they should be called when loading a savegame to get the needed data up to date.

Currently Operation Stealth throws an assertion in addAni function which is called from processSeqListElement which is called from processSeqList which is in turn called from the main loop (Now *that* was a nice elongated sentence if any ). I did some debugging to see what values addAni is called with when playing the game and the only values I saw in the beginning of game were values related to the player character’s animation:

Calls to addAni when walking around in the first room

• param1=3, objIdx=1, var8=1, var14=0, param3=0 (Walking down)
• param1=2, objIdx=1, var8=1, var14=0, param3=0 (Walking up)
• param1=0, objIdx=1, var8=1, var14=1, param3=1 (Walking right)
• param1=1, objIdx=1, var8=1, var14=2, param3=1 (Walking left)
• param1=7, objIdx=1, var8=1, var14=0, param3=1 (Facing down)
• param1=6, objIdx=1, var8=1, var14=0, param3=1 (Facing up)
• param1=4, objIdx=1, var8=1, var14=0, param3=1 (Facing right)
• param1=5, objIdx=1, var8=1, var14=0, param3=1 (Facing left)

Some clarifications:

• E.g. facing left means that the player character is standing still and looking left.
• var8 and var14 are actually member variables of a SeqListElement that’s given as a reference parameter to addAni.

Hypotheses made from the given addAni calls:

• param3 is 0 when the player character is walking vertically, otherwise it’s 1.
• element.var14 is 1 when the player character is walking right, 2 when his walking left and otherwise 0.
• param1 identifies the used animation (Walking right, walking left, standing still and facing left etc).

So, wish me luck with the debugging and let’s hope it’ll be over in no time!

So to get this bug smashed next… ROAR!

My idea why I wanted to reverse engineer and use the information from the Operation Stealth’s disassembly for the savegame handling is that if I know what the original engine thought was important enough to be saved to a savegame file I can use that information to save pertinent parts of the Cine engine too. So what was good enough for the original Operation Stealth engine should be good enough for the Cine engine as well.

A word of warning about the savegames: The savegames for Operation Stealth are going to be very much a work in progress and therefore backward compatibility can be broken at practically any time and the changing formats will not be supported later. They are only a tool for development. A bit later it’d be very nice to share resources with the SCUMM & Tinsel engines so the three engines could use the same code for handling their savegames.