Developer Documentation for the Gob Engine

[g99k]

Hi, everyone. Does anyone know if there is some developer documentation available for the Gob engine or the Gob data files?

Somewhere in the forums it said that it's the smallest engine in ScummVM, and that it was originally researched by Ivan Dubrov, but I haven't found more information.

I looked at the source code and the parse*.cpps, but I found the code is rather sparsely commented. If there's any other documentation about the engine or the data format, please tell me. Since some people already spent so much time one the engine, I thought it'd be possible that they had shared their findings somehow.

There's no special task that I need the documentation for. I just find the games intriguing yet they appear to be relatively simple, so I'd like to know how the inner game logic works.

(On a sidenote: for some reason, I wasn't allowed to access the Doxygen web pages. Are they under construction?)

Thanks in advance!

[DrMcCoy]

Hi, everyone. Does anyone know if there is some developer documentation available for the Gob engine or the Gob data files?

Apart from the source in ScummVM, not much, I'm afraid.
For the video files, there are entries in the MultimediaWiki for IMDs (apparently based on our code) and VMDs.

If you want to write some documentation based on the code yourself, you'd be welcome, of course.
Once you know where to start, the engine shouldn't be that hard to understand.

Somewhere in the forums it said that it's the smallest engine in ScummVM

From the number of lines, directory size, etc., that's not true (anymore?).

I looked at the source code and the parse*.cpps, but I found the code is rather sparsely commented.

Well, it was constructed by reverse-engineering, after all.
Ivan apparently didn't comment much and I'm unfortunately not quite the chatty type either . I often refrain from commenting because I think that piece of code is rather obvious (although it might not be for an outsider).

Yes, the parse*-files are quite hard to follow, but they're not needed for understanding the basic file formats. The functions there parse expression found in the script byte code (like "(VAR_23+6)/2").

Take a look at dataio.cpp, for example. It's for handling STK/ITK files, which are archive files bundling all the other data files. (Or take a look at a small STK extractor I've wrote for testing. It should be clearly readable.)

If there's any other documentation about the engine or the data format, please tell me.

As I said, not really, sorry.

Since some people already spent so much time one the engine, I thought it'd be possible that they had shared their findings somehow.

Well, I for one can give you pointers on where to start looking, answer specific questions, etc..

There's no special task that I need the documentation for. I just find the games intriguing yet they appear to be relatively simple, so I'd like to know how the inner game logic works.

As for the gob games, most of the game logic is in the scripts.

[g99k]

As for the gob games, most of the game logic is in the scripts.

The SCRIPTS!!! That's the word I was looking for! Yes, I'm after the scripts!

My goal is to be able to read and understand several sample scripts. I don't care too much for the bytecode format itself, I just want to understand the logic.

I often refrain from commenting because I think that piece of code is rather obvious (although it might not be for an outsider).

Of course! I forgot that code speaks for itself. Here's what I got so far from reading your code:

Many classes are subclassed accourding to their goblin game version number (vXX). The global GobEngine variable _vm seems to have two main member variables to handle the different games:

• _inter of class Inter_vXX handles the game logic,
• _game of class Game_vXX holds the game data.

The Game_vXX::checkCollisions method checks the mouse click collisions with the help of the Game_vXX::checkMousePoint method. Each collision block has a pointer associated with it which points to a function in the script data blocks.

The actual parsing then seems to take place in Inter::funcBlock. The pointer is evaluated and 2 command opcodes plus parameters are read from the script data and passed for execution to Inter_vXX::executeFuncOpcode. Here, the 2 opcodes are combined to a single number which is used as an index to look up a function pointer in the _opcodesFuncVXX array, which is then executed with the parameters.

Code: Select all

OpcodeFuncProcV1 op = _opcodesFuncVXX[i*16 + j].proc;
...
return (this->*op) (params); 

One of the more special functions is probably oXX_goblinFunc, since it just reads yet another (goblin) opcode to a (goblin) function with (goblin) parameters and executes it. This goblin opcode is passed for execution to Inter_vXX::executeGoblinOpcode. Here, the goblin opcode is first mapped via the _goblinFuncLookUp array to another, more consecutively numbered list, then a goblin function pointer is retrieved from the _opcodesGoblinVXX array and executed with the goblin parameters. There are also level 2 drawing functions which work much like the goblin functions.

(I'm from Javaland, and I must say, C++ function pointers totally rule when it comes to producing obscure code. )


Okay, here's my questions:

1. Can you maybe explain the mappings and different levels of opcode? Are there any rules for which function resides in which array?
2. How do control flow statements (like "o1_if") work, and where are the tested variables stored?
3. How do I enable script dumping? The command line argument didn't seem to work for me, even though I created a "dumps" folder.
4. What does "tot" stand for? Is that a name for the script data?

Thanks for your help and for sharing your insights!

[DrMcCoy]

The SCRIPTS!!! That's the word I was looking for! Yes, I'm after the scripts!

I'd advice you to not take the GobEngine scripts. They are pretty...confusing/complex.

Of course! I forgot that code speaks for itself.

Exactly.

Each collision block has a pointer associated with it which points to a function in the script data blocks.

2 actually. One for entering and one for leaving.
Engine versions 2 and above have an additional third one that's used for variable coordinates.

Can you maybe explain the mappings and different levels of opcode?

Well: the original exectuable did it like that!
Or rather, the orignal executable had huge switches and I changed it to use a table of function pointers instead (after what our SCUMM-engine does).

In the disassembly, funcBlock first checks through 0-4 of the first value and for each case, switches again.
o1_drawOperations just takes one byte and switches over it.
o1_goblinFunc takes two bytes and doesn't check for completely consecutive values, therefore I first run it through a translation table, so I won't need a function pointer table with 2006 entries.

Are there any rules for which function resides in which array?

You'd have to ask that the original devs.
Apparently, o1_drawOperations has many drawing related functions and in Gob1, goblinFunc had to do with goblins related function, that's why Ivan named then that way.

Although in later engine versions, goblinFunc doesn't do anything with the goblins anymore. The functions even change with games that otherwise use exactly the same engine version (Gob2, Ween and Bargon, for example).

How do control flow statements (like "o1_if") work, and where are the tested variables stored?

They parse and evaluate an expression (with the functions in the Parse_*-classes).
The variables are stored in Global::_inter_variables. And here's the tricky/ugly part:
Global::_inter_variables gets initialized as a consecutive block of memory for n (read from the game data) 32bit variables. But the scripts can access each variable as either:

• 1 32bit variable
• 2 16bit variables
• 4 8bit variables

or even:

• Arrays of above
• Strings

which ignore the variable borders altogether.
Which variable is of which type is completely managed by the scripts, the engine doesn't know about that (it only gets ordered to, for example, write these 8bit to position x in the variables memory).
The animation properties of each object are mapped onto that, too.

How do I enable script dumping?

You don't/can't.
You can comment out the "return;" in Parse::printExpr() and run ScummVM with "-d5 --debugflags=DrawOpcodes,FuncOpcodes,GoblinOpcodes", that will dump some script-related stuff to stdout as it goes.
You can also use the aforementioned STK-extractor to dump the *.TOT files, these are the scripts in raw byte code.
There's no script-decompiler for the GobEngine (yet), though.

What does "tot" stand for? Is that a name for the script data?

Yeah, the files with the scripts in it are all called *.TOT. Why that is, I don't know. Probably an acronym for a related french term.

[sev]

Somewhere in the forums it said that it's the smallest engine in ScummVM, and that it was originally researched by Ivan Dubrov, but I haven't found more information.

If you're just looking for some scripts example, then Gob engine is not the best choice. It used to be one of the smallest engines when it supported just Gobliiins (1st in the series), but now it became one of the most complex ones .

A quite good example of simple engine, which isn't going to become much bigger is AGI. Moreover, there are nice game editors for it, such as WinAGI which let you create whole game from the scratch. Also number of Fanmade games are shipped with complete and sometimes commented scripts sources.


Eugene

[g99k]

I'd advice you to not take the GobEngine scripts. They are pretty...confusing/complex.

If you're just looking for some scripts example, then Gob engine is not the best choice. It used to be one of the smallest engines when it supported just Gobliiins (1st in the series), but now it became one of the most complex ones .

Well, I guess you guys are probably right. I tried to look deeper into the script interpretation parts, but it's quite time-consuming. What you say about the variables being stored as a consecutive block doesn't sound too encouraging either.

Maybe I'll really give up for the time being and try AGI instead, as you suggested. If I run into problems there, I'll open a new thread. At least it has way more comments, I noticed.

Two questions on the way out:

• If I can't enable script dumping, what's the command line option intended for? Are there some engines where it works?
• In case I get back to the Gob engine: Do you know the ScummVM version number where Gobliiins 1 support was complete, but the other Gob games hadn't yet been added? Maybe that single-game version is less confusing.

Still, congrats to you for re-engineering the Gob engine and getting all the games to work! I now caught a glimpse of the difficulties involved.

[sev]

If I can't enable script dumping, what's the command line option intended for? Are there some engines where it works?

-u option works only in SCUMM engine. For AGI engine you should just use WinAGI which will nicely decompile everything you need.

In case I get back to the Gob engine: Do you know the ScummVM version number where Gobliiins 1 support was complete, but the other Gob games hadn't yet been added?

0.8.0


Eugene