Extract Gobliiins content

[LunaVorax]

Hi everyone !

I wanted to know how to extract the content of Gobliiins, such as sprites, backgrounds and sounds (and especially sounds).

I know there's a lot of easier ways, but I want the hard one.

Can anyone help me please ?
Thanks for your interest

Oh, and sorry if I speak bad english.

[DrMcCoy]

but I want the hard one.

That's good, because it won't be "just click that"-easy.
You will have to read the code of ScummVM's gob engine (in engines/gob).

If that frightens you and/or you don't think you're up to that, stop reading here.


Most of the data is in the STK-files (*.stk, *.itk), which consist of lots of files packed together, sometimes compressed with LZ77. Look in dataio.cpp for exactly how, or look at a simple extractor I once wrote.

LET-files are the fonts (Util::loadFont(), and Video::drawLetter()).
SND-files are sounds (raw, mono signed 8-bit; Inter::loadSound() and sound.cpp).
TOT-files are scripts (inter*.cpp and parse*.cpp)
ADL-files are AdLib-music files (music.cpp).
INS- and DUM-files are instrument and music data files for the Amiga music (sound/mods/infogrames.cpp)
AVO-files store the screen layout (only in Gobliiins, the other games do it differently; Map_v1::loadMapObjects()).
In DAT-, ALL-, ESP-, ITA-, USA-, NDL-, KOR-, ISR- and ANG-files, you'll find text for different languages (Draw_v2::printTotText()).
IMD- and VMD-files are videos (videoplayer.cpp and coktelvideo.cpp)

Resources can also be in TOT-, EXT- and IM-files, accessable per ID (Game::loadTotResource() and Game::loadExtData()), and can also be compressed with LZ77.
What they are isn't specified there, only the game scripts know that (i.e. you'd have to find that out yourself).
The images are stored in raw data, so you'd have to located the right palette using the scripts, too.

[LunaVorax]

Thank you so much for this (very) complete and useful answer.

Really, I love you

[theruler]

Now that we're in topic...
Mighty DrMcCoy, do you plan to support audio compression even for itk files? (goblins 3 and Woodruff)

cheers.

[DrMcCoy]

Thank you so much for this (very) complete and useful answer.

If you have any more specific questions, ask away.

Really, I love you

Mighty DrMcCoy

O_o

do you plan to support audio compression even for itk files? (goblins 3 and Woodruff)

Short answer: No.

Slightly longer answer: Mu (as in "Your question cannot be answered because it depends on incorrect assumptions")

Long answer: (See this feature request; updates in bold)
Well, that's not that easy. *g*
First of all, ITKs are the same as STKs, only an archive format bundling several files together.
The actual speech is within the IMD videos [VMD for Woodruff]. And there lies the problem, the sound is interleaved with the video. There are n sound frames as a preliminary buffer, after that, there are alternating video and sound frames.
So a compressor would have to take the IMDs, go through them, collect all sound frames, compress the resulting sound, split the compressed sounds into chunks having _exactly_ the same lengths as the orignal ones and put re-interleave them with the video. Moreover, all offsets to different parts
of the video have to be re-adjusted, and there are lots in a normal IMD.
All in all, rather complicated.

Also, it wouldn't make the ITKs that smaller, since only a part, the sound, is compressed then, while the video stays that way.
And yes, the video frames are already (losslessly) compressed.
[The 16-bit sound in VMDs is also already delta pulse-code modulated (DPCM), which is a form of simple compression]

[Only applies to Goblins 3]
Ah, and no, you can't just take only one copy from each video and throw the rest (from the other languages) away, the videos are timed so that the mouth-movement fits with the spoken word.
You can however only keep the file for the language(s) in which you intend to play. If you play, for example, in english, the game doesn't need the other files.

[theruler]

incorrect assumptions

oops!

The 16-bit sound in VMDs is also already delta pulse-code modulated (DPCM), which is a form of simple compression

I thought that was the larger part of the 500MB itk file of Woodruff...
But now I understand.

Qualified and exhaustive answer.
Thanks a lot.

the videos are timed so that the mouth-movement fits with the spoken word.

WOW! tricky!


You know,
I love you too, Mr.
You've been promoted in my personal hero list... ALLMighty DrMcCoy.

[theruler]

I have another question about this topic:

zzzzzzzz.zzz, what the heck is that?
It seem it share the same code and index of the itk/stk files...
there are a lot of IMD (videos as you explained) files... but it is not needed nor used even in DOS ambient.
I wonder since I was a child.

I have coktel2.imd and mdos.imd too...
Funny, I named them "coktel.imd" and played it through the game intro... mdos.imd is a logo I've never seen.

Is there a .imd file player around the net?
DrMcCoy, could you please address me towards the source code in the gob engine that allow this?

cheers.

[DrMcCoy]

zzzzzzzz.zzz, what the heck is that?

No idea. It's obviously not needed and I haven't seen it used in the disassembly, either.
Some kind of residue left behind from development/debugging, I'd guess.

I have coktel2.imd and mdos.imd too...
Funny, I named them "coktel.imd" and played it through the game intro... mdos.imd is a logo I've never seen.

coktel2.imd seems to be smaller (in file size) version of the Coktel logo with less frames (that's why it stutters).
The MDO logo is probably a placeholder when the real logo wasn't finished yet (the group of people who did the programming call themselves "MDO").

Is there a .imd file player around the net?

Not that I know of. Could be that FFmpeg will someday play them. It already plays VMDs and while these are vastly used in Sierra games, they are clearly a derivated from IMDs.

DrMcCoy, could you please address me towards the source code in the gob engine that allow this?

The video playing? Look in coktelvideo.cpp and videoplayer.cpp. The former implements the actual video formats, while the latter uses that to play videos in the gob games.

[chojin]

I have to ask the annoying question again ;)

The file woodruff.itk is about 533 MByte in size (german best of sierra version) and zip-compressed together with intro.stk it's 204 MByte. I don't know how much is possible with sound and video compression here.

I remember The Feeble Files compression (4CD/Windows/German) was a funny marathon with several tools and steps. Do you think there will be a similar way, or is it simply not worth the trouble?

[fingolfin]

The actual speech is within the IMD videos [VMD for Woodruff]. And there lies the problem, the sound is interleaved with the video. There are n sound frames as a preliminary buffer, after that, there are alternating video and sound frames.
So a compressor would have to take the IMDs, go through them, collect all sound frames, compress the resulting sound, split the compressed sounds into chunks having _exactly_ the same lengths as the orignal ones and put re-interleave them with the video. Moreover, all offsets to different parts
of the video have to be re-adjusted, and there are lots in a normal IMD.
All in all, rather complicated.

Forewarning: I know little about the Gob engine, and less about VMDs/IMDs, so the following might be nonsense.

However, DrMcCoy, would it maybe instead be possible to do this: Instead of trying to go the indeed very complicated rout above, would it be possible to undo the interleaving, encode the video as DXA (which might be a tad better compressed than the existing compression -- worth a try, maybe?), and then use a different A/V syncing method -- we already do A/V syncing for DXA+ Ogg Vorbis/MP3/Flac in various other engines.

In short: Go through the IMDs, collect all sound frames, compress the resulting sound, but do NOT split it again. Rather, do time based syncing (in some A/V container formats, this is enhanced by storing a table of some frame count <-> time mappings, to allow for resyncing every couple ms/frames or so). Err, I could elaborate, but I guess you get the gist. Mind you, I don't say that this would be *easy*, but maybe less complicated than trying to fit everything back into the original format...


I understand that the lip syncing might increase the difficulty level some more. However, wouldn't it still be possible to do frame based lip syncing? Well, I do not know how lip syncing works in Gob3, so maybe this is harder than I imagine.

Of course, it would still require "somebody" to do all that work. So this is more a hypothetical discussion. But I am curious nevertheless

[DrMcCoy]

Not really... I would lose information these files hold additional to audio and video.
Like default X and Y values, new palettes or the changes rectangle. Or the foot anchors in VMDs (frame types 3-5, with 0 as a separator).
The more state-machine-ish IMD also allows for jumping to another frame within the opcodes of a frame.

Having the audio as an Ogg Vorbis/MP3/FLAC might even make it larger for some files, because IMDs/VMDs can specify sound frames as empty, they get filled up with silence while playing (and the files in all games currently supported use 8bit audio only).

[gomtuu]

Hello, I found this thread while I was looking for information on Ween: The Prophecy in order to extract its music files.

Thanks for the helpful info, DrMcCoy! That, combined with the ScummVM source, allowed me to find the game's AdLib files. However, now comes the hard part: AdPlug won't play the files, so I guess they're not in one of the formats it recognizes.

I know the ScummVM Gob engine AdLib player's source is available here, so I could adapt that to output a file AdPlug would recognize, but is there a quicker way? Has someone already written a utility like that, for instance?

I know I could use DOSBox to dump the OPL instructions to a DRO file, but 0.72 produces bad dumps and 0.73 uses a new DRO format that AdPlug doesn't understand. Besides, capturing music with DOSBox is tricky because you have to time it just right or edit the DRO file later.

I'd appreciate any pointers. Thanks,
gomtuu

[bobdevis]

Slightly longer answer: Mu (as in "Your question cannot be answered because it depends on incorrect assumptions")

Sneaking a null pointer into general language... how very C++ of you...

[deadagain]

So, any hints on the recent compress_gob.exe (and its somewhat cryptic <Conf file> input)??

[gomtuu]

So, any hints on the recent compress_gob.exe (and its somewhat cryptic <Conf file> input)??

No, I've only used extract_gob_stk and degob.