SCI Compression Tool

General chat related to ScummVM, adventure gaming, and so on.

Moderator: ScummVM Team

KuroShiro
Posts: 473
Joined: Thu May 15, 2008 7:42 am
Location: Somewhere Out There

Post by KuroShiro »

Don't worry, I know how to use a hex editor. :) The version of the game is also 1.000. I don't think there was another one for the CD version.

Actually, it seems to be in the middle of a bunch of data for me as well. The only time "SOL" appears in the file is at the beginning though. I'm starting to think that my resource.aud file might have just gotten messed up somewhere, the disc image has been archived and moved between various harddrives for quite a few years. The in-game speech all works fine though, so I'm not sure what exactly the problem with it is.
KuroShiro
Posts: 473
Joined: Thu May 15, 2008 7:42 am
Location: Somewhere Out There

Post by KuroShiro »

Just wanted to post a quick update -- for some reason, after building from the current HEAD, everything now compresses fine with both the CLI and the GUI. Nice work, whatever you changed! :D

The issue I was having before with the GUI was that it is not capable of creating directories, and I was trying to make a new one for the compressed file. Also, for some reason it seems that the GUI compresses files roughly 2x as much as the CLI.
Last edited by KuroShiro on Sun Aug 29, 2010 6:19 am, edited 1 time in total.
Kyahx
Posts: 13
Joined: Mon Jul 28, 2008 2:35 pm

Post by Kyahx »

Are there any nightly builds of the tools available anywhere?

I would like to compress my SCI games, but I'm not having any luck building the tools for OSX myself :(
KuroShiro
Posts: 473
Joined: Thu May 15, 2008 7:42 am
Location: Somewhere Out There

Post by KuroShiro »

Kyahx wrote:Are there any nightly builds of the tools available anywhere?

I would like to compress my SCI games, but I'm not having any luck building the tools for OSX myself :(
Yes, but not for Mac that I know of -- you'll have to build them yourself. To do so, in addition to all the libraries listed on the compiling ScummVM page, you will also need to install wxWidgets. Other than that, if you can compile ScummVM, you should be able to build the tools. Someone with more experience building on OSX could correct me. :)
agf
Posts: 8
Joined: Fri May 13, 2011 5:39 am

Post by agf »

Digging up this old thread because I have a related question.

The audio file from KQ5 from the KQ collection is in a different format -- it's not RESOURCE.[AUD|SFX], it's AUDIO001.002 -- but it still has a .MAP file associated with it. It's not recognized by the compression utility.

Is this simply unimplemented or is it a format that doesn't lend itself to compression as well as the RESOURCE.AUD format, or is it already compressed in some rudimentary way that renders additional compression less useful?

If it's just unimplemented I might take a crack at it myself.

Thanks.
User avatar
md5
ScummVM Developer
Posts: 2250
Joined: Thu Nov 03, 2005 9:31 pm
Location: Athens, Greece

Post by md5 »

agf wrote:Digging up this old thread because I have a related question.

The audio file from KQ5 from the KQ collection is in a different format -- it's not RESOURCE.[AUD|SFX], it's AUDIO001.002 -- but it still has a .MAP file associated with it. It's not recognized by the compression utility.

Is this simply unimplemented or is it a format that doesn't lend itself to compression as well as the RESOURCE.AUD format, or is it already compressed in some rudimentary way that renders additional compression less useful?

If it's just unimplemented I might take a crack at it myself.

Thanks.
The audio in KQ5CD is in an earlier format, but the compression tool does support it, AFAIK. The digital audio file decoding code can be found in the SCI engine in sci/resource_audio.cpp. Check loadResource(), loadFromAudioVolumeSCI1(), loadFromAudioVolumeSCI11(), readAudioMapSCI1() and readAudioMapSCI11()
agf
Posts: 8
Joined: Fri May 13, 2011 5:39 am

Post by agf »

md5 wrote:The audio in KQ5CD is in an earlier format, but the compression tool does support it, AFAIK. The digital audio file decoding code can be found in the SCI engine in sci/resource_audio.cpp. Check loadResource(), loadFromAudioVolumeSCI1(), loadFromAudioVolumeSCI11(), readAudioMapSCI1() and readAudioMapSCI11()
Thanks for the quick response.

Code: Select all

./scummvm-tools-cli.exe --tool compress_sci --vorbis -o AUDIO001.00g 
../../../games/sierra/King\'s\ Quest\ Collection/KQ5/AUDIO001.002
Fatal Error : Input file doesn't seem to be a valid sci audio resource file
The same command works fine for KQ6's RESOURCE.AUD. It fails with a mingw compiled current git checkout for the KQ5 AUDIO001.002. Renaming the file has no effect.

I tried the GUI (1.2.0), and after selecting the file it suggests either compress_agos or encode_dxa, which are both obviously incorrect. I tried compress_agos anyway and it quickly gives a wrong format error (as expected) once it starts. Renaming the file to RESOURCE.AUD causes the GUI to select compress_kyra, which errors out with an assertion failed once it starts.

EDIT: compress_sci isn't listed in the 1.2.0 tools readme, was it perhaps added after that?

EDIT2: looking at the compress_sci source, this error is thrown if it doesn't find a recognized header at the beginning of the file -- maybe there is no header in the SCI1 audio file.

EDIT3: So, according to resource_audio, there is no header in SCI1 audio resources, and the map file looks like this:

Code: Select all

AUDIOnnn.MAP contains 10-byte entries:
Early format:
w 5 bits resource type and 11 bits resource number
dw 7 bits volume number and 25 bits offset
dw size
Later format:
w nEntry
dw offset+volume (as in resource.map)
dw size
ending with 10 0xFFs
Doesn't appear to be supported.

EDIT Final:

From somewhere else in compress_sci.c:

Code: Select all

	// First find out how many samples are in this file
	//  We only support SOL audio and WAVE
	//  We can't support games that use "raw" audio (would have to walk through audio map for those, which would
	//   complicate this code - all talkie games that use this (kq5) don't use much space anyway, so not supporting
	//   those isn't a big issue
agf
Posts: 8
Joined: Fri May 13, 2011 5:39 am

Post by agf »

I wrote a python script to convert the KQ5 AUDIO001.002 to a format compress_sci will work on. I just use the AUDIO001.MAP file to run each resource through python's wave module then write them out to one file.

compress_sci dropped the file size (using vorbis default settings) from 93,229KB to 35,700KB -- this seems like a worthwhile improvement to me.

Next step is to change scummvm/engines/sci/resource_audio.cpp to recognize this format for KQ5.

EDIT: The error I get right now is "could not translate offset to compressed offset in audio volume!"

EDIT2: Realized what I missed. I didn't actually write the new offsets to a new map file.
agf
Posts: 8
Joined: Fri May 13, 2011 5:39 am

Post by agf »

OK, I'm using KQ5 now with compressed audio.

The python script below rewrites the AUDIO001.002 from raw sounds to WAVE sounds, so compress_sci understands it.

It also rewrites AUDIO001.MAP to match the new WAVE formatted file, which compress_sci needs to create an offset translation table.

The files created by this program work fine with ScummVM without compression as well, and the AUDIO001.002 file is actually slightly smaller than the original because the raw sounds were padded to 2048 bytes.

The script should work on any normal python2 install (2.5+)


Code: Select all

#!/usr/bin/env python2
from __future__ import with_statement
from struct import unpack, pack
from StringIO import StringIO
from wave import open as wave_open

# dir with KQ5 installed
kq5_dir = "C:\\Users\\adam\\Games\\sierra\\King's Quest Collection\\KQ5"
# original AUDIO001.002
kq5_aud = "AUDIO001.002.bak"
# original AUDIO001.MAP
kq5_map = "AUDIO001.MAP.bak"
# new AUDIO001.002
aud_new = "AUDIO001.002.new"
# new AUDIO001.MAP
map_new = "AUDIO001.MAP.new"

def parse_map():
    with open('\\'.join((kq5_dir, kq5_map)), 'rb') as file: map = file.read()
    return tuple((fields[0], fields[1] >> 28, fields[1] & 0x0fffffff, fields[2])
                    for fields in &#40;unpack&#40;'<HII', map&#91;n&#58;n+10&#93;&#41;
                        for n in range&#40;0, len&#40;map&#41; - 10, 10&#41;&#41;&#41;

def convert_to_wav&#40;&#41;&#58;
    res_map = parse_map&#40;&#41;
    with open&#40;'\\'.join&#40;&#40;kq5_dir, kq5_aud&#41;&#41;, 'rb'&#41; as aud_in&#58;
        with open&#40;'\\'.join&#40;&#40;kq5dir, aud_new&#41;&#41;, 'wb'&#41; as aud_out&#58;
            with open&#40;'\\'.join&#40;&#40;kq5dir, map_new&#41;&#41;, 'wb'&#41; as new_map&#58;
                for num, volume, offset, size in res_map&#58;
                    aud_in.seek&#40;offset&#41;
                    temp = StringIO&#40;''&#41;
                    wav = wave_open&#40;temp, 'wb'&#41;
                    wav.setparams&#40;&#40;1, 1, 11025, 0, 'NONE', 'not compressed'&#41;&#41;
                    wav.writeframes&#40;aud_in.read&#40;size&#41;&#41;
                    new_offset = aud_out.tell&#40;&#41;
                    aud_out.write&#40;temp.getvalue&#40;&#41;&#41;
                    new_map.write&#40;pack&#40;'<HII', num,
                                       &#40;&#40;volume << 28&#41; & 0xffffffff&#41; + new_offset,
                                       aud_out.tell&#40;&#41; - new_offset&#41;&#41;
                new_map.write&#40;pack&#40;'<HII', 0xffff, 0xffffffff, 0xffffffff&#41;&#41;

if __name__ == '__main__'&#58; convert_to_wav&#40;&#41;
User avatar
md5
ScummVM Developer
Posts: 2250
Joined: Thu Nov 03, 2005 9:31 pm
Location: Athens, Greece

Post by md5 »

Thanks for your work :)
This functionality should be included in the compression tool eventually, but until then your script is very useful indeed!
agf
Posts: 8
Joined: Fri May 13, 2011 5:39 am

Post by agf »

I've posted a patch to the tracker to add this functionality to compress_sci.

https://sourceforge.net/tracker/?func=d ... tid=418822

EDIT: And I posted a bug report about an issue I spotted while writing the patch.

https://sourceforge.net/tracker/?func=d ... tid=418820
User avatar
NoiZje
Posts: 24
Joined: Wed May 25, 2011 6:17 pm
Location: Antwerpen

Post by NoiZje »

The CD version of Jones in the Fast Lane also uses the AUDIO001.MAP AUDIO001.002 file structure.
agf
Posts: 8
Joined: Fri May 13, 2011 5:39 am

Post by agf »

NoiZje wrote:The CD version of Jones in the Fast Lane also uses the AUDIO001.MAP AUDIO001.002 file structure.
I don't have this game, but you could try changing the

Code: Select all

if &#40;_input.size&#40;&#41; == 0x05C9B000&#41; &#123;
line in the patch to

Code: Select all

if &#40;_input.size&#40;&#41; == 0x05C9B000 or _input.size&#40;&#41; == 0xjones_file_size_in_hex&#41; &#123;
with the size of your AUDIO001.002 file and see if it works.

It's dependent on the sample format (8bit mono 11025 with each sample padded to multiples of 2048 bytes) as well as the AUDIO001.MAP format.
SimSaw
Posts: 70
Joined: Tue Nov 01, 2005 11:10 am
Location: Germany
Contact:

Re: SCI Compression Tool

Post by SimSaw »

Sorry for bringing this old topic up again, but no matter what I try, I always get the error message:
Unexpected input file 'AUDIO001.002'!

I've been trying with the command
scummvm-tools-cli --tool compress_sci --flac AUDIO001.002
after using the python tool above and I made sure I indeed have WAVE files in the archive.

Am I doing something wrong or is the tool defective?
User avatar
Praetorian
ScummVM Developer
Posts: 883
Joined: Tue May 08, 2007 8:54 am
Location: Greece
Contact:

Re: SCI Compression Tool

Post by Praetorian »

SimSaw wrote: Tue Jun 18, 2024 8:51 pm Sorry for bringing this old topic up again, but no matter what I try, I always get the error message:
Unexpected input file 'AUDIO001.002'!

I've been trying with the command
scummvm-tools-cli --tool compress_sci --flac AUDIO001.002
after using the python tool above and I made sure I indeed have WAVE files in the archive.

Am I doing something wrong or is the tool defective?
From the source code, it looks like the compress_sci tool expects an input file strictly named in the format of "resource.*".
https://github.com/scummvm/scummvm-tool ... ci.cpp#L55

Copy the AUDIO001.002 and AUDIO001.MAP files out of the King's Quest game data folder, and rename the first (AUDIO001.002) to "resource.002". Keep the map file with its original name (AUDIO001.MAP).

Run the command line script:

Code: Select all

scummvm-tools-cli.exe --tool compress_sci --flac -o AUDIO001.flac resource.002
I am unsure if the python script fix has been integrated in the tool, or it's needed to be run before using the compress_sci tool.
I was able to run the tool with either version of AUDIO001.002 resource (original or modified with the python script), without errors, and the result had the same size, but the actual contents were different.

Also keep in mind that the above script as pasted needs editing and a couple of fixes to run correctly.
My "corrected" version looks like this (with the kq5_dir also edited to point to the folder for my KQ5 data files):

Code: Select all

#!/usr/bin/env python2
from __future__ import with_statement
from struct import unpack, pack
from StringIO import StringIO
from wave import open as wave_open

# dir with KQ5 installed
kq5_dir = "C:\\Users\\adam\\Games\\sierra\\King's Quest Collection\\KQ5"
# original AUDIO001.002
kq5_aud = "AUDIO001.002.bak"
# original AUDIO001.MAP
kq5_map = "AUDIO001.MAP.bak"
# new AUDIO001.002
aud_new = "AUDIO001.002.new"
# new AUDIO001.MAP
map_new = "AUDIO001.MAP.new"

def parse_map():
    with open('\\'.join((kq5_dir, kq5_map)), 'rb') as file: map = file.read()
    return tuple((fields[0], fields[1] >> 28, fields[1] & 0x0fffffff, fields[2])
                    for fields in (unpack('<HII', map[n:n+10])
                        for n in range(0, len(map) - 10, 10)))

def convert_to_wav():
    res_map = parse_map()
    with open('\\'.join((kq5_dir, kq5_aud)), 'rb') as aud_in:
        with open('\\'.join((kq5_dir, aud_new)), 'wb') as aud_out:
            with open('\\'.join((kq5_dir, map_new)), 'wb') as new_map:
                for num, volume, offset, size in res_map:
                    aud_in.seek(offset)
                    temp = StringIO('')
                    wav = wave_open(temp, 'wb')
                    wav.setparams((1, 1, 11025, 0, 'NONE', 'not compressed'))
                    wav.writeframes(aud_in.read(size))
                    new_offset = aud_out.tell()
                    aud_out.write(temp.getvalue())
                    new_map.write(pack('<HII', num,
                                       ((volume << 28) & 0xffffffff) + new_offset,
                                       aud_out.tell() - new_offset))
                new_map.write(pack('<HII', 0xffff, 0xffffffff, 0xffffffff))

if __name__ == '__main__': convert_to_wav()
Disclaimer: The results may be unusable, I cannot tell. I can only test that the tool produces something and does not error out.
Post Reply