Grim Fandango Deluxe - Original Thread [Locked]

ultraneonoirantihero · Mon Oct 10, 2011 6:12 pm

So is there any priority list of things needed to be done first?

Also, how are we gonna distribute those updated textures? Zip, extract, replace and repack back the lab files?

I asked earlier if someone could make ResidualVM detect additional lab (zip?) files and automatically load new textures from those "standalone updates"? Easier then: backup, extract, unpack, replace and repack.

ioannis · Post by **ioannis** » Mon Oct 10, 2011 6:49 pm

I'm running a script in the background using the tools I got from residual to extract mats to ppms (almost the same as bmps).

It will automatically create all possible combinations of each material / palette. (This can take awhile)

I know it will propably make thousands of files, but perhaps I could simple pick out the correct ones through viewing them in thumbnails.

Is there really no other way? Am I just wasting time?

If my calculations are correct, I'll get at LEAST (some files have multiple sizes of textures inside'em or something) 104673 files. Properly structured though.

My script outputs files like this:
For above_door_01.mat check with above_door.cmp,
it creates:

out/above_door_01.mat/above_door.cmp/ (here are the extracted the ppms)

for above_door_01.mat -> action.cmp

out/above_door_01.mat/action.cmp/

...and goes on and on...

guess it will finish ... in about 3 hours or so :p

JohnnyWalker2001 · Post by **JohnnyWalker2001** » Mon Oct 10, 2011 7:26 pm

ioannis wrote:I'm running a script in the background using the tools I got from residual to extract mats to ppms (almost the same as bmps).

It will automatically create all possible combinations of each material / palette. (This can take awhile)

I know it will propably make thousands of files, but perhaps I could simple pick out the correct ones through viewing them in thumbnails.

Is there really no other way? Am I just wasting time?

If my calculations are correct, I'll get at LEAST (some files have multiple sizes of textures inside'em or something) 104673 files. Properly structured though.

My script outputs files like this:
For above_door_01.mat check with above_door.cmp,
it creates:

out/above_door_01.mat/above_door.cmp/ (here are the extracted the ppms)

for above_door_01.mat -> action.cmp

out/above_door_01.mat/action.cmp/

...and goes on and on...

guess it will finish ... in about 3 hours or so :p

Holy crap. Maybe you could share this script?

ioannis · Post by **ioannis** » Mon Oct 10, 2011 7:32 pm

Sure thing. It's in BASH though.
I'm running it on a Ubuntu Virtual Machine.

It requires:
1. Compiled mat2ppm from residual tools
2. folder mats with .mat files
3. folder cmps with .cmp files
4. an empty folder out

-
After it checks the first mat it will display folder exists , that's for out/[mat name goes here]. No worries it is still running fine

Sorry didn't mean to publish it, so it really is sucky-written.

Code: Select all

#!/bin/bash
MATS=mats/*
CMPS=cmps/*
for c in $CMPS
do
  echo "Using $c palette..."
  	for m in $MATS
	do
  	 echo "on $m texture..."
		mfile=`echo $m | cut -d"/" -f2`
		cfilename=`echo $c | cut -d"/" -f2`
		mfilename=`echo $mfile | cut -d"." -f1`
	
		mkdir out/$mfile
		./mat2ppm -c $c $m
		mv "$mfilename"_0.ppm out/$mfile/"$mfile"_"$cfilename".ppm
  	done
  
done

If my output compressed is still under 500mb, I will upload all files for you to check

ioannis · Post by **ioannis** » Mon Oct 10, 2011 7:38 pm

Wait a sec. Come to think of it...

We do not need to actually combine mats and cmps...
We could easily just rescale the mats in their original format. Just an idea...

But converting them to bmps and back to mat, also means we could re-create these textures for even higher quality.

somaen · Post by **somaen** » Mon Oct 10, 2011 10:56 pm

Just rescaling them in their original format is basically no change at all, unless you intend to draw in any of the missing detail

ioannis · Post by **ioannis** » Tue Oct 11, 2011 5:25 am

Ok script did finish, resulting in a whooping number of about 210k files. 6.2 GBs !

Compressing now, just for the fun of it.

JohnnyWalker2001 · Post by **JohnnyWalker2001** » Tue Oct 11, 2011 2:28 pm

Great, thanks for posting that. I think I can do something with mat2ppm myself... although how does it know the right CMP file? (I presume it's hard coded?) I'll take a look.

ioannis wrote:Wait a sec. Come to think of it...

We do not need to actually combine mats and cmps...
We could easily just rescale the mats in their original format. Just an idea...

But converting them to bmps and back to mat, also means we could re-create these textures for even higher quality.

I can't follow what you're saying here. Why would making the images in the MAT files four times bigger be in any way desirable? They wouldn't look any better.

Have you read the following, I wonder?
http://grimfandangodeluxe.blogspot.com/ ... angos.html

ultraneonoirantihero · Wed Oct 12, 2011 2:08 pm

I have no idea what you are currently talking about so I have to ask: what are mats, ppms, cmps and why are there 210.000 of them?

ioannis · Post by **ioannis** » Wed Oct 12, 2011 7:24 pm

.mat files are "Materials" (containing only pixelmap) thus Textures (in essence)
.cmp files are "Colormaps" thus Palettes (in essenece)
.ppm files are Materials with a Colormap, thus a proper image. (similar to .bmp , or jpg. can be opened just fine in photoshop)

One thing we're missing is which material is combined with which colormap, in order to get the right texture. Thus, I tried all combinations for the heck of it. Result is 210k files :/ , I browsed it through Adobe's Bridge (nice previews) but still can't be SURE which mat goes with which cmp. Check the files yourselves if you want to.

hxxp://www.filesonic.com/file/2534677921/out.rar

Warning (210k files can take a nice fragmentation road on their way to extraction. Prefer an external drive or a non-fragmented drive)

P.S File doesn't not contain luc@s @rts original files so I guess it's ok sharing it...

how does it know the right CMP file? (I presume it's hard coded?)

After browsing ResidualVM's source for a while, it seems like the engine/game itself loads the apropriate Colormaps- still not sure though...

Perhaps we could force the game in residual to load all 3d models and log every resource file it used? Just an idea...

giucam · Post by **giucam** » Wed Oct 12, 2011 8:21 pm

ioannis wrote:After browsing ResidualVM's source for a while, it seems like the engine/game itself loads the apropriate Colormaps- still not sure though...

Yes, the colormaps are set either by the costumes or, if they don't set any, a default one is used, which can be one set by the current room or, if there isn't any, item.cmp is used.

ioannis wrote:Perhaps we could force the game in residual to load all 3d models and log every resource file it used? Just an idea...

I don't know if it's possible to load all the models but it's surely possible to output which colormap is used for which costume, and so, more or less, for each 3do.
A same 3do can have different colormaps set at different times, so there's no a 1<->1 relation between costumes and 3dos.

JohnnyWalker2001 · Post by **JohnnyWalker2001** » Wed Oct 12, 2011 8:48 pm

Thanks to cplhenshaw we have a complete list of .3DO -> .CMP relationships here:

https://docs.google.com/spreadsheet/pub ... utput=html

(As posted on the previous page.)

ultraneonoirantihero · Thu Oct 13, 2011 10:49 am

ioannis wrote:.mat files are "Materials" (containing only pixelmap) thus Textures (in essence)
.cmp files are "Colormaps" thus Palettes (in essenece)
.ppm files are Materials with a Colormap, thus a proper image.

So ppm = mat + cmp? Just like: video image = chroma + luma?

ioannis wrote: One thing we're missing is which material is combined with which colormap, in order to get the right texture.

So how did Johnny Walker managed to remaster Manny with the right texture and correct color map?

giucam wrote:Yes, the colormaps are set either by the costumes or, if they don't set any, a default one is used, which can be one set by the current room or, if there isn't any, item.cmp is used.

JohnnyWalker2001 wrote:Thanks to cplhenshaw we have a complete list of .3DO -> .CMP relationships here:
https://docs.google.com/spreadsheet/pub ... utput=html

So the 3D models have the reference to the color pallete which is used by the model's texture? AKA: The 3DO file uses MAT file and has a reference to the CMP file? Then the 3do file has all the necessary data?

ioannis wrote:Result is 210k files :/

Rough count of those 3d models from the 3do-cmp list is ~550, and some of them use the same colormap, so... let's say ~550 models and ~ 350 colormaps, that is: 550 x 350 = 192 500. Pretty close to that " 210k files " :)

So, since the 3d0-cmp-mat relation is known, can the right combination of cmp and mat files be found?

P.S. I restored the deleted post.

ioannis · Post by **ioannis** » Thu Oct 13, 2011 11:03 am

Not sure if there is an easy way to do that... My approach is a little odd/ bruteforcing actually. My guess is you could identify the right combination from my output, by sight... still trying to find a programmatically way to do it though.

Ive been going over the residual code but ts somehow hard to make out what's going on.

cplhenshaw · Post by **cplhenshaw** » Thu Oct 13, 2011 1:42 pm

Each .3do file specifies the .mat files that it uses, and we also have a list of .cmp files used by each .3do file. So if you know which model a texture belongs to, the palette should be the .cmp listed for that model.