MONKEY: Wrong Palette for Amiga-Versions

[ceztko]

NB: This thread continues the discussions in this bug report.

WARNING: It seems Winuae is unreliable concerning colors displayed compared to e-uae (unix platform). IMO, there's some Windows platform problem, as e-uae in linux is working as I'm expecting: see comparison between quickgrab.005.png and screen1-euae.png below. Maybe 32bit palette used in window isn't a regular RGB palette like 12bit/24bit ones are, according to [1], or some kind RGB overlay is used by default in Windows and I can't get rid off. So this is my recommendation: don't use Winuae for testing purpose. Use e-uae (morgoth repositories[3]) in a unix enviroment with a 24bit display mode.

@nolange: I realize you opened this bug report to see 2) solved. I wanted to talk with you personally before intervening, but kirben asked me to disccus here, so I did. It seems you are convinced that:
a) is impossible to map amiga native colors, as a 12bit palette is not a subset of a 24bit palette;
b) because of a, accuracy of colors in screenshots taken in a real amiga depend on how a 12bit color is converted in a 24bit color;
c) because of a, accuracy of colors in screenshots taken in a Winuae depend on how Winuae converts a 12bit color in a 24bit color;
d) because of a and b, it's ok to use Winuae as a reference because you trust more Winuae 12bit->24bit conversion;
e) scummvm uses the same 24bit colors that winuae actually (recent versions) displays.

My opinion is that a,d and e are not true:

a) 4096 colors is the OCS theoric palette (even if you could use really the full palette only on HAM mode). According [1]:

(OCS Palette) = 4096 = 2^12 = 2^4 * 2^4 * 2^4 --> 4-4-4 representation
(Truecolor) = 2^24 = 2^8 * 2^8 * 2^8 --> 8-8-8 representation

2^8 / 2^4 = 2^4

Do you want to perform a conversion of the color x = (r,g,b) in a 12bit palette, 4-4-4 representation, to a color y in a 24bit palette, 8-8-8 representation? y = (c * 2^4, g * 2^4, b * 2^4)

The conversion is so easy that it doensn't need any rounding/truncating. Or the conversion is wrong?

d) As of 26 Sep 2005, (Win)UAE was using a bugged 12bit -> 24bit conversion, as I discovered [2];
e) As far as I know, palette that are used NOW in scummvm were grabbed before 26 Sep 2005, so your affirmatation "all fixed palettes in ScummVM use the "WinUAE colors" may have been true before 26 Sep 2005, but it is no more actual.

Now, let pass to something more objective. The following is one of the screenshots I took in 2003:

http://web.tiscali.it/franzo/QuickGrab.005

It's in ILBM format, so probably it stored a 12bit rappresentations of colors (!). This was later converted in png:

http://web.tiscali.it/franzo/quickgrab.005.png

Here is a screenshot taken today in e-uae (any recent build [3]):

http://web.cheapnet.it/imeriopretto/screen1-euae.png

With your favourite tool/app compare numerically the colors. Here are the results I got:

Code: Select all

                                        Real Amiga              e-uae
                                        (quickgrab.005.png)     [screen1-euae.png]

Red (Guybrush's skin)                   (255,85,85)             (255,85,85)
Yellow (the pixel on Gubrush's shoes)   (255,255,85)            (255,255,85)
Violet (Governor's hat)                 (170,0,170)             (170,0,170)
Brown (Guybrush's hairs)                (170,85,0)              (170,85,0)
...

There is a 100% match. I'm taking this as a confirm to my suppositions. In a reply to a my question, Toni Wilen wrote me:

> Maybe amiga native screenshots are not paragonable with the grabs taken with winuae? ...
This was with or without AGA? Non-AGA modes should have exact the same colors, even at 16-bit.

--

Now, assuming e-uae and scummvm colors are comparable (and I think they should be, as e-uae IMO doesn't need to do any truncating/rounding), here is a scummvm screenshot:
NOTE: to compare e-uae colors with scummvm, I ran scummvm 0.13.1 in linux, to be 100% sure screenshots are really comparable.

http://web.cheapnet.it/imeriopretto/screen1-scummvm.png

and here is a numeric comparison between e-uae and scummvm colors:

Code: Select all

                                        e-uae                   scummvm 0.13.1
                                        [screen1-euae.png]      [screen1-scummvm.png]

Red (Guybrush's skin)                   (255,85,85)             (248,84,80)
Yellow (the pixel on Gubrush's shoes)   (255,255,85)            (248,252,80)
Violet (Governor's hat)                 (170,0,170)             (168,0,168)
Brown (Guybrush's hairs)                (170,85,0)              (168,84,0)
...

Why they are different? Because IMO the palette grabbed for scummvm is wrong because a bugged Winuae version was used (and this is the nth times I am repeating this).

To conclude: nolange, is good you want to fix the flower color (I confirm the bug) and be sure Guybrush's face color is ok, but I think I gave you enough hints that palettes used NOW in scummvm need first to be wholly rechecked.
As I told before, these are my recommendations: use e-uae in a 24bit display mode and gimp to grab screenshot. DON'T use Winuae.


[1] http://en.wikipedia.org/wiki/List_of_mo ... B_palettes
[2] http://www.freelists.org/post/uae/Palet ... and-WinUAE
[3] http://morgoth.free.fr/ubports/

[DrMcCoy]

The values between e-uae and ScummVM 0.13.1 are just a little bit off, hardly in a noticable way. I'd say they're probably a result of fact that the end surface in the SDL backend is a 16bit surface and therefore some granularity is lost (similarly, gradients, in for example Ween, don't look as smooth as they should).

[ceztko]

I'd say they're probably a result of fact that the end surface in the SDL backend is a 16bit surface and therefore some granularity is lost (similarly, gradients, in for example Ween, don't look as smooth as they should).

Even e-uae (not Winuae) use SDL as a backend, but it displays same colors as my screenshots taken in real Amiga. Sure, it could be using SDL differently from what scummvm does. Whatever, I'd like you to move focus of attention to the REAL problem.

If scummvm amiga palettes were discovered using Winuae before 26 Sep 2005, and Winuae has been proved to be bugged (no doubt in this) --> scummvm palettes are wrong unless someone fixed them in the mean.

Give me a proof scummvm 0.13.1 palettes are now using internally the same colors of recent build of e-uae (as I've told before, e-uae and NOT winuae, because of Windows platform related issues), and I won't speak of amiga palettes in scummvm forever

[fingolfin]

Do you want to perform a conversion of the color x = (r,g,b) in a 12bit palette, 4-4-4 representation, to a color y in a 24bit palette, 8-8-8 representation? y = (c * 2^4, g * 2^4, b * 2^4)

The conversion is so easy that it doensn't need any rounding/truncating. Or the conversion is wrong?

That conversion works, and is not completely wrong. But actually it is not the best either, because it does in fact lack rounding, which *is* required for an absolutely faithful conversion despite what you say. It will turn the color (0xF,0xF,0xF) into (0xF0,0xF0,0xF0) -- correct would be (0xFF,0xFF,0xFF).

The proper way to do the conversion would be this (I only show it for the R-component -- of course it's the same for the others):
rNew := (rOld *255) / 15 = rOld * 17 = rOld<<4 + rOld
This is so because the old range was 0-15, the new range is 0-15, and we want to map the old minimum 0 to the new minimum 0, and the old maximum 15 to the new maximum 255.

Why they are different? Because IMO the palette grabbed for scummvm is wrong because a bugged Winuae version was used (and this is the nth times I am repeating this).

Just to clarify this: The bug report above is to the best of my knowledge the first time (and not the nth) that you reported this as a proper bug report. A forum post unfortunately is never a good way to report bugs (and in fact we are quite explicitly against it in our forum rules), because many devs never or rarely read the forums, and thus never would see those report. Also forum posts can be overlooked or forgotten, while bug tracker entries stay around.

Anyway, I appreciate your detailed analysis and report of the issue.

[DrMcCoy]

Even e-uae (not Winuae) use SDL as a backend, but it displays same colors as my screenshots taken in real Amiga.

Probably not with a 16bit surface.
A 16bit surface does change values, because it only leaves 5 bits each for 2 and 6 bit for one color component.

For example, with RBG565, (255,85,85) [(11111111b, 1010101b, 1010101b)] would turn to (248,84,80) [(11111000b, 1010100b, 1010000b)], since the lower bits (3 for R and G, 2 for G) are lost. Do these values look familiar?


EDIT:

Code: Select all

e-uae                   e-uae in RGB565         scummvm 0.13.1
[screen1-euae.png]                              [screen1-scummvm.png]

(255,85,85)             (248,84,80)             (248,84,80)
(255,255,85)            (248,252,80)            (248,252,80)
(170,0,170)             (168,0,168)             (168,0,168)
(170,85,0)              (168,84,0)              (168,84,0)

That looks right this far at least.

[ceztko]

@fingolfin: Yes! You are right! And I've just discovered that my guessed conversion is just what (Win)uae did before 26 Sep 2005. Here is what Toni Wilen (actual maintainer of Winuae) replied me[1]:

Code: Select all

Forget my earlier post..


I think gfxutil.c, alloc_colors64k() should be something like this or 0xfff (OCS/ECS Amiga's color space) will get translated to 0xf0f0f0, not 0xffffff (24-bit color space) etc..

I think this been wrong since the beginning..


void alloc_colors64k (int rw, int gw, int bw, int rs, int gs, int bs, int aw, int as, int alpha)
{
int i;

for &#40;i = 0; i < 4096; i++&#41; &#123;
int r = &#40;&#40;i >> 8&#41; << 4&#41; | &#40;i >> 8&#41;;
int g = &#40;&#40;&#40;i >> 4&#41; & 0xf&#41; << 4&#41; | &#40;&#40;i >> 4&#41; & 0x0f&#41;;
int b = &#40;&#40;i & 0xf&#41; << 4&#41; | &#40;i & 0x0f&#41;;
xcolors&#91;i&#93; = doMask&#40;r, rw, rs&#41; | doMask&#40;g, gw, gs&#41; | doMask&#40;b, bw, bs&#41; | doAlpha &#40;alpha, aw, as&#41;;
&#125;

[1] http://www.freelists.org/post/uae/Palet ... d-WinUAE,4

[ceztko]

@DrMcCoy: I'm pleased you proved me to be wrong. After all, my actual wish was to ask for a double check palettes in scummvm. At this point there are various hypothesis:
- scummvm palettes had been always correct, because the dev (quietust?) who grabbed colors using winuae did it in a smart way.
- palettes had been recently fixed using fixed winuae/e-uae.

No matter what is the right one, the important thing is to ensure palettes in scummvm are right.

I'd like to test remaining colors myself. Where are stored scummvm palettes for MI1?

Trying to fix the flower color (see the bug report), would using a 24bit surface help? Let's suppose palettes are stored in 12bit format in scummvm (and it seems the case, now): a correct 12bit->16bit conversion would be better to actually rescale from 24bit->16bit?

[nolange]

@nolange: I realize you opened this bug report to see 2) solved. I wanted to talk with you personally before intervening, but kirben asked me to disccus here, so I did. It seems you are convinced that:
a) is impossible to map amiga native colors, as a 12bit palette is not a subset of a 24bit palette;
b) because of a, accuracy of colors in screenshots taken in a real amiga depend on how a 12bit color is converted in a 24bit color;
c) because of a, accuracy of colors in screenshots taken in a Winuae depend on how Winuae converts a 12bit color in a 24bit color;
d) because of a and b, it's ok to use Winuae as a reference because you trust more Winuae 12bit->24bit conversion;
e) scummvm uses the same 24bit colors that winuae actually (recent versions) displays.

My opinion is that a,d and e are not true:

a) 4096 colors is the OCS theoric palette (even if you could use really the full palette only on HAM mode). According [1]:

(OCS Palette) = 4096 = 2^12 = 2^4 * 2^4 * 2^4 --> 4-4-4 representation
(Truecolor) = 2^24 = 2^8 * 2^8 * 2^8 --> 8-8-8 representation

2^8 / 2^4 = 2^4

Do you want to perform a conversion of the color x = (r,g,b) in a 12bit palette, 4-4-4 representation, to a color y in a 24bit palette, 8-8-8 representation? y = (c * 2^4, g * 2^4, b * 2^4)

The conversion is so easy that it doensn't need any rounding/truncating. Or the conversion is wrong?

The conversion isnt wrong per se, its just that you can map the 12 bit colors to 24 in several ways, eg 0x567 -> 0x506070 like your method, or you could use the full 24bit range by using 0x567 -> 0x556677 (what Fingolfin explained). In fact the WinUAE bugfix you posted did mean they switched from your method (0x506070) to Fingolfin`s method (0x556677).

Whats definitly wrong tough is the assumption that those pictures taken from a real Amiga actually dictate the "right way" to convert to 24bit. If you grab screens from (OCS/ECS) Amiga they are only 12 bit, the only reason you see 24bit values is because the program you used to look at them converted the colors from 12bit

d) As of 26 Sep 2005, (Win)UAE was using a bugged 12bit -> 24bit conversion, as I discovered [2];
e) As far as I know, palette that are used NOW in scummvm were grabbed before 26 Sep 2005, so your affirmatation "all fixed palettes in ScummVM use the "WinUAE colors" may have been true before 26 Sep 2005, but it is no more actual.

I checked the source-code before posting that(ie a couple days ago), all values are multiples of 17, so its the same way WinUAE generates 24bit values.

Why they are different? Because IMO the palette grabbed for scummvm is wrong because a bugged Winuae version was used (and this is the nth times I am repeating this).

Sorry, but I doubt anyone picked the colors from WinUAE, rather extracted from the game itself. Also you used ScummVM 13.1 which hasnt the fix for Guybrush` palette.

To conclude: nolange, is good you want to fix the flower color (I confirm the bug) and be sure Guybrush's face color is ok, but I think I gave you enough hints that palettes used NOW in scummvm need first to be wholly rechecked.
As I told before, these are my recommendations: use e-uae in a 24bit display mode and gimp to grab screenshot. DON'T use Winuae.

Nothing wrong with that, but as two others already pointed out the errors in the lower 4bit arent due to a wrong palette, but to differences how emulators/viewers/framegrabbers convert 12 to 24 bit. 24bit just doesnt exist on nonAGA Amigas and thus you dont have an exact reference

[nolange]

@DrMcCoy: I'm pleased you proved me to be wrong. After all, my actual wish was to ask for a double check palettes in scummvm. At this point there are various hypothesis:
- scummvm palettes had been always correct, because the dev (quietust?) who grabbed colors using winuae did it in a smart way.
- palettes had been recently fixed using fixed winuae/e-uae.

The palette in Monkey Island is broken, I fixed it only in my branch. Even tough I repeat myself - the palette wasnt taken from WinUAE but from the binary in 12bit and I converted it to 24bit.

I'd like to test remaining colors myself. Where are stored scummvm palettes for MI1?

Here is the fixed palette, line 171
Note that this palette is wrong for all games except Monkey, so use my Version for MI and regular ScummVM for everything else.

[ceztko]

Nothing wrong with that, but as two others already pointed out the errors in the lower 4bit arent due to a wrong palette, but to differences how emulators/viewers/framegrabbers convert 12 to 24 bit. 24bit just doesnt exist on nonAGA Amigas and thus you dont have an exact reference

Very good nolange. Please note that I don't claim to be correct, and really I'm pleased you proved me to be wrong. I'm just a tester and I've zero experience with real world C programming and graphics formats: I like seeing people reasoning together . I've just replied DrMcCoy, that said scummvm is using a 16 bit SDL surface internally:

Trying to fix the flower color (see the bug report), would using a 24bit surface help? Let's suppose palettes are stored in 12bit format in scummvm (and it seems the case, now): a correct 12bit->16bit conversion would be better to actually rescale from 24bit->16bit?

[nolange]

...Very good nolange. Please note that I don't claim to be correct, and really I'm pleased you proved me to be wrong. I'm just a tester and I've zero experience with real world C programming and graphics formats: I like seeing people reasoning together . I've just replied DrMcCoy, that said scummvm is using a 16 bit SDL surface internally..

No problem, I dint see your 2 last posts when I wrote my reply.

[ceztko]

Let's suppose palettes are stored in 12bit format in scummvm (and it seems the case, now)

Nope, they aren't (as nolange just pointed me)? At this point, I think is better if scummvm would just store the original 12bit representation of the colours in the source code. Assuming scummvm is using a 16 bit SDL backend surface: do a proper 12bit->16bit conversion. Assuming scummvm is using a 24 bit SDL backend surface: do a proper 12bit->24bit conversion.

[DrMcCoy]

Trying to fix the flower color (see the bug report), would using a 24bit surface help?

No, that 16bit loss of granularity can't be responsible for such a crass color difference. That must be a whole different issue altogether. The one nolange partly fixed in his branch apparently.

[ceztko]

Hmmm....maybe, better handling amiga palettes would lead to have more accurate colours but just in theory. This is a quote from Ron Gilbert's blog.

Back then, the randomly generated forest was cutting edge technology. Disk space was at a premium. Everything had to fit on 5 floppy disks. Sierra would ship games on 8 or 10 floppy disks. That was always a sore point for us.

Maybe the wrong flower colour is not really palette related but is just a bug in the random forest generating algorithm in scummvm?

[LogicDeLuxe]

The sprites use the EGA graphics in the Amiga version while the backgrounds have 32 colors. Now, there are two different versions of the Amiga executable.

One version which has the same palette for the spites as the earlier 16 color SCUMM games on Amiga, which looks ugly in most scenes, as the skin is far too bright. Also some animations like the picture in the mansion falling from the wall and LeChuck shaking the grog machine look really ugly with this palette. Nevertheless, there seems to be quite some people preferring this version.

The other version of the executable uses a palette which looks like real EGA as seen in the screenshots above (http://web.tiscali.it/franzo/quickgrab.005.png), and which ScummVM 0.13.1 is using. This palette fits the background images far better, imho.

The best way for ScummVM to handle this would be supporting both palettes, as both exist and were hardcoded in the original interpreters. Maybe switching between EGA and Amiga in ScummVM setup could be used for this, though it only should affect the sprites but not the backgrounds then.