*sad update* ESRGAN AI Upscaling (ScummVM fork and tools pulled from github by author)

[criezy]

That comment from Eugene refers to the possibility of using ESRGAN as a real time scaler to upscale the game to a higher resolution.

Another approach discussed in this thread is to use ESRGAN to generate a set of upscaled assets beforehand, and modify ScummVM to render the game directly at higher resolution using those assets (instead of using the original resolution and scaling the display on the fly). In term of development, this is a lot of work that needs to be done separately for each engine, and is unlikely to be integrated in the official ScummVM. And the user also has a lot of work to do as due to copyright reasons the distribution of such assets is not possible (unless the copyright holders agree to it), and thus each user would need to generate their own upscaled assets, for example using ESRGAN on assets extracted from their original copy of the game. But such an approach remain technically possible, even if not very practical, and for example work is being done in a separate fork (https://github.com/MiLO83/scummvmext) for the SCI engine.

Since Curse of Monkey Island is not an SCI game, that work will not help though and somebody would need to do similar work for the SCUMM engine. So don't expect it to be available any time soon, or even ever. It could happen but I would not hold my breath.

[ZTylerDurden]

That comment from Eugene refers to the possibility of using ESRGAN as a real time scaler to upscale the game to a higher resolution.

Another approach discussed in this thread is to use ESRGAN to generate a set of upscaled assets beforehand, and modify ScummVM to render the game directly at higher resolution using those assets (instead of using the original resolution and scaling the display on the fly). In term of development, this is a lot of work that needs to be done separately for each engine, and is unlikely to be integrated in the official ScummVM. And the user also has a lot of work to do as due to copyright reasons the distribution of such assets is not possible (unless the copyright holders agree to it), and thus each user would need to generate their own upscaled assets, for example using ESRGAN on assets extracted from their original copy of the game. But such an approach remain technically possible, even if not very practical, and for example work is being done in a separate fork (https://github.com/MiLO83/scummvmext) for the SCI engine.

Since Curse of Monkey Island is not an SCI game, that work will not help though and somebody would need to do similar work for the SCUMM engine. So don't expect it to be available any time soon, or even ever. It could happen but I would not hold my breath.

Great explanation. Thank you!

Is there a scumm dev tool to extract/dump the game's textures?

[sev]

Is there a scumm dev tool to extract/dump the game's textures?

There are no textures. There are pictures for backgrounds and sprites for objects, all are in a quite bizarre format. But for actors, there is a whole separate language that directs which parts of the actor to draw and at what position and time (see the list of opcodes here). To browse through those you may use tools like ScummEX.


Eugene

[ZTylerDurden]

Is there a scumm dev tool to extract/dump the game's textures?

There are no textures. There are pictures for backgrounds and sprites for objects, all are in a quite bizarre format. But for actors, there is a whole separate language that directs which parts of the actor to draw and at what position and time (see the list of opcodes here). To browse through those you may use tools like ScummEX.


Eugene

Awesome thank you. I'm gonna take a look.

[Kasenshlogga]

So it's been like a almost a year, are there any completed Sierra High Res packs yet?

If there are any I'm not sure where to look for them. It's not like these games have Nexusmods or ModDB pages.

[MusicallyInspired]

I'm working on SQ4 and SQ5 at the moment but it's slow going because I've got a bunch of other things to do. It's a meticulous process because some models don't work well with some backgrounds. So I'm doing a lot of pre and post-upscale pixel modifications to backgrounds and sprites to make sure they come out properly. Also, some sprites are used for background animation so if you upscale these animation frames separately from the background they're meant for then once ingame they'll stick out like a sore thumb because of the different upscalings. So there's things like that to consider. Also, some screens (especially in SQ5) use paletted screens for colour cycling. ScummVM-ext also supports this but the resulting upscaled image has to have the exact same 8-bit palette as the original resource in order for the colour cycling to actually work. That's also a complicated (but doable) process. Ditto for sprites that colour cycle.

I have recreated (not upscaled) the SQ4 and SQ5 GUI interface though.

sq4interface.png (389.78 KiB) Viewed 29396 times

[Kasenshlogga]

I'm working on SQ4 and SQ5 at the moment but it's slow going because I've got a bunch of other things to do. It's a meticulous process because some models don't work well with some backgrounds. So I'm doing a lot of pre and post-upscale pixel modifications to backgrounds and sprites to make sure they come out properly. Also, some sprites are used for background animation so if you upscale these animation frames separately from the background they're meant for then once ingame they'll stick out like a sore thumb because of the different upscalings. So there's things like that to consider. Also, some screens (especially in SQ5) use paletted screens for colour cycling. ScummVM-ext also supports this but the resulting upscaled image has to have the exact same 8-bit palette as the original resource in order for the colour cycling to actually work. That's also a complicated (but doable) process. Ditto for sprites that colour cycle.

I have recreated (not upscaled) the SQ4 and SQ5 GUI interface though.

sq4interface.png

That UI looks so clean, great job!

Yeah, I guess it makes sense that it's a meticulous process, if you want to do the artwork justice and not make it look like you just ran it through an algorithm.

[Blakes7]

I'm working on SQ4 and SQ5 at the moment but it's slow going because I've got a bunch of other things to do. It's a meticulous process because some models don't work well with some backgrounds. So I'm doing a lot of pre and post-upscale pixel modifications to backgrounds and sprites to make sure they come out properly. Also, some sprites are used for background animation so if you upscale these animation frames separately from the background they're meant for then once ingame they'll stick out like a sore thumb because of the different upscalings. So there's things like that to consider. Also, some screens (especially in SQ5) use paletted screens for colour cycling. ScummVM-ext also supports this but the resulting upscaled image has to have the exact same 8-bit palette as the original resource in order for the colour cycling to actually work. That's also a complicated (but doable) process. Ditto for sprites that colour cycle.

I have recreated (not upscaled) the SQ4 and SQ5 GUI interface though.

sq4interface.png

I'm not sure if you are still working on this but any update?

[MusicallyInspired]

I haven't touched it in a while, no. Lots of stuff going on.

[nachazo]

Hi. I had been replaying "Touché" (note: first time using ScummVM) and thought on using Real-ESRGAN AI, much better upscaling for old-painted games than ESRGAN imo. The results are pretty impressive (comparable to MI special editions for example), even just using screenshots as input. The art-style are totally respected and mostly no artifact (except for characters). Take a look:

1: https://cdn.knightlab.com/libs/juxtapos ... 95d9b17862
2: https://cdn.knightlab.com/libs/juxtapos ... 95d9b17862
3: https://cdn.knightlab.com/libs/juxtapos ... 95d9b17862
4: https://cdn.knightlab.com/libs/juxtapos ... 95d9b17862
5: https://cdn.knightlab.com/libs/juxtapos ... 95d9b17862
6: https://cdn.knightlab.com/libs/juxtapos ... 95d9b17862

(NOTE: I edited some results, inpainting characters and removing HUD, as I firstly want use some of them as wallpapers)

Real-ESRGAN (https://github.com/xinntao/Real-ESRGAN) have some trained models available: https://github.com/xinntao/Real-ESRGAN/ ... del_zoo.md

So, come here as I wan't to "investigate" or ask (as I think is fairly impossible because the old-engine of these games...):

- Could we "decompile" for get the original backgrounds/assets? It's possible to obtain it? And maybe "recompile" in the ScummVM structure?
- What "engine" is Touché for example?
- I saw some people that make things like that aren't "replacing original game files", instead are using the "Extra" ScummVM folder. How I could know the "file name" that I must use? Using the original game file name? (Never seen on assets, only audio)
- I saw also people re-mading the soundtrack files (capturing and recording midi) and users may use the "Extra" folder for it (awesome!): https://www.pixelrefresh.com/touche-the ... r-scummvm/
- It's possible to go throught this direction or just a dream?
- "Remade" all game is not an option (oh, imagine fullscreen and UI with mouse...), but using original programming replacing assets...
- I haven't touched much more in the upscaled results. But tried some noise to so-clean spaces and in another games could be edited/fixed not-well colored zones and so on.
- Yeah, some background could need some image editing for small zones.
- Maybe some objects couldn't be worked as are just pixel plain painted by the engine?

So...:
- As I far know, "we could" make a "remake" "easely" for example for Touché. But "decompile-compile" and in what formats and size-limits... are the clue.
- About characters: these could not be directly "upscaled" as the backgrounds. Manual editing must be done.
- About animations: We could improve adding more steps in the animations. No problem manual editing characters helped with interpolation frames (...if the engine supports...).
- About music: Well, it's done.
- About sounds: That's some low quality recorded sounds (i.e. owl in the game beginning), could be replaced with free ones easely.
- About voices: Are clearly in mono. I could stereoize, clean and revamp with some easy editing.

What do you think?

[MusicallyInspired]

Very nice output indeed!

[sev]

The results are indeed impressive.

If I'd attempt to go this route, I wouldn't even bother about "repacking" into the original structures but have my own code path that would ease development and access, e.g. loading from an external specially named PNG file instead of the original background.

Touché game engine is conveniently named 'touche' in the ScummVM sources.

There is an inherent problem with this approach. While you may spend a chunk of your time improving the looks of a game, you can not distribute the results legally unless you get explicit permission from the relevant IP owner (or be an owner). Thus, as the ScummVM project, we will not add relevant code just to support such assets as that could be considered as condoning piracy.

You mention the soundtrack remasters performed by James Woodcock. This is a great example of how to do it properly. Before even starting on a project, James contacts the original IP rights owners and gets their blessing. I understand that he also shows the original authors his work before going public. This is why we happily support his work, which is compatible with the ScummVM out of the box.

In your case, however, no work in this regard has been done. Thus, I would recommend trying to approach the authors, maybe via James, since he could have their contacts. If you succeed, we will even want to give a helping hand with development and support the results of your work. Until that, it stays as your nice personal exercise.

While some people may argue that this is currently a "grey zone" law-wise, I wouldn't be eager to test the waters and put the ScummVM project at potential legal risk.


Eugene

P.S. I have an irrelevant question. How fast is ESRGAN these days? E.g. how long does it take to process a single background image?

[Veda]

Probably a stupid question but I’m going to ask it anyway...

What if only the trained models are distributed, along with some kind of script/utility, so that everyone that owns the game can recreate the upscaled backgrounds and use them in the game?

I don’t think that would be illegal at all. After all, the scripts used for the Monkey Island Talkie Edition do a very similar thing: they extract data from the original files, manipulate those files and repack them.

Why using the sounds/voices from the Special Editions in ScummVM doesn’t break any copyright, but using upscaled background images that can be recreated from the original files would?

Moreover, suppose that ScummVM is modified to use the hi-res backgrounds that can be extracted from the original files of the Special Editions using tools like Monkey Island Explorer by Quick and Easy software. Why should that be any different than supporting the sounds/voices from the Special Editions?

And what about Grim Fandango? Even if it is not progressing, there is a Mod called Grim Fandango deluxe that is supported, and it includes some updated 3D models (shouldn’t this be the same as distributing upscaled sprites/background?) and there is also a mod for widescreen support.

[MusicallyInspired]

MiLO83 did the same thing with some tools he made in conjunction with utilizing Cupscale so it allows you to extract resources from SCI games and upscale them in batch processing.

[nachazo]

The results are indeed impressive.

If I'd attempt to go this route...

Firstly, really recommend to fit Real-ESRGAN, not ESRGAN. The Real-ESRGAN trained model works awesome with handmade painted adventure games. In fact, results with Touché are better than others because that (seems Real-ESRGAN are trained with more anime-painted focus). Other games (older specially, in a couple of tests) have similar results to AdvMame 4x, because the AI can't "recompose comparing another paints" (and maybe not worth "remastering" this way at all).

The processing time: depending the input image size (as the output size is 4x - there are 2x model). With a NVIDIA GTX with CUDA, a small image (i.e. 320px) 2/3 seconds. A bigger one (768 o 1024) like 10 seconds (with my GTX 3070 and).

About the "topic" and owners. Yeah, totally understandable. I thought more in a "modding" way. That's why "first ask the experts".

So, if I understand, the way could be the "extra" folder where the "reworked assets" could be placed, and the "touche" ScummVM, if detects, overload it as a "facade", is it?
Some questions then:
- This way, as is "ScummVM loading" images, we haven't problems with file size/formats? We could use PNGs >4MB for example?
- Also new sounds (not only music) could be used?
- I suposse that this depends on original Touché (using as example), but animations (as for sure are images with steps) could be "enlarged" to add more steps and make more soft animations?
- I also have curiosity about text font: it's a image with letters and then loaded by position? (this will allow to update font also).
- Also about UI and so on, but I think this will need further analysis of original game.

Just for know if all of this worth or not. Or maybe just backgrounds and plain characters and its animation steps.

Anyway I will contact with James (I crossed a pair of tweets with him) and ask about. This is just "get info" and then see what can be done.

Probably a stupid question but I’m going to ask it anyway...

I don't know, sev for sure could answer this. Don't know if "reworked/remaster" fits in a "modding" context, but I understand totally that ScummVM as project think slowly this things. But may, as you need the original game and so on.

Anyway, scripting could be (as MusicallyInspired said) but not direct because for make a good remaster the Real-ESRGAN results may be edited. For example characters. Also this will miss another cool improves as sounds or voices. But as a "simple direct improve", if anyone want it, could be (AI model is open source and you'll need the original name for extract assets). The problem is the same I said: access to the original assets (and "overload" it in ScummVM).

At the moment could't be make in real time (you will get some weird results on characters this way), as you need CUDA cores and takes some seconds, doesn't makes senses versus pre-render.