*sad update* ESRGAN AI Upscaling (ScummVM fork and tools pulled from github by author)

[Robot_Maker20]

Can this be applied to preprocess low-res FMV cutscenes? Maybe even ones with line-skipping like the Z-vision games?

[Ze_PilOt]

Can this be applied to preprocess low-res FMV cutscenes? Maybe even ones with line-skipping like the Z-vision games?

- The GAN need to be trained with frames where half of the line is missing (it's a question of preparing the training set properly)
- The GAN need to be temporally coherent - That's still an issue but feasible.

So, I would say yes... The main problem would be to handle compression artefacts. You will need to train your GAN with everykind of artefacted videos.
That depend of the compression factor & technique (ie. some FMV are compressing red very much, some a bit less), so no magic bullet for every game I'm afraid.

[Laserschwert]

I took another stab at the CoMI backgrounds and used ESRGAN for dither removal first and upscaling after that. The results are even better now:
https://t.co/fGScl1VzL7

And here's a comparison showing the before -> dither removal -> upscaling: https://t.co/BsC3fW85qg

There's quite a lot of ESRGAN models in this WIKI: https://github.com/alsa64/AI-wiki/wiki/Model-Database

[MusicallyInspired]

The scene is really coming along! Sadly, without me. My desktop PC mobo died and I'm stuck on my laptop without a great GPU. Those results look fantastic! Who needs a special edition with this capability!.....if ScummVM gets a "texture pack/asset swap" feature, that is.

[BeleG]

This looks absolutely wonderful. I am sure there are all kinds of technical issues preventing use of the upscaled backgrounds without significant changes to the code, but would it be possible to use the undithered ones in the engine (e.g. no colour palette support issues)? They are already a significant improvement, are 100% faithful to the original material, and are unlikely to clash with the 640x480 animations (which would be a big problem for the upscaled backgrounds).

[Laserschwert]

I agree, having the undithered backgrounds in the game would be a great first step. Any ScummVM devs reading this?

[Laserschwert]

I've posted many more comparisons of the CMI backgrounds here:

https://www.reddit.com/r/GameUpscale/co ... ey_island/

[MusicallyInspired]

Man that's good!

[Laserschwert]

I've finally managed training my own models for ESRGAN, and right now I'm working on a model to clean up the compression artifacts in the "Curse of Monkey Island" cutscenes. Here are a few comparison shots: https://imgsli.com/NDYzNQ/28/29 (more comparisons can be selected from the drop-downs)

And here's a first test in motion (cleaned up and upscaled 2x): https://www.youtube.com/watch?v=4sqje_OFfi0

Would it be doable to implement loading and playing of external cutscenes (like for example in MP4 format) in CoMI?

[criezy]

Would it be doable to implement loading and playing of external cutscenes (like for example in MP4 format) in CoMI?

Possibly. I am not really familiar with the SCUMM engine, but generally speaking the engines in ScummVM work at the resolution of the original game (for example for rendering the graphics and handling mouse events) and don't know anything about the resolution at which the backend ends up displaying the game. The backends makes all the graphics scaling, and converts the mouse position between the screen/window coordinates and the game coordinates.

This means that to render something at a resolution higher than the original game resolution, the engine would actually need to work at that higher resolution. For cutscenes it might not be too hard to implement as the engine could switch resolution to use a higher resolution only when playing the cutscene (for example the zvision engine does that when playing Zork GI with the high resolution movies from the DVD version of the game), and if there is no interaction it might not need to convert mouse position back to the lower resolution that the game script would expect.

But since I am not sure how videos are handled in the SCUMM engine, I can't comment on whether this is doable or not in that specific engine.

[MusicallyInspired]

Beautiful!

[Laserschwert]

But since I am not sure how videos are handled in the SCUMM engine, I can't comment on whether this is doable or not in that specific engine.

Didn't the "Broken Sword" games use specifically re-compressed cutscenes in older ScummVM versions, because the original Smacker format wasn't supported? Maybe that's a lead...

[criezy]

Yes, they did (and we can still uses those recompressed mpeg videos), but they were at the same resolution as the original videos and the rest of the game.
And the Broken Sword games also use a completely different engine, so this is not really relevant for the SCUMM games.

[Laserschwert]

I see. Well, technically using MP4 files at the original 640x480 resolution of CoMI would already allow for much better looking cutscenes, because compression wouldn't be an issue.

[criezy]

Regarding the specific codec, we don't have anything that uses mpeg4 yet in ScummVM so that would be some additional work to support that. But theora is already being used by a few engines, and would thus require less work for us (we already have a decoder for it in ScummVM).