*sad update* ESRGAN AI Upscaling (ScummVM fork and tools pulled from github by author)

[Bluddy]

So, now that AI upscaling has come to the mainstream with NVIDIA DLSS 2.0, and can run in real-time and enable incredible rendering... I think even if ScummVM doesn't do it, some open source project needs to figure it out.

[Kasenshlogga]

So, now that AI upscaling has come to the mainstream with NVIDIA DLSS 2.0, and can run in real-time and enable incredible rendering... I think even if ScummVM doesn't do it, some open source project needs to figure it out.

Would DLSS even work on 2D content? The way I understand it is they train a model by letting the algorithm look at the game's textures and 3D assets from various angles and distances. 2D content is always pre-baked so it would need a pre-trained general purpose model to generate more detail from the image.

[invwar]

If I understand DLSS and FidelityFX (AMD) correct, its all based on presets made on the high resolution, so instead of really rendering e.g. the 4K, the cards know roughly how the graphic should look like based on the preset and make it look like it is the actual resolution. I highly doubt that this system would work with 2D games as here is no high resolution source which could be analyzed and then used as preset.
At least NVIDIA and AMD now supports integer scaling, which definitely improve full screen on low resolution games. But its just normal scaling, making the picture at least sharp and not blurred, but 640x480 still remains as this.

I don't know either the limitations of ScummVM, but I would expect that theoretically it would be possible to do as Laserschwert suggested. Like replacing the backgrounds and other assets. I would really like to know from a programmer what would happen, if the backgrounds get swapped by higher resolutions.
But what definitely should be feasible is replacing the old FMVs by better quality ones. For the start, a ESRGAN upscaled Movie could be again downscaled in higher fidelity to the normal resolution. As its just a video, I don't expect this would break anything but would increase the quality drastically (back then compression was just not good enough). If this works fine, we could try to use higher resolutions. Of course, it would massively help, if someone would make it possible to support avi (or other container) support for videos like the enhanced audio tracks. So we just drop the video in the folder and this is played.

[Laserschwert]

To all "non-programmers" here. ESRGAN will not come to ScummVM. The reason is simple: it is not real time, and is unable to produce 60 frames per second on modern computers. There are attempts to work around that by lowering the quality of the output, and algorithms like RTSRGAN produce 15 images per second on a single 2.3GHz CPU.

Thus, no, this topic is futile.


Eugene

Even though realtime upscaling with ESRGAN isn't the suggestion here, your proposed 60 frames-per-second are hardly necessary for adventure games. No classic adventure game runs at 60fps, plus most times the backgrounds are mostly static, so those need to be updated just once on load.

By the way, by now I have finished my remaster of the CMI intro cutscene:
https://youtu.be/3KRddnLgFZg

And here's a comparison with the original:
https://youtu.be/QfkJLDD2jCI

While ESRGAN was the primary tool for this, a good amount of manual cleanup went in as well.

[MusicallyInspired]

Nice!

[Bluddy]

Beautiful job! I'd love to have this instead of the intro we have. Is there any way to integrate it?

It's funny how picky one can get. Like, it's a shame the ESRGAN can't get rid of all of the compression artifacts.

[Kasenshlogga]

Beautiful job! I'd love to have this instead of the intro we have. Is there any way to integrate it?

It's funny how picky one can get. Like, it's a shame the ESRGAN can't get rid of all of the compression artifacts.

You tell us, ScummVM dev.

[Jacob_]

Good to see there's still interest in this. Made some more progress on Riven, I've gotten the videos upscaled with a custom EDVR model + DAIN. The compression artifacts were so bad that processing each frame individually with ESRGAN didn't work well. https://streamable.com/czimjh

There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.

[Praetorian]

There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.

Is this your own custom build of ScummVM? Are you using the "--disable-debug --enable-release" options for configure prior to building (make)? Maybe that could help

[MusicallyInspired]

Wow! That video is much smoother! Looks incredible!

[Jacob_]

There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.

Is this your own custom build of ScummVM? Are you using the "--disable-debug --enable-release" options for configure prior to building (make)? Maybe that could help

Yeah this is for a custom build that can load and display higher resolution images and video. Using those options fixes the slowness, thanks!

[Blakes7]

Jim Leonard upscaled Gabriel Knight 2 video from 308x167 playing back at 10fps, and uploaded to YouTube at 720p60
https://www.youtube.com/watch?v=6HJndgXMKYY

This was done back in 2018 so who knows whats possible now.

[Kasenshlogga]

Jim Leonard upscaled Gabriel Knight 2 video from 308x167 playing back at 10fps, and uploaded to YouTube at 720p60
https://www.youtube.com/watch?v=6HJndgXMKYY

This was done back in 2018 so who knows whats possible now.

That's not bad, I didn't even know this was doable back in 2018. With the tools we have available today we might already have everything to remaster the entire game. Once ScummVM supports it, that is.

[MiLO83]

https://youtu.be/pmtRH1hcG50

https://youtu.be/bQ3JFdbWCJ0

https://github.com/MiLO83/scummvmx (Modified SCI Engine)
https://github.com/MiLO83/SCICompanionX (Modified to Batch Dump SCI Graphics [Views] as individual Cels required for mod). SCICompanionX currently dumps backgrounds as .bmp, these need to be Converted to .png. (Remember to somehow Enlarge all graphics exactly 4x.) [ESRGAN for instance.. ]

Not all SCI Games work, this is a W.i.P Alpha demonstration.

- MiLO83

[MusicallyInspired]

You are my new favourite person! ScummVM's first High Resolution Pack capability! Well done.