*sad update* ESRGAN AI Upscaling (ScummVM fork and tools pulled from github by author)
Moderator: ScummVM Team
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
So, now that AI upscaling has come to the mainstream with NVIDIA DLSS 2.0, and can run in real-time and enable incredible rendering... I think even if ScummVM doesn't do it, some open source project needs to figure it out.
-
- Posts: 68
- Joined: Thu Jul 18, 2019 1:31 pm
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
Would DLSS even work on 2D content? The way I understand it is they train a model by letting the algorithm look at the game's textures and 3D assets from various angles and distances. 2D content is always pre-baked so it would need a pre-trained general purpose model to generate more detail from the image.Bluddy wrote: ↑Wed Jul 15, 2020 12:46 am So, now that AI upscaling has come to the mainstream with NVIDIA DLSS 2.0, and can run in real-time and enable incredible rendering... I think even if ScummVM doesn't do it, some open source project needs to figure it out.
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
If I understand DLSS and FidelityFX (AMD) correct, its all based on presets made on the high resolution, so instead of really rendering e.g. the 4K, the cards know roughly how the graphic should look like based on the preset and make it look like it is the actual resolution. I highly doubt that this system would work with 2D games as here is no high resolution source which could be analyzed and then used as preset.
At least NVIDIA and AMD now supports integer scaling, which definitely improve full screen on low resolution games. But its just normal scaling, making the picture at least sharp and not blurred, but 640x480 still remains as this.
I don't know either the limitations of ScummVM, but I would expect that theoretically it would be possible to do as Laserschwert suggested. Like replacing the backgrounds and other assets. I would really like to know from a programmer what would happen, if the backgrounds get swapped by higher resolutions.
But what definitely should be feasible is replacing the old FMVs by better quality ones. For the start, a ESRGAN upscaled Movie could be again downscaled in higher fidelity to the normal resolution. As its just a video, I don't expect this would break anything but would increase the quality drastically (back then compression was just not good enough). If this works fine, we could try to use higher resolutions. Of course, it would massively help, if someone would make it possible to support avi (or other container) support for videos like the enhanced audio tracks. So we just drop the video in the folder and this is played.
At least NVIDIA and AMD now supports integer scaling, which definitely improve full screen on low resolution games. But its just normal scaling, making the picture at least sharp and not blurred, but 640x480 still remains as this.
I don't know either the limitations of ScummVM, but I would expect that theoretically it would be possible to do as Laserschwert suggested. Like replacing the backgrounds and other assets. I would really like to know from a programmer what would happen, if the backgrounds get swapped by higher resolutions.
But what definitely should be feasible is replacing the old FMVs by better quality ones. For the start, a ESRGAN upscaled Movie could be again downscaled in higher fidelity to the normal resolution. As its just a video, I don't expect this would break anything but would increase the quality drastically (back then compression was just not good enough). If this works fine, we could try to use higher resolutions. Of course, it would massively help, if someone would make it possible to support avi (or other container) support for videos like the enhanced audio tracks. So we just drop the video in the folder and this is played.
- Laserschwert
- Posts: 280
- Joined: Mon Mar 06, 2006 11:48 pm
Re: ESRGAN: AI Neural Net Upscaling
Even though realtime upscaling with ESRGAN isn't the suggestion here, your proposed 60 frames-per-second are hardly necessary for adventure games. No classic adventure game runs at 60fps, plus most times the backgrounds are mostly static, so those need to be updated just once on load.sev wrote: ↑Sun Mar 15, 2020 10:30 am To all "non-programmers" here. ESRGAN will not come to ScummVM. The reason is simple: it is not real time, and is unable to produce 60 frames per second on modern computers. There are attempts to work around that by lowering the quality of the output, and algorithms like RTSRGAN produce 15 images per second on a single 2.3GHz CPU.
Thus, no, this topic is futile.
Eugene
By the way, by now I have finished my remaster of the CMI intro cutscene:
https://youtu.be/3KRddnLgFZg
And here's a comparison with the original:
https://youtu.be/QfkJLDD2jCI
While ESRGAN was the primary tool for this, a good amount of manual cleanup went in as well.
- MusicallyInspired
- Posts: 1138
- Joined: Fri Mar 02, 2007 8:03 am
- Location: Manitoba, Canada
- Contact:
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
Beautiful job! I'd love to have this instead of the intro we have. Is there any way to integrate it?
It's funny how picky one can get. Like, it's a shame the ESRGAN can't get rid of all of the compression artifacts.
It's funny how picky one can get. Like, it's a shame the ESRGAN can't get rid of all of the compression artifacts.
-
- Posts: 68
- Joined: Thu Jul 18, 2019 1:31 pm
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
Good to see there's still interest in this. Made some more progress on Riven, I've gotten the videos upscaled with a custom EDVR model + DAIN. The compression artifacts were so bad that processing each frame individually with ESRGAN didn't work well. https://streamable.com/czimjh
There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.
There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.
- Praetorian
- ScummVM Developer
- Posts: 879
- Joined: Tue May 08, 2007 8:54 am
- Location: Greece
- Contact:
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
Is this your own custom build of ScummVM? Are you using the "--disable-debug --enable-release" options for configure prior to building (make)? Maybe that could helpJacob_ wrote: ↑Thu Jul 30, 2020 2:31 am There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.
- MusicallyInspired
- Posts: 1138
- Joined: Fri Mar 02, 2007 8:03 am
- Location: Manitoba, Canada
- Contact:
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
Wow! That video is much smoother! Looks incredible!
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
Yeah this is for a custom build that can load and display higher resolution images and video. Using those options fixes the slowness, thanks!Praetorian wrote: ↑Thu Jul 30, 2020 6:20 amIs this your own custom build of ScummVM? Are you using the "--disable-debug --enable-release" options for configure prior to building (make)? Maybe that could helpJacob_ wrote: ↑Thu Jul 30, 2020 2:31 am There's a problem though, playing 1080p 30fps video in ScummVM gets really slow for me, even on a fast PC (notice how the dome slows down as you approach). Question for the devs: any way to make it run faster? MinGW build, videos are QT/MJPEG codec but that can be changed if another one would be more optimized.
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
Jim Leonard upscaled Gabriel Knight 2 video from 308x167 playing back at 10fps, and uploaded to YouTube at 720p60
https://www.youtube.com/watch?v=6HJndgXMKYY
This was done back in 2018 so who knows whats possible now.
https://www.youtube.com/watch?v=6HJndgXMKYY
This was done back in 2018 so who knows whats possible now.
-
- Posts: 68
- Joined: Thu Jul 18, 2019 1:31 pm
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
That's not bad, I didn't even know this was doable back in 2018. With the tools we have available today we might already have everything to remaster the entire game. Once ScummVM supports it, that is.Blakes7 wrote: ↑Tue Dec 29, 2020 5:16 am Jim Leonard upscaled Gabriel Knight 2 video from 308x167 playing back at 10fps, and uploaded to YouTube at 720p60
https://www.youtube.com/watch?v=6HJndgXMKYY
This was done back in 2018 so who knows whats possible now.
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
https://youtu.be/pmtRH1hcG50
https://youtu.be/bQ3JFdbWCJ0
https://github.com/MiLO83/scummvmx (Modified SCI Engine)
https://github.com/MiLO83/SCICompanionX (Modified to Batch Dump SCI Graphics [Views] as individual Cels required for mod). SCICompanionX currently dumps backgrounds as .bmp, these need to be Converted to .png. (Remember to somehow Enlarge all graphics exactly 4x.) [ESRGAN for instance.. ]
Not all SCI Games work, this is a W.i.P Alpha demonstration.
- MiLO83
https://youtu.be/bQ3JFdbWCJ0
https://github.com/MiLO83/scummvmx (Modified SCI Engine)
https://github.com/MiLO83/SCICompanionX (Modified to Batch Dump SCI Graphics [Views] as individual Cels required for mod). SCICompanionX currently dumps backgrounds as .bmp, these need to be Converted to .png. (Remember to somehow Enlarge all graphics exactly 4x.) [ESRGAN for instance.. ]
Not all SCI Games work, this is a W.i.P Alpha demonstration.
- MiLO83
- MusicallyInspired
- Posts: 1138
- Joined: Fri Mar 02, 2007 8:03 am
- Location: Manitoba, Canada
- Contact:
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
You are my new favourite person! ScummVM's first High Resolution Pack capability! Well done.