*sad update* ESRGAN AI Upscaling (ScummVM fork and tools pulled from github by author)
Moderator: ScummVM Team
- Laserschwert
- Posts: 280
- Joined: Mon Mar 06, 2006 11:48 pm
Re: ESRGAN: AI Neural Net Upscaling
Right now all of this is not more than tossing around ideas anyway. I'm no developer, so there needs to be a dev who's actually interested in working on stuff like this.
This includes swapping out the original backgrounds either with cleaned up ones (at the same resolution, but a higher color depth) or even upscaled ones at a higher resolutions. I don't know if Scumm games can be made resolution independent... cursor and click coordinates would probably be trivially easy to "scale", but stuff like walk-boxes or other layers not so much. Being able to swap out (or "override") assets that get loaded would make modding these games a lot easier. Still, I figured replacing the videos would require considerably less work than all that (which is still a lot of work, I guess ).
This includes swapping out the original backgrounds either with cleaned up ones (at the same resolution, but a higher color depth) or even upscaled ones at a higher resolutions. I don't know if Scumm games can be made resolution independent... cursor and click coordinates would probably be trivially easy to "scale", but stuff like walk-boxes or other layers not so much. Being able to swap out (or "override") assets that get loaded would make modding these games a lot easier. Still, I figured replacing the videos would require considerably less work than all that (which is still a lot of work, I guess ).
-
- Posts: 68
- Joined: Thu Jul 18, 2019 1:31 pm
Re: ESRGAN: AI Neural Net Upscaling
I would actually pay money if somebody ever worked on such a feature for ScummVM.Laserschwert wrote: ↑Wed Jul 17, 2019 5:54 pm Right now all of this is not more than tossing around ideas anyway. I'm no developer, so there needs to be a dev who's actually interested in working on stuff like this.
This includes swapping out the original backgrounds either with cleaned up ones (at the same resolution, but a higher color depth) or even upscaled ones at a higher resolutions. I don't know if Scumm games can be made resolution independent... cursor and click coordinates would probably be trivially easy to "scale", but stuff like walk-boxes or other layers not so much. Being able to swap out (or "override") assets that get loaded would make modding these games a lot easier. Still, I figured replacing the videos would require considerably less work than all that (which is still a lot of work, I guess ).
Re: ESRGAN: AI Neural Net Upscaling
I really want to see support for this and the topic seams dead, I'm considering implementing the necessary support myself (And while i'm at it a shader engine [though lower priority] which would probably be compatible with the Arch Slang format as i'd created a similar engine years ago).
If there are other people willing to do the graphic side we should be in good standing, i could do the graphics myself but have limited time so would take a lot longer.
I'm also thinking we have 3 versions of the graphical updates
an ESRGAN pack which is all games upscaled via ESRGAN to 1080/4K (i'd be interested to see ESRGAN multiple pass results to get the lower res games to 4K)
an original pack which is all the original art paintings converted at high resolution using original palettes
and a hybrid pack which is original where possible and ESRGAN for everything else
Personally I'd love to see Gabriel Knight ESRGAN'd up as I've always wanted a high res version and have no desire to play the remake
If there are other people willing to do the graphic side we should be in good standing, i could do the graphics myself but have limited time so would take a lot longer.
I'm also thinking we have 3 versions of the graphical updates
an ESRGAN pack which is all games upscaled via ESRGAN to 1080/4K (i'd be interested to see ESRGAN multiple pass results to get the lower res games to 4K)
an original pack which is all the original art paintings converted at high resolution using original palettes
and a hybrid pack which is original where possible and ESRGAN for everything else
Personally I'd love to see Gabriel Knight ESRGAN'd up as I've always wanted a high res version and have no desire to play the remake
- Praetorian
- ScummVM Developer
- Posts: 879
- Joined: Tue May 08, 2007 8:54 am
- Location: Greece
- Contact:
Re: ESRGAN: AI Neural Net Upscaling
There's been some very recent discussion on the ScummVM discord server (#offtopic channel) about supporting such upscale like Laserschwert's work for Monkey Island 3.CyberAxe wrote: ↑Tue Mar 03, 2020 10:52 am I really want to see support for this and the topic seams dead, I'm considering implementing the necessary support myself (And while i'm at it a shader engine [though lower priority] which would probably be compatible with the Arch Slang format as i'd created a similar engine years ago).
If there are other people willing to do the graphic side we should be in good standing, i could do the graphics myself but have limited time so would take a lot longer.
I'm also thinking we have 3 versions of the graphical updates
an ESRGAN pack which is all games upscaled via ESRGAN to 1080/4K (i'd be interested to see ESRGAN multiple pass results to get the lower res games to 4K)
an original pack which is all the original art paintings converted at high resolution using original palettes
and a hybrid pack which is original where possible and ESRGAN for everything else
Personally I'd love to see Gabriel Knight ESRGAN'd up as I've always wanted a high res version and have no desire to play the remake
https://twitter.com/Laserschwert_/statu ... 0986010625
https://twitter.com/Laserschwert_/statu ... 7822681088
ScummVM Discord server link:
https://discord.gg/ReGSkVs
- MusicallyInspired
- Posts: 1138
- Joined: Fri Mar 02, 2007 8:03 am
- Location: Manitoba, Canada
- Contact:
Re: ESRGAN: AI Neural Net Upscaling
I'd love to see something come of this. Since I lost my computer I haven't been able to upscale without my GPU. But I've recently figured out how to do it via CPU instead but it takes way longer. Been doing some experiments with the plethora of upscaling models out there now intended for different types of games and art and the results are interesting. Still gotta try out training new models though.
Upscaling raw images seems to not be the best way to go as there are usually some visual errors. But doing some pre and post edits would work nicely. Manga109, for example, doesn't like hard pixel staircase edges for contrasting colours and preserves the staircasing effect so it's necessary to manually use a blur tool so that when upscaling the result looks smoother. Different things like that.
Upscaling raw images seems to not be the best way to go as there are usually some visual errors. But doing some pre and post edits would work nicely. Manga109, for example, doesn't like hard pixel staircase edges for contrasting colours and preserves the staircasing effect so it's necessary to manually use a blur tool so that when upscaling the result looks smoother. Different things like that.
Re: ESRGAN: AI Neural Net Upscaling
Note that as lovely as you can make the backgrounds, ESRGAN doesn't work on the small, pixelated characters. This means the best you can do is to have some filtering on the main characters, and the results will look very jarring. The alternative is to redo all the character sprites by hand.
- MusicallyInspired
- Posts: 1138
- Joined: Fri Mar 02, 2007 8:03 am
- Location: Manitoba, Canada
- Contact:
Re: ESRGAN: AI Neural Net Upscaling
Yeah. But even so.
-
- Posts: 68
- Joined: Thu Jul 18, 2019 1:31 pm
Re: ESRGAN: AI Neural Net Upscaling
I think even with the character sprites untouched it would be a significant upgrade in most games.Bluddy wrote: ↑Fri Mar 13, 2020 1:34 pm Note that as lovely as you can make the backgrounds, ESRGAN doesn't work on the small, pixelated characters. This means the best you can do is to have some filtering on the main characters, and the results will look very jarring. The alternative is to redo all the character sprites by hand.
Re: ESRGAN: AI Neural Net Upscaling
To all "non-programmers" here. ESRGAN will not come to ScummVM. The reason is simple: it is not real time, and is unable to produce 60 frames per second on modern computers. There are attempts to work around that by lowering the quality of the output, and algorithms like RTSRGAN produce 15 images per second on a single 2.3GHz CPU.
Thus, no, this topic is futile.
Eugene
Thus, no, this topic is futile.
Eugene
- MusicallyInspired
- Posts: 1138
- Joined: Fri Mar 02, 2007 8:03 am
- Location: Manitoba, Canada
- Contact:
Re: ESRGAN: AI Neural Net Upscaling
Eugene, nobody is saying ESRGAN should come to ScummVM. We're merely talking about the possibility of hi res replacement art assets that can be loaded in on the fly with ESRGAN being used outside the game to prerender images. Like a "high resolution pack." Similar to the high quality music packs that have been made in the past for various ScummVM games. Obviously real-time neural net upscaling won't be a feasible concept for quite a number of years yet.sev wrote: ↑Sun Mar 15, 2020 10:30 am To all "non-programmers" here. ESRGAN will not come to ScummVM. The reason is simple: it is not real time, and is unable to produce 60 frames per second on modern computers. There are attempts to work around that by lowering the quality of the output, and algorithms like RTSRGAN produce 15 images per second on a single 2.3GHz CPU.
Thus, no, this topic is futile.
-
- Posts: 68
- Joined: Thu Jul 18, 2019 1:31 pm
Re: ESRGAN: AI Neural Net Upscaling
Exactly and ResidualVM already supports this, there's a really beautiful The Longest Journey HD mod that utilizes ESRGAN for all it's backgrounds.MusicallyInspired wrote: ↑Sun Mar 15, 2020 1:06 pm Eugene, nobody is saying ESRGAN should come to ScummVM. We're merely talking about the possibility of hi res replacement art assets that can be loaded in on the fly with ESRGAN being used outside the game to prerender images. Like a "high resolution pack." Similar to the high quality music packs that have been made in the past for various ScummVM games. Obviously real-time neural net upscaling won't be a feasible concept for quite a number of years yet.
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
Laserschwert, this is incredible! Just removing the dither alone is beautiful. Is there somewhere I could download the dithered and final versions somewhere (the two different versions)? I know there's an IMGUR link, but the original PNGs would be preferable. Thanks!
-
- Posts: 68
- Joined: Thu Jul 18, 2019 1:31 pm
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
Tried some AI frame interpolation with DAIN: https://imgur.com/a/HBiCP72
Now if there was a way to use this on individual sprites + AI upscaling that would be something.
Now if there was a way to use this on individual sprites + AI upscaling that would be something.
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
I've been working on ESRGAN upscaling for Riven (code). It involves a lot of coding since any references to pixel coordinates or dimensions have to be multiplied by the upscaling factor. And then there's the trouble of dealing with different resolutions since all of the numbers are hard-coded, seems like the only option is going to be setting it to something like 1920x1080 and hoping that works for everyone.
Video upscaling is another problem, and the main reason I haven't released anything yet. I trained a custom model for EDVR (same authors as ESRGAN) and it was much better than I expected at dealing with the compression artifacts, but the output was smooth and not very detailed compared to the images. Hopefully a new, improved technique will be published soon.
Video upscaling is another problem, and the main reason I haven't released anything yet. I trained a custom model for EDVR (same authors as ESRGAN) and it was much better than I expected at dealing with the compression artifacts, but the output was smooth and not very detailed compared to the images. Hopefully a new, improved technique will be published soon.
- MusicallyInspired
- Posts: 1138
- Joined: Fri Mar 02, 2007 8:03 am
- Location: Manitoba, Canada
- Contact:
Re: ESRGAN: AI Neural Net Upscaling (For HiRes Packs?)
Beautiful work on Riven!