It's been over a decade and we're still stuck with 2D maps and boxy untextured buildings.
It use neural network techniques, but it's not strictly using NN.
it do the same result as Nerf from google in 30min, nvidia result in 7minute. It can achieve more than 100fps if you let it train longer.
https://www.inria.fr/fr/3d-gaussian-splatting-vision-ordinat...
One of the guys working on this is Federico Ponchio. His 2008 PhD thesis, which provided the core insight for Unreal Engine's Nanite, is referenced at bottom.
You barely ever need to look at 3D photogrammetry buildings for anything and there aren't many questions it answers outside of curiosity.
I do wonder if they could integrate street view images into it better.
Also, is it bad that the first thing I thought of was that commanders in the Ukraine war could use this? E.g.: stitch together the video streams from thousands of drones to build up an up-to-date view of the battlefield?
I'm guessing that adding physics, collision-detection etc. on top of this is non-trivial compared to using a mesh?
But I feel like for stuff like tree foliage (where maybe you don't care about collisions?), this would be really awesome, given the limitations of polygons. + also just any like background scenery, stuff out of the player's reach.
The good news is that splats are really simple once they’ve been generated. Maybe simpler than triangles depending on how you look at it. It’s just a matter of doing the work to set up new tools and pipelines.
These scenes come with real world lighting baked in. This is great because it looks amazing, it's 100% correct, far better than the lighting computed by any game engine or even movie-quality offline ray tracer. This is a big part of why they look so good! But it's also a curse. Games need to be interactive. When things move, lighting changes. Even something as simple as opening a door can have a profound effect on lighting. Anything that moves changes the lighting on itself and everything around it. Let alone moving actual lights around, changing the time of day, etc.
There's absolutely no way to change the baked-in lighting in one of these captures in a high quality way. I've seen several papers that attempt it and the results all suck. It's not the fault of the researchers, it's a very hard problem. There are two main issues:
One, in order to perfectly re-light a scene you first have to de-light it, that is, compute the lighting-independent BRDF of every surface. The capture itself doesn't even contain enough information to do this in an unambiguous way. You can't know for sure how a surface would react under different lighting conditions than were present in the pictures that made up the original scan. Maybe in theory you can guess well enough in most cases and extrapolate, and AI can likely help a lot here, but in practice we are far away from good quality so far.
Two, given the BRDF of all surfaces and a set of new lights, you have to apply the new lighting to the scene. Real-time solutions for lighting are very approximate and won't be anywhere near the quality of the lighting in the original scan. So you'll lose some of that photorealistic quality when you do this, even if your BRDFs are perfect (they won't be). It will end up looking like regular game graphics instead of the picture-perfect scans you want. If you try to blend the new lighting with the original lighting, the boundaries will probably be obvious. You're competing with perfection! Even offline rendering would struggle to match the quality of the baked-in lighting in these captures.
To me the ultimate solution needs to involve AI. Analytically relighting everything perfectly is infeasible, but AI can likely do approximate lighting that looks more plausible in most cases, especially when trying to match captured natural lighting. I'm not sure exactly how it will work, but AI is already being used in rendering and its use will only increase.
People should stop basing all of this new research on proprietary software, when we have open source implementations [1][2].
[1] gsplat: https://github.com/nerfstudio-project/gsplat [2] opensplat: https://github.com/pierotofy/opensplat
Looks like this: https://i.imgur.com/wcCJmbd.png
But yes, I know what you are getting at. This would normally be done via a software/shader pipeline rather than a GPU's polygonal process.
And when you zoom right into streets, storefronts and stuff are barely visible because they haven't properly integrated street level imagery.
Time to switch I'd say...
Polygons are a poor fit, especially for trees and windows and stuff that needs to be semitransparent/fluffy.
I suspect the gaussians will compress better, and give better visual quality for a given amount of data downloaded and GPU VRAM. (the current polygon model uses absolutely loads of both, leading to a poor experience for those without the highest end computers and fast internet connections).
Do you know if anyone has tried this? Or otherwise, what're the best current attempts at solving it?
The other problem is you can only update them in sunny weather. So SAR is a lot more useful because it can see through clouds.
It's a pretty basic "free for non-commercial use, contact us for commercial use" license.
And the Inria rasterizer is not proprietary either. It's non-commercial open source with the option to purchase a commercial license.
These are perfectly reasonable tech stacks for research projects to build off of. If you have an issue with the license, implement it yourself based on the papers (which all outline the necessary details to do so).
It’s almost like for a pure ML solution you would need a nerf (or similar) which is conditioned on the entire (dynamically changing) scene geometry and lighting positions?
Graphics isn’t my area of ML though, so I’m sure there’s a lot of nuance that I don’t appreciate.