Splatt3R: Zero-Shot Gaussian Splatting from Uncalibrated Image Pairs

Think of a single splat as a Gaussian ellipsoid placed in (usually) 3 dimensions, with size, shape, alpha falloff, color (varying both at any point in the ellipsoid, and when viewed from any angle at that point.)

Now think to yourself “Could I approximate, say Kirby with 10 splats? And could I get a GPU to hone in on the best splats to approximate Kirby, maybe using gradient descent?” Then ask yourself “could I get a 4k+ resolution photographic grade 3D scene that included transmissive and reflective behavior using this method?”

If your answer to the second is “obviously!” Then you have a good head for this ML stuff. Somewhat surprisingly (shockingly?) The answer is ‘yes’, and also somewhat surprisingly, you can use ML pipelines and autograd type tech to hone in on what the 18 or so implied variables for a splat should be, and when you have millions of them, they look like AMAZING reconstructions of scenes. And also are incredibly quick to render.

Splats are pretty cool.