←back to thread

Show HN: Real-Time Gaussian Splatting

(github.com)
142 points markisus | 5 comments | 15 May 25 13:26 UTC | HN request time: 0.413s | source

LiveSplat is a system for turning RGBD camera streams into Gaussian splat scenes in real-time. The system works by passing all the RGBD frames into a feed forward neural net that outputs the current scene as Gaussian splats. These splats are then rendered in real-time. I've put together a demo video at the link above.
Show context
sendfoods ◴[15 May 25 13:58 UTC] No.43995165[source]▶
Please excuse my naive question - isn't Gaussian Splatting usually used to create 3D imagery from 2D? How does providing 3D input data make sense in this context?
replies(3): >>43995231 #>>43995323 #>>43996188 #
markisus ◴[15 May 25 14:15 UTC] No.43995323[source]▶
Yes, the normal case uses 2D input, but it can take hours to create the scene. Using the depth channel allows me to create the scene in 33 milliseconds, from scratch, every frame. You could conceptualize this as a compromise between raw pointcloud rendering and fully precomputed Gaussian splat rendering. With pointclouds, you have a lot visual artifacts due to sparsity (low texture information, seeing "through" objects"). With Gaussian splatting, you can transfer a lot more of the 2D texture information into 3D space and render occlusion and view-dependent effects better.
replies(1): >>43995429 #
Retr0id ◴[15 May 25 14:23 UTC] No.43995429[source]▶
How do the view-dependent effects get "discovered" from only a single source camera angle?
replies(1): >>43995549 #
1. markisus ◴[15 May 25 14:36 UTC] No.43995549[source]▶
Actually there are multiple source cameras. The neural net learns to interpolate the source camera colors based on where the virtual camera is. Under the hood it's hard to say exactly what's going on in the mind of the neural net, but I think it's something like "If I'm closer to camera A, take most of the color from camera A."
replies(2): >>43995824 #>>44002891 #
2. ttoinou ◴[15 May 25 15:06 UTC] No.43995824[source]▶
So we’re not sure how it works exactly ?
replies(1): >>43995891 #
3. markisus ◴[15 May 25 15:13 UTC] No.43995891[source]▶
Yup, this is the case for all neural nets.
4. BSVogler ◴[16 May 25 08:05 UTC] No.44002891[source]▶
Gaussian splatting does not use neural nets. It runs an optimizer on the Gaussian splattering parameters. I think in your comment you are talking about Neural Radiance Fields (Nerfs).
replies(1): >>44007474 #
5. Retr0id ◴[16 May 25 16:45 UTC] No.44007474[source]▶
Traditionally you'd use an optimizer, but OP isn't doing it traditionally, which is what makes it interesting. NeRFs work differently.