←back to thread

Pixar's Render Farm

(twitter.com)
382 points brundolf | 5 comments | 02 Jan 21 19:56 UTC | HN request time: 0.552s | source
Show context
banana_giraffe ◴[02 Jan 21 21:38 UTC] No.25616781[source]▶
One of the things they mentioned briefly in a little documentary on the making of Soul is that all of the animators work on fairly dumb terminals connected to a back end instance.

I can appreciate that working well when people are in the office, but I'm amazed that worked out for them when people moved to work from home. I have trouble getting some of my engineers to have a stable connection stable enough for VS Code's remote mode. I can't imagine trying to use a modern GUI over these connections.

replies(6): >>25616815 #>>25616858 #>>25617057 #>>25617074 #>>25618038 #>>25628067 #
lattalayta ◴[02 Jan 21 22:15 UTC] No.25617074[source]▶
That is correct. It's pretty common for a technical artist to have a 24-32 core machine, with 128 GB of RAM, and a modern GPU. Not to mention that the entirety of the movie is stored on a NFS and can approach many hundreds of terabytes. When you're talking about that amount of power and data, it makes more sense to connect into the on-site datacenter.
replies(1): >>25618185 #
__turbobrew__ ◴[03 Jan 21 00:39 UTC] No.25618185[source]▶
I’m guessing Pixar is using a distributed file system opposed to traditional NFS? Do you have any idea what storage system render farms tend to use?

At my workplace we have a smallish HPC center and ended up moving off of NFS at about 2PB of storage since we were starting to hit the limits of NFS (think 1TB of RAM and 88 cores on a single NFS server).

replies(1): >>25618539 #
1. aprdm ◴[03 Jan 21 01:40 UTC] No.25618539[source]▶
Everywhere I worked has been traditional NFS, and I've seen more than 3 times the figure you quoted working well. Usually you have different mountpoints/vfs`s in different servers for different kinds of files.
replies(2): >>25618878 #>>25643065 #
2. __turbobrew__ ◴[03 Jan 21 02:48 UTC] No.25618878[source]▶
Interesting, maybe the scientific computations we are doing are more I/O intense than render applications? How do studios manage disaster recovery? What happens when a multi petabyte NFS server keels over? Are there tape drive backups? It seems risky to have a such a critical system serviced by only a single node.
replies(2): >>25619450 #>>25622246 #
3. dagmx ◴[03 Jan 21 04:27 UTC] No.25619450[source]▶
They're serviced by multiple nodes and have very strong backup policies.

At rhythm and Hues, you could request footage all the way back from the founding of the studio for example.

CG work is fairly IO intensive for tasks like rendering where you're reading hundreds or even thousands of geometry caches per frame. But for other things, your IO isn't as frequent since it's not about constant r/w as there are long computational or artist time between saves and reads.

4. mprovost ◴[03 Jan 21 15:23 UTC] No.25622246[source]▶
At Weta we divided up the NFS servers into "src" and "dat" - "src" was everything made by artists, and "dat" was the output from the renderwall. We backed up "src" every night, but "dat" was never backed up. Every once in a while there would be some mass deletion event but it was always faster to re-render the lost data than to restore from backups.

Also none of the high end commercial filers are single node - they're all clusters of varying sizes.

5. JustinGarrison ◴[05 Jan 21 07:27 UTC] No.25643065[source]▶
Same here. I had some passing information from Pixar, WDAS, and ILM and they were pretty much all NFS. Lots of NFS caching (avere) and high performance NFS appliances in use.