←back to thread

Pixar's Render Farm

(twitter.com)
382 points brundolf | 3 comments | 02 Jan 21 19:56 UTC | HN request time: 0s | source
Show context
banana_giraffe ◴[02 Jan 21 21:38 UTC] No.25616781[source]▶
One of the things they mentioned briefly in a little documentary on the making of Soul is that all of the animators work on fairly dumb terminals connected to a back end instance.

I can appreciate that working well when people are in the office, but I'm amazed that worked out for them when people moved to work from home. I have trouble getting some of my engineers to have a stable connection stable enough for VS Code's remote mode. I can't imagine trying to use a modern GUI over these connections.

replies(6): >>25616815 #>>25616858 #>>25617057 #>>25617074 #>>25618038 #>>25628067 #
mroche ◴[02 Jan 21 22:12 UTC] No.25617057[source]▶
The entire studio is VDI based (except for the Mac stations, unsure about Windows), utilizing the Teradici PCoIP protocol, 10Zig zero-clients, and (at the time, not sure if they've started testing the graphical agent), Teradici host cards for the workstations.

I was an intern in Pixar systems for 2019 (at Blue Sky now), and we're also using a mix of PCoIP and NoMachine for home users. We finally figured out a quirk with our VPN terminal we sent home with people that was throttling connections, but the experience post-that fix is actually really good. There are a few things that can cause lag (such as moving apps like Chrome/Firefox), but for the most part unless your ISP is introducing problems it's pretty stable. And everyone with a terminal setup has two monitors, either 2*1920x1200 or 1920x1200+2560x1440.

I have a 300Mbps/35Mbps plan (turns into a ~250/35 on VPN) and it's great. We see bandwidth usage ranging from 1Mbps to ~80 on average. The vast majority being sub-20. There are some outliers that end up in mid-100s, but we still need to investigate those.

We did some cross country tests with our sister studio ILM over the summer and was hitting ~70-90ms latency which although not fantastic, was still plenty workable.

replies(2): >>25617917 #>>25618614 #
1. dgrant ◴[03 Jan 21 00:00 UTC] No.25617917[source]▶
Hi. I used to work at Teradici. It was always interesting that Pixar went with VDI because it meant the CPUs that were being used as desktops during the day could be used for rendering at night. Roughly speaking. The economics made a lot of sense. A guy from Pixar came to Teradici and gave a talk all about it. Amazing stuff.

Interesting contrast with other companies that switched to VDI where it made very little sense. VMware + server racks + zero clients compared to desktops never made economic sense, at the time. But oftent here is some other factor that tips things in VDI's favour.

replies(1): >>25618231 #
2. mroche ◴[03 Jan 21 00:47 UTC] No.25618231[source]▶
Yep, all of their workstations were dual socket servers, where each socket was a workstation VM with PCIe passthrough, and each getting their own hostcard+GPU. Each VM had dedicated memory, but no ownership of the cores they were pinned to, so overnight if the 'workstations' were idle, another VM (also with dedicated memory) would spin up (the other VMs would be backgrounded) and consume the available cores and add itself to the render farm. An artist could then log in and suspend the job to get their performance back (I believe this was one of the reasons behind the checkpointing feature in RenderMan).

The Teradici stuff was great, and from an admin perspective having everything located in the DC made maintenance SO much better. Switching over to VDI is a long term goal for us at Blue Sky as well, but it'll take a lot more time and planning.

replies(1): >>25620189 #
3. a_e_k ◴[03 Jan 21 07:20 UTC] No.25620189[source]▶
That's one reason for the checkpoint feature, yes, but there are others. A few years back (Dory-era), I participated in a talk at SIGGRAPH '15 about some of them:

https://dl.acm.org/doi/abs/10.1145/2775280.2792573

http://eastfarthing.com/publications/checkpoint.pdf