Google's Liquid Cooling

It’s very odd when mainframes (S/3x0, Cray, yadda yadda) have been extensively water-cooled for over 50 years, and super-dense HPC data centers have used liquid cooling for at least 20, to hear Google-scale data center design compared to PC hobbyist rigs. Selective amnesia + laughably off-target point of comparison.

Hyper-scale data centers normally need not be concerned with power density, and their designers might avoid density because of the problems it causes. Arguably modern HPC clusters that are still concerned about density are probably misguided. But when it comes to ML workloads, putting everything physically close together starts to bring benefits in terms of interconnectivity.

From the article:

> Liquid cooling is a familiar concept to PC enthusiasts, and has a long history in enterprise compute as well.

And the trend in data centers was to move towards more passive cooling at the individual servers and hotter operating temperatures for a while. This is interesting because it reverses that trend a lot, and possibly because of the per-row cooling.

This abundance of marketing (not necessarily this blog post) is happening because of all the environmental chatter about AI and data centers recently.

Google wants you to know it recycles its water. It's free points.

Edit: to clarify, normal social media is being flooded with stories about AI energy and water usage. Google isn't greenwashing, they're simply showing how things work and getting good press for something they already do.

Google uses plenty of water in cooling towers and chillers. Sure, water cooling loops to the server may reduce the amount a little bit compared to fans, but this isn't "recycling" in any normal sense.

It's mostly a play for density.

We've basically been watching Google gradually re-discover all the tricks of supercomputing (and other high performance areas) over the past 10+ years. For a long time, websearch and ads were the two main drivers of Google's datacenter architecture, along with services like storage and jobs like mapreduce. I would describe the approach as "horizontal scaling with statistical multiplexing for load balancing".

Those style of jobs worked well but as Google has realized it has more high performance computing with unique workload characteristics that are mission-critical (https://cloud.google.com/blog/topics/systems/the-fifth-epoch...) their infrastructure has had to undergo a lot of evolution to adapt to that.

Google PR has always been full of "look we discovered something important and new and everybody should do it", often for things that were effectively solved using that approach a long time ago. MapReduce is a great example of that- Google certainly didn't invent the concepts of Map or Reduce, or even the idea of using those for doing high throughput computing (and the shuffle phase of MapReduce is more "interesting" from a high performance computing perspective than mapping or reducing anyway).

It's not so surprising when considering googles history coming from inexpensive commodity hardware. It's pretty similar to how it took decades for x86 servers and operating systems to gain mainframe functionality like virtualisation.

https://blog.codinghorror.com/building-a-computer-the-google...

The environmental impact of water usage seems way overblown to me.

Last year, U.S. data centers consumed 17 billion gallons of water. Which sounds like a lot, but the US as a whole uses 300 billion gallons of water every day. Water is not a scarce resource in much of the country.

LOLWUT? Hyperscalers and HPC data centers have been very concerned about power and thermal density for decades IME. If you're just running web sites or VM farms, sure keep racks cooler and more power efficient. But for those that deeply care about performance, distance drives latency. That drives a huge demand to "pack things closely" and that drives thermal density up, up, up. "Hot enough to roast a pig" was a fintech data center meme of 20 years go, back at 20kW+ racks. Today you're not really cooking until you get north of 30kW.

Personally, I agree. That said, I think it might be worth considering the impact of water usage in local areas that are relatively water constrained.

As somebody that worked on Google data centers after coming from a high performance computing world I can categorically say that Google is not “re-learning” old technology. In the early days (when I was there) they focused heavily on moving from thinking of computers to thinking of compute units. This is where containers and self contained data centers came from. This was actually a joke inside of Google because it failed but was copied by all the other vendors for years after Google had given up on it. They then moved to stop thinking about cooling as something that happens within a server case to something that happens to a whole facility. This was the first major leap forward where they moved from cooling the facility and pushing conditioned air in to cooling the air immediately behind the server.

Liquid cooling at Google scale is different than mainframes as well. Mainframes needed to move heat from the core out to the edges of the server where traditional data center cooling would transfer it away to be conditioned. Google liquid cooling is moving the heat completely outside of the building while it’s still liquid. That’s never been done before as far as I am aware. Not at this scale at least.

Name a hyper-scale data center where the size and shape suggests that power density ever entered the conversation.

To be clear all talk of water usage has been focusing on local usage which isn't well represented by national numbers.

[bri3d pointed out that I missed an element of this. There is a transfer between rack level and machine level coolant which makes this far less novel than I had initially understood. See their direct comment to this]

I posed this further down in a reply-to-a-reply but I should call it out a little closer to the top: The innovation here is not “we are using water for cooling”. The innovation here is that they are direct cooling the servers with chillers that are outside of the facility. Most mainframes will use water cooling to get the heat from the core out to the edges where traditional where it can be picked up by the typical heatsink/cooling fans. Even home PCs do this by moving the heat to a reservoir that can be more effectively cooled.

What Google is doing is using the huge chillers that would normally be cooling the air in the facility to cool water which is directly pumped into every server. The return water is then cooled in the chiller tower. This eliminates ANY air based transfer besides the chiller tower. This is one being done a server or a rack.. its being done on the whole data center all at once.

I am super curious how they handle things like chiller maintenance or pump failures. I am sure they have redundancy but the system for that has to be super impressive because it can’t be offline long before you experience hardware failure!

[Edit: It was pointed out in another comment that AWS is doing this as well and honestly their pictures make it way clearer what is happening: https://www.aboutamazon.com/news/aws/aws-liquid-cooling-data...]

So every time they plug in a server they also plug in water lines?

It's possible it never made it into production; but when I was helping to commission a 4 rack "supercomputer" circa 2010 we used APC's in-row cooling (which did glycol exchange to the outside but still maintains the hot/cold aisle) and I distinctly remember reading a whitepaper about racks with built in water cooling and the problems with pressure loss, dripless connectors, and corrosion. I no longer recall if the direct cooling loop exited the building or just cycled in the rack to an adjacent secondary heat exchanger. (And I don't remember if it was an APC whitepaper or some other integrator.)

There's also all the fun experiments with dunking the whole server into oil, but I'll give you that again I've only seen setups described with secondary cooling loops - probably because of corrosion and wanting to avoid contaminants.

As to MapReduce, I think you're fundamentally mistaken. You can talk about map and reduce in the lambda calculus sense of the term, but in terms of high performance distributed calculations, MapReduce was definitely invented at Google (by Jeff Dean and Sanjay Ghemawat in 2004).

Looks like it. New server means power, internet, and water.

[I am not a current Google Employee so my understanding of this is based on externally written articles and “leap of faith” guestimation]

Yes. A supply and return line along with power. Though if I had to guess how its setup this would be done with some super slick “it just works” kind of mount that lets them just slide the case in and lock it in place. When I was there almost all hardware replacement was made downright trivial so it could just be more or less slide in place and walk away.

> cooling is moving the heat completely outside of the building while it’s still liquid.

We have been doing this for decades, it's how refrigerants work.

The part that is new is not having an air-interface in the middle of the cycle.

Water isn't the only material being looked at, mostly because high pressure PtC (Push to Connect) fittings, and monitoring/sensor hardware has evolved. If a coolant is more expensive but leaks don't destroy equipment, and can be quickly isolated then it becomes a cost/accounting question.

"From the core to the edges of the server"—what does that even mean?

Unless Google has discovered a way to directly transfer heat to the aethereal plane, nothing they’re doing is new. Mainframes were moving chip and module heat entirely outside the building decades ago. Immersion cooling? Chip, module, board, rack, line, and facility-level work? Rear-door and hybrid strategies? Integrated thermal management sensors and controls? Done. Done. Done. Done. Richard Chu, Roger Schmidt, and company were executing all these strategies at scale long before Google even existed.

I remember reading somewhere that they don't operate at the level of servers; if one dies they leave it in place until they're ready to replace the whole rack. Don't know if that's true now, though.

It does sound like connections do involve water lines though. As they are isolating different water circuits, in theory they could have a dry connection between heat exchanger plates, or one made through thermal paste. It doesn't sound like they're doing that though.

This was before I was born, so I'm hardly an expert, but I've heard of feeding IBM mainframes chilled water. A quick check of wikipedia found some mention of the idea: https://en.wikipedia.org/wiki/IBM_3090

> The part that is new is not having an air-interface in the middle of the cycle.

I wasn’t clear when I was writing but this was the point I was trying to make. Heat from the chip is transferred in the same medium all the way from the chip to the exterior chiller without intermediate transfers to a new medium.

> MapReduce is a great example of that- Google certainly didn't invent the concepts of Map or Reduce, or even the idea of using those for doing high throughput computing (and the shuffle phase of MapReduce is more "interesting" from a high performance computing perspective than mapping or reducing anyway).

The “Map” in MapReduce does not originally stand for the map operation, it comes from the concept of “a map” (or, I guess, a multimap). MapReduce descends from “the ripper”, an older system that mostly did per-element processing, but wasn't very robust or flexible. I believe the map operation was called “Filter()” at the time, and reduce also was called something else. Eventually things were cleaned up and renamed into Map() and Reduce() (and much more complexity was added, such as combiners), in a sort of backnaming.

It may be tangential, but it's not like the MapReduce authors started with “aha, we can use functional programming here”; it's more like the concept fell out. The fundamental contribution of MapReduce is not to invent lambda calculus, but to show that with enough violence (and you should know there was a lot of violence in there!), you can actually make a robust distributed system that appears simple to the users.

just like humans.

Not quite. Google brilliantly rebranded the work of John McCarthy, C.A.R. Hoare, Guy Steele, _et al_ from 1960 ff. e.g. https://dl.acm.org/doi/10.1145/367177.367199

Dean, Ghemawat, and Google at large deserve credit not for inventing map and reduce—those were already canonical in programming languages and parallel algorithm theory—but for reframing them in the early 2000s against the reality of extraordinarily large, scale-out distributed networks.

Earlier takes on these primitives had been about generalizing symbolic computation or squeezing algorithms into environments of extreme resource scarcity. The 2004 MapReduce paper was also about scarcity—but scarcity redefined, at the scale of global workloads and thousands of commodity machines. That reframing was the true innovation.

When our mainframe in 1978 sprung a leak in its water cooling jacket it took down the main east/west node on IBMs internal network at the time. :-). But that was definitely a different chilling mechanism than the types Google uses.

Much of the Google use of liquid chillers was protected behind NDAs as part of its "hidden advantage" with respect to the rest of the world. It was the secret behind really low PUE numbers.

Maybe similar to a gasoline hose breakaway

https://www.opwglobal.com/products/us/retail-fueling-product...

Why make up such an absurd grievance to get fake-outraged about? Nowhere in the article does it say that Google claims to have invented liquid cooling. In fact, nowhere does it say they claim any part of this is a new invention.

But the point of this kind of paper is typically not what is new, it's what combination of previously known and novel techniques have been found to work well at massive scale over a timespan of years.

I believe Map in MapReduce stood for "map" function, not a multimap- I've never heard or read otherwise (maps can operate over lists of items, not just map types). That's consistent both with the original mapreduce paper: """Our abstraction is inspired by the map and reduce primitives present in Lisp and many other functional languages. We realized that most of our computations involved applying a map operation to each logical “record” in our input in order to compute a set of intermediate key/value pairs, and then applying a reduce operation to all the values that shared the same key, in order to combine the derived data appropriately"""

and with the internal usage of the program (I only started in 2008, but spoke to Jeff extensively about the history of MR as part of Google's early infra) where the map function can be fed with recordio (list containers) or sstable (map containers).

As for the ripper, if you have any links to that (rather than internal google lore), I'd love to hear about it. Jeff described the early infrastructure as being very brittle.

The claim is that Google has larger pipes that go all the way out of the building. While mainframes have short pipes that go only to a heat exchanger on the end of the hack.

IMO, it's not a big difference. There are probably many details more noteworthy than this. And yeah, mainframes are that way because the vendor only creates them up to the hack-level, while Google has the "vendor" design the entire datacenter. Supercomputers have had single-vendor datacenters for decades too, and have been using large pipes for a while too.

CERN was doing the equivalent of MapReduce before Google existed.

Definitely not true for these workloads. TPUs are interconnected, one dying makes the whole cluster significantly less useful.

Interestingly, entire supercomputers have been decommissioned [1] due to faulty quick disconnects causing water spray.

So you can get a single, blind-mating connector combining power, data and water - but you might not want to :)

[1] https://gsaauctions.gov/auctions/preview/282996

It has not been true for a LONG time. That was part of Google early “compute unit” strategy that involved things like sealed containers and such. Turns out that’s not super efficient or useful because you leave large swaths of hardware idle.

In my day we had software that would “drain” a machine and release it to hardware ops to swap the hardware on. This could be a drive, memory, CPU or a motherboard. If it was even slightly complicated they would ship it to Mountain View for diagnostic and repair. But every machine was expected to be cycled to get it working as fast as possible.

We did a disk upgrade on a whole datacenter that involved switching from 1TB to 2TB disks or something like that (I am dating myself) and total downtime was so important they hired temporary workers to work nights to get the swap done as quickly as possible. If I remember correctly that was part of the “holy cow gmail is out of space!” chaos though, so added urgency.

Maybe we can declutter things if they get PWoE(power and water over ethernet) or just a USB-W standard.

You can see the male quick disconnect fittings for the liquid cooling at each corner of the server in this photo:

https://substackcdn.com/image/fetch/$s_!8aMm!,f_auto,q_auto:...

Looks like the power connector is in the centre. I'm not sure if backplane connectors are covered up by orange plugs?

It worked for MONIAC.

Got a link? HEP data is really about triggers and widely distributed screening that's application specific AFAIK.

I don't think this comment is accurate based on the article, although you cite personal experience elsewhere so maybe your project wasn't the one that's documented here?

> What Google is doing is using the huge chillers that would normally be cooling the air in the facility to cool water which is directly pumped into every server.

From the article:

> CDUs exchange heat between coolant liquid and the facility-level water supply.

Also, I know from attaching them at some point that plenty of mainframes used this exact same approach (water to water exchange with facility water), not water to air to water like you describe in this comment and others, so I think you may have just not had experience there? https://www.electronics-cooling.com/2005/08/liquid-cooling-i... contains a diagram in Figure 1 of this exact CDU architecture, which it claims was in use in mainframes dating back to 1965 (!).

I also don't think "This eliminates ANY air based transfer besides the chiller tower." is strictly true; looking at the photo of the sled in the article, there are fans. The TPUs are cooled by the liquid loop but the ancillaries are still air cooled. This is typical for water cooling systems in my experience; while I wouldn't be surprised to be wrong (it sure would be more efficient, I'd think!), I've never seen a water cooling system which successfully works without forced air, because there are just too many ancillary components of varying shapes to successfully design a PCB-waterblock combination which does not also demand forced air cooling.

CPC is in this market too https://www.cpcworldwide.com/Liquid-Cooling/Products/Blind-M...

Non-spill fluid quick disconnects are established tech in industries like medical, chemical processing, beverage dispensing, and hydraulic power, so there are plenty of design concepts to draw on.

My main reference is the head of computing at CERN, who explained this to me. He gave some early examples of ROOT (https://en.wikipedia.org/wiki/ROOT) using parallel processing of the ROOT equivalent of SSTables.

Google almost certainly uses evaporative cooling towers, it’s almost twice as efficient as air cooled chillers by themselves.

> and with the internal usage of the program (I only started in 2008, but spoke to Jeff extensively about the history of MR as part of Google's early infra) where the map function can be fed with recordio (list containers) or sstable (map containers).

I worked on the MapReduce team for a while (coincidentally, around 2008), together with Marián Dvorský, who wrote up a great little history of this. I don't think it was ever made public, though.

> As for the ripper, if you have any links to that (rather than internal google lore), I'd love to hear about it. Jeff described the early infrastructure as being very brittle.

I believe it's all internal, unfortunately.

> > CDUs exchange heat between coolant liquid and the facility-level water supply.

Oh interesting I missed that when I went through in the first pass. (I think I space bared to pass the image and managed to skip the entire paragraph in between the two images so that’s on me.

I was running off an informal discussion I had with a hardware ops person several years ago where he mentioned a push to unify cooling and eliminate thermal transfer points since they were one of the major elements of inefficiency in modern cooling solutions. By missing that as I browsed through it I think I leaned too heavily on my assumptions without realizing it!

Also, not all chips can be liquid cooled so there will always be an element of air cooling so the fans and stuff are still there for the “everything else” cases and I doubt anybody will really eliminate that effectively. The comment you quoted was mostly directed towards the idea that Cray-1 had liquid cooling, it did, but it transferred to air outside of the server which was an extremely common model for most older mainframe setups. It was rare for the heat to be kept liquid along the whole path.

The parent poster is just either extremely confidently wrong or talking about a very different project from the one in the linked article - here's an article from 2005 with Figure 1 dating from (according to the article) 1965 (!!) showing the same CDU architecture shown in the chipsandcheese article: https://www.electronics-cooling.com/2005/08/liquid-cooling-i...

I do think Google must be doing something right, as their quoted PUE numbers are very strong, but nothing about what's in the linked chipsandcheese article seems groundbreaking at all architecturally, just strong micro-optimization. The article talks a lot about good plate/thermal interface design, good water flow management, use of active flow control valves, and a ton of iteration at scale to find the optimal CDU-to-hardware ratio, but at the end of the day it's the same exact thing in the diagram from 1965.

it's not the center one, it's the side ones. Center seems to be a power supply.

Having to pre chill water (via a refrigeration cycle) is radically less efficient than being able to collect and then disperse heat. It generates considerably more heat ahead of time, to deliver the chilled water. This mode of gathering the heat and sending it out, dealing with the heat after it is produced rather than in advance, should be much more energy efficient.

I don't know what surprises me about it so much, but having these rack-sized CDU heat-exchangers was quite a surprise, quite novel to me. Having a relatively small closed loop versus one big loop that has to go outside seems like a very big tradeoff, with a somewhat material and space intensive demand (a rack with 6x CDUs), but the fine grained control does seem obviously sweet to have. I wish there were a little more justification for the use of heat exchangers!

The way water is distributed within the server is also pretty amazing, with each server having it's own "bus bar" of water, and each chip having it's own active electro-mechanical valve to control it's specific water flow. The TPUv3 design where cooling happens serially, each chip in sequence getting hotter and hotter water seems common-ish, where-as with TPUv4 there's a fully parallel and controllable design.

Also the switch from lidded chips to bare chips, with a cold plate that comes down to just above, channeling water is one of those very detailed fine-grained optimizations that is just so sweet.

Yeah. There are metrics for how water stressed a community is. We know where data centers have been built, and we know how these water metrics have changed over time.

The correct metric is something like, what's the probability that the launch of data center in a location results in nearby communities to drop significantly in these water metrics.

> We've basically been watching Google gradually re-discover all the tricks of supercomputing (and other high performance areas)

Google isn't claiming to have invented water cooling. This article recaps their talk at Hot Chips where they showed some details of their implementation.

Data center cooling is also a different beast than supercomputer cooling. Datacenter cooling operates at a larger scale and has different needs for maintenance operations like maintenance cycles.

There are also some interesting notes in there about new things Google is doing, like the direct-die cooling.

Water cooling is a big field. Data center operations is a big field. There is interesting content in the article.

And a 12V battery.

I think that's a little unfair.

Building out an idea for a bespoke supercomputer and making an iteration of that idea that applies to globally scaled workloads is a different thing. They are building computing factories, as is Amazon, MS, Facebook, etc.

That said, PR is PR, and companies always crow about something for their own reasons.

Great, now I’ll have to figure out if my USB Hi-Flow cable is slowing down my USB Full-Flow drink bottle refilling.

It's not. Full flow is less than hi-flow... Your bottle might slow down other fills on the bus though :p

It's interesting because I've never seen mainframes do water to water (though I'm sure that was possible).

The only ones I've ever seen do water to compressor (then gas to the outdoor condenser, obviously)

The CDUs are essentially just passive water to water heat exchangers with some fancy electronics attached. You want to run a different chemical mix outside to the chillers as you do on the internal loop, it also helps regulate flow/pressure and leak detection with auto cutoff is all fairly essential.

Running direct on facility water would made day to day operations and maintenance a total pain.

Most ES/9000 series and earlier stuff had water to water SKUs. I remember seeing an installation with one that exchanged heat into a facility loop which went through another water to water exchanger to a pond-fountain chiller system.

Starting with S/390 G4 they did a weird thing where the internals were cooled by refrigeration but the standard SKUs actually had the condenser in the bottom of the cabinet and they required raised floor cooling.

They brought water to air back with the later zSeries, but the standard SKUs mimicked the S/390 strategy with raised floor. I guess you could buy a z196 or a ec12 with a water to water cabinet but I too have never seen one.

    > not all chips can be
    > liquid cooled.

Why not? It's just a heatsink except with water running through cavities within it, instead of a fan sitting on top of the heatsink.

> super-dense HPC data centers have used liquid cooling for at least 20

Hasn't this just been for things like rack doors and such?

In the last ~two generations of servers it seems like now there's finally DLC (direct liquid cooling) into the actual servers themselves (similar to the article). Intel kind of forced that one on us, with their high-end SKUs. This has been a pain becuase it doesn't fit into legacy datacenters as easily as the door/rack-based systems.

I won't say which server vendor it is, but I've put in more than one service ticket for leaking coolant bags (they hang them on the racks).

One of the biggest problems with water cooling, especially on boards that weren’t designed for it, can be passive components which don’t usually have a heatsink and therefore don’t offer a good surface for a water block, but end up in a thermal design which requires airflow - resistors and FETs are common culprits here. Commodity assemblies are also a big problem, with SFPs being a huge pain point in designs I’ve seen.

The problem is often exacerbated on PCBs designed for air cooling where the clearance between water cooled and air cooled components is not high enough to fit a water block. Usually the solution when design allows is to segment these components into a separate air cooled portion of the design, which is what Google look to have done on these TPU sleds (the last ~third of the assembly looks like it’s actively air cooled by the usual array of rackmount fans).

Glycol is cheap and safe, but it has lower specific heat capacity and higher viscosity. So that's why water is still being used.

The next step is probably evaporative cooling, with liquid coolant ("freon") pumped to individual racks.

Do we know if other hyperscalers also use liquid chillers to achieve very low PUE values? I think I saw photos from xAI's new data center and there was liquid cooling.

Not sure what Google specifically is using here, but PG25 (25% propylene glycol, 75% water) is somewhat common in data center applications. The glycol takes care of avoiding microbial growth.

It seems like a very minor problem next to the huge super obvious one, electricity use.

    > part of the “holy cow gmail is out of space!” chaos

This sounds like an interesting story. Can you share more details.

It has been done. See how the leonardo supercomputer is cooled. Closed circuit water that is passively cooled completely outside the building: https://www.youtube.com/watch?v=mckX_R0GeRY

> they are direct cooling the servers with chillers that are outside of the facility

That is exactly what the Cray Y-MP EL that I worked with in the 90s/2000s did.

Indeed, and the problem is once you've committed to fans and liquid cooling you can reduce the complexity and plate size massively by just cooling the big wins (CPU/GPU). I've actually seen setups where they only cold plate the GPU and leave the CPU and it's entire motherboard on air cooling.

Messy.

I wonder if you could just put a conventional heatsink in there to cool the air inside the box?

You would have a liquid block on the CPU but you'd also have a heat sink on top that transfers heat from the air to the coolant block, working in reverse compared to normal air cooling heatsinks. The temperature difference would cause passive air circulation and the liquid cooling would now cool both the CPU and the air in the box, without fans.

Seems like something someone would have thought about and tested already though.

Related, I vaguely recall a concept from a Tesla charger for semis that proposed a charging cable with active coolant flowing through it as well to keep the wire cooled.

edit: https://www.teslarati.com/tesla-liquid-cooled-supercharger-c...

If you're blasting enough air around to cool a 600W GPU, you don't care if your GPU's power connector dissipates 10W under certain circumstances - the massive airflow will take care of it.

Take that airflow away and you have to be a good deal more careful with your connector selection, quality control and usability or you'll risk melted connectors.

Water-cooling connectors and cables isn't common, outside of things like 250kW EV chargers.

I'd love to work in a datacenter at that scale sometime, it sounds like it's like working in a warehouse where you get a list of orders, servers to remove and pick up, but at the scales of the Googles et al, that's hundreds of server replacements a day, and production lines of new servers being built and existing ones being repaired or decommissioned.

It's a fascinating industry, but only in my head as the only info you get about it is carefully polished articles and the occasional anecdote on HN, which is also carefully polished due to NDAs.

My impression is a lot of modern data centres are total-power-constrained, but not really space-constrained.

At least, if you look at them on streetview a lot of them seem to be in the middle of nowhere, surrounded by miles of undeveloped scrubland. If Google's Henderson, NV data centre [1] needed more space they could simply buy out the adjacent car wrecking yard or pet crematorium, or reconsider the gigantic gatehouse and vast expanses of beige gravel.

Even in Belgium, with its higher population density [2] the car park is bigger than the data centre.

The layout makes me think they were told they could only have a certain amount of power, but essentially as much land as they needed. So they're concerned about power and thermals, but maybe not about power and thermal density so much.

[1] https://www.google.com/maps/place/Google+Data+Center+-+Hende... [2] https://www.google.com/maps/place/Google+Data+Center/@55.557...

Once again IT comes full circle.

SPMD was definitely a thing before MapReduce.

I don’t know of any Co-lo types that did it, Microsoft had some creative ideas about things like that but I don’t know if they brought any to fruition. I suspect there are still opportunities there.

This is why I risk jokes on HN. Normally they're frowned upon but it's worth it to find interesting tech that meets the eccentric idea.

yea I totally missed the CDU. I thought this was a project I had talked with a hardware person about a few years ago where there was no intermediate transfer and when I read the article I completely missed the section between the images. Rack level water cooling is interesting and I am sure they are doing some really cool bits on it but it’s not as revolutionary as a zero transfer system that I had thought they were talking through. I updated the comment to call out my error and reduce my excitement. =/

[I am still annoyed at how many people are dismissive of Google’s datacenter work simply because “severs have been water cooled before” which completely misses the point of datacenter level cooling. I also learned that AWS is doing this already, along with some elements of OVH] =)

Not really practical it wouldn't transfer much energy at all. Let's say that your coolant comes in at 30 degrees c, well if your air is 40° and you've got no fans you can do the maths but it may as well be 0.

It's exactly those kind of problems (though these systems will be SXM like).

I was imagining the coolant comes in at a lower temp like 20 and maybe keeps the air from going above 40.

It doesn't have to do that much, but maybe you're right. I'm sure they'd be doing this if it was practical, being able to onit thousands of fans would probably save a pretty penny both on hardware and electricity.