Performance is a tricky, multi-dimensional thing, so there are many benchmarks that try to map to different workloads. For example, specint is often used for exactly your "single threaded task" benchmark, but if what you work on is numerical computing, you mostly don't care (you want specfp at the least and even that is bad).
Some people seem to really like Coremark these days. Others like specintrate. What kind of application do you care about? I'd guess plenty of folks here can provide a better estimate with that info.
(Asking because Azure supports nested virtualization but only on some machine types. AWS doesn't support nested virtualization at all. Google Cloud seems to support nested virtualization on other machine types.)
Also, why 224 rather than 256?
As for 224, we've always reserved threads on each host for I/O and so on. Figure 2 from the Snap paper [1] is probably the best public reference. We also don't make it clear (on purpose) what size the underlying host processors are, though you can clearly guesstimate pretty easily.
Any particular reason for that limitation, or just "not implemented yet"? (Not asking for product roadmaps, just wondering if there's a specific technical issue that makes it more difficult to support.)
> As for 224, we've always reserved threads on each host for I/O and so on. Figure 2 from the Snap paper [1] is probably the best public reference.
That's helpful, thank you.
Modifying instance group templates is your friend.
At this point Intel literally only makes sense if you have one of those single threaded work loads where it still excels and you absolutely must have the fastest single thread performance.
Wow, half of the comments so far start with this disclosure! Tons of Google people lurking on HN :)
I don't work with Rails anymore but the last Rails app(single threaded, Unicorn) I worked with raw CPU compute was not the bottleneck usually as with most CRUD apps time was mostly spent in I/O. This effect was so pronounced that I had set up most scaling groups on M or R instances as the memory used by the gems was the bottleneck on the number of Rails processes I could spawn in the box without exhausting resources. However I do remember even if CPU was not the bottleneck, moving to a processor with a better single thread performance did improve the median response time at the cost of making the same request throughput costlier.
https://cloud.google.com/compute/all-pricing#n2_machine_type...
Jeez, I didn't realize how expensive cloud compute was. I always wondered why my school still has a datacenter. Having your own servers still makes sense for a lot of orgs.
For example, the N2D instances are basically the price of the N1 instances or even cheaper with committed-use discounts. Given that they provide 39% more performance, should the N1 instances be considered obsolete once the N2D exits beta? I know that there could be workloads that would be better on Intel than AMD, but it seems like there would be little reason to get an N1 instance once the N2D exits beta.
Likewise, the N2D has the basically same sustained-use price as the E2 instances (which only have the performance of N1 instances). What's the point of E2 instances if they're the same price? Shouldn't I be getting a discount given that Google can more efficiently use the resources?
It's great to see the improvements at Google Cloud. I'm glad to see lower-cost, high-performance options available. However, I guess I'm left wondering who is choosing what. I look at the pricing and think, "who would choose an N1 or N2 given the N2D?" Sure, there are people with specific requirements, but it seems like the N2D should be the default in my mind.
This might sound a bit like complaining, but I do love how I can just lookup memory and CPU pricing easily. Rather than having to remember name-mappings, I just choose from one of the families (N1, N2, E2, N2D) and can look at the memory and CPU pricing. It makes it really simple to understand what you're paying. It's just that as more families get added and Google varies how it applies sustained-use and committed-use discounts between the families, it becomes more difficult to choose between them.
For example, if I'm going for a 1-year commitment, should I go with an E2 at $10.03/vCPU or an N2D at $12.65/vCPU. The N2D should provide more performance than the 26% price increase, yes? Why can't I get an EPYC based E-series to really drive down costs?
Again, I want to reiterate that Google Cloud's simpler pricing is great, but complications have crept in. E2 machines don't get sustained-use discounts which means they're really only valuable if you're doing a yearly commitment or non-sustained-use. The only time N1 machines are cheaper is in sustained-use - they're the same price as Intel N2 machines if you're doing a yearly commitment or non-sustained-use. Without more guidance on performance differences between the N2D and N2, why should I ever use N2? I guess this is a bit of rambling to say, "keep an eye on pricing complexity - I don't like spending a lot of time thinking about optimizing costs".
Customers rarely have the time/energy/expertise to continuously reoptimize their cloud usage.
As the name implies, N2 is a newer generation than N1. I don't think Google has announced any official N1 deprecation timeline, but that product line clearly has an expiration date.
The more direct comparison would be Intel's N2 instances, vs. AMD's N2D instances. In that case, N2 instances are likely faster on a per-core basis and support some Intel-specific instructions, whereas N2D instances are substantially less expensive.
> Again, I want to reiterate that Google Cloud's simpler pricing is great, but complications have crept in.
That seems like an unavoidable consequence of maturing as a product offering: more options means more complexity. If Google tried to streamline everything and removed options to keep things simple, they'd have another cohort of users (including myself) screaming that the product doesn't meet their needs.
I suppose a "Help Me Choose" wizard that provides some opinionated guidance can be helpful to onboarding new users, but otherwise, I don't see how Google can win here.
Most performance, most performance per watt, most performance per cost. Also, more performance per thread than high-threadcount intel chips. (Although, some of their low-threadcount Xeons do have an edge on that one.)
Oh, and best memory interface and best IO, too.
Compared to the second-generation Epyc processors that Google is using, the first generation has lower clock speeds, can execute fewer instructions per clock (particularly in terms of floating-point operations), has substantially less cache, and has a more complicated memory topology that can negatively impact the performance of workloads that aren't NUMA-aware.
In short, your experience with AMD in AWS isn't relevant to Google's offerings.
What AMD is doing is really insane in my opinion. I'm not sure if they are pricing their processors low on purpose and/or if they have found a way to manufacture cheaper and/or Intel was screwing consumers with their pricing since they were so dominate.
No matter what, AMD is able to provide something that is measurably better and significantly cheaper than the incumbent, and if the blue ocean strategy holds, they should become the new incumbent in the near future.
Both. AMD uses chiplets for higher yields compared to Intel's huge monolithic processors (HCC, XCC), which lowers costs, and Intel jacked prices up because they had a monopoly.
Another follow-up article: https://www.servethehome.com/amd-epyc-7702p-review-redefinin...
AMD is offering incredible performance on every metric: single threaded, multithreaded, total RAM per socket, PCIe 4.0, power consumption, total performance, total price, performance for price, etc.
Outside of some very niche applications, the only reason someone would choose Intel for servers right now is because "no one ever got fired for choosing Intel."
AMD's Epyc Rome processors are truly excellent, best-in-class processors.
So yes, they figured out how to produce cheaper solutions.
The argument was always "no one can use more than X cores" - but software seems to trail hardware in these examples, not the reverse. When Zen was first released, many of the less expensive 6 core options performed worse than Intel's similarly priced 4 core chips. But when comparing modern software using those old parts, AMD's 6 core offerings tend to hold up better.
It feels like AMD is finally ushering us into an era where being able to take advantage of large amounts of parallelism is going to become important for almost every developer.
For my business' workloads, Threadripper 3 (same gen 2 Zen, same IO chiplet, etc) would likely be a much better fit (and competitive with Intel) if AMD sold it with the same kind of enterprisey guarantees they do for Epyc (ECC, etc). Threadripper 3970x, for example, comes with 32 cores and a base clock of 3.7 GHz. That's a much better fit for us than Epyc 7742 or 7302.
As compared to what, Azure? :)
The benchmarks are not just "distributed compilation" either... that's a very misleading characterization. There was one compilation benchmark for the Linux kernel, and that's the only compilation benchmark I remember seeing.
No one benchmarks nginx because nginx can easily saturate the network card on a server without saturating the processor.
Here's a postgres benchmark: https://openbenchmarking.org/embed.php?i=2002066-VE-XEONEPYC...
Or a rocksdb benchmark: https://openbenchmarking.org/embed.php?i=2002066-VE-XEONEPYC...
MariaDB was a rare win for Intel: https://openbenchmarking.org/embed.php?i=2002066-VE-XEONEPYC...
("rare win" is literally the wording used in the Phoronix article: https://www.phoronix.com/scan.php?page=article&item=linux55-...)
ServeTheHome had access to more comprehensive Intel hardware, so I preferred to link to their articles, but Phoronix saw more of the same stuff.
Intel was thoroughly destroyed in every Linux review of Rome vs Intel's latest that I've seen. Intel can eke out some rare wins when applications are heavily optimized for the nuances of their CPUs, but it's not guaranteed even then.
If you can't be bothered to read articles to understand the answer to the question you asked, then this is my last reply.
Everyone to their own, I am just stating that there is a need.
I didn't read those articles in the last 4 minutes because I read them when they were published. A massively parallel run of 7zip was a really stupid benchmark in August and it remains stupid today.
These other benchmarks are certainly more relevant but none of them jumps out at me as a killer claim. An EPYC 7402 with 50% more cores, drawing 80% more power, and costing 35% more dollars than a Xeon Silver 4216 delivers 24% more pgsql ops per second. What TCO equation do you plug that into? I would describe these results as mixed.
For EPYC, AMD is using nine chips: https://images.anandtech.com/doci/13561/amd_rome-678_678x452...
That's 1x I/O chip (kind of like a router), and 8x chips, each of which has 8x cores on it. Total for 64-cores / 128-threads across 8-compute chips, talking together through a central 1x I/O and Memory chip.
The I/O chip is the biggest for reasons: 1. Its made on a cheaper process. 2. It has worse performance than the compute chips. 3. Its required to be big because driving external I/O requires more power.
So the I/O chip can be made on a cheap / inefficient 14nm process, while the CPUs can be made on a more expensive 7nm process (maximizing clock rates, power-efficiency). The big I/O ports are going to eat up a lot of power regardless of 7nm or 14nm process, so might as well save money here.
Except that they could simplify it without reducing flexibility.
For example, the difference between E-series and N-series is that E-series instances have the whole balancing thing. Instead of being a different instance type, it could be simplified into an option on available types and it would just give you a discount.
Likewise, some of it is about consistency. How much should sustained-usage give you a discount? 20%? 30%? 0%? There seems to be little difference to Google whether sustained-use an E2, N2, N2D, or N1 in terms of their costs and planning and yet the discount varies a lot.
It's not about fewer choices. It's more that the choices aren't internally consistent. N2 instances are supposed to be simply superior to N1 instance, but N1 instances cost the same as N2 instances for 1-year contract, 3-year contract, and on-demand. They're only more expensive for sustained-use which seems odd. Likewise, E2 instances are meant to give you a discount and they do give you a discount for 3 out of the 4 usage scenarios. The point is that there's no real reason for the pricing not to be consistent across the 4 usage scenarios (1-year, 3-year, on-demand, and sustained-use). That's where the complexity creeps in.
It's really easy to look and say, "ok, I have E2, N2D, and N2 instances in ascending price order and I can choose what I want." Except that the pricing doesn't work out consistently.
> N2 instances are likely faster on a per-core basis
Are they meant to be? Google's announcement makes it seem like they should be equivalent: "N2D instances provide savings of up to 13% over comparable N-series instances".
--
The point I'm trying to make isn't that they shouldn't offer choice. It's that the choice should be consistent to be easily understandable. E2 instances should offer a consistent discount. If N2 machines are the same price as N1 machines across 3 usage scenarios, they should be the same price across all 4. When you browse the pricing page, you can get into situations where you start thinking, "ok, the N1 instances are cheaper so do I need the improvements of the N2?" And then you start looking and you're like, "wait, the N2s are the same price....oh, just the same price most of the time." Then you start thinking, "I can certainly deal with the E2's balancing...oh, but it's the same price...well, it's cheaper except for sustained-use".
There doesn't seem to be a reason why sustained-use on N1s should be cheaper for Google than sustained-use on N2s. There doesn't seem to be a reason why sustained-use on E2s offers no discount - especially given that the 1-year E2 price offers the same 37% discount that the N1s offer.
It would be nice to go to the page and say, "I'm going with E2s." However, I go to the page and it's more like, "I'm going with E2s when I am going to do a 1-year commitment, but I'm going with N2Ds when I'm doing sustained-use without a commitment since those are the same price for better hardware with seemingly no reason and the N1s are just equal or more expensive so why don't they just move them to a 'legacy machine types' page". It's the inconsistency in the pricing for seemingly no reason that makes it tough, not the options. The fact that N2Ds are the same monthly price as E2s for sustained-use, but E2s are significantly cheaper in all other scenarios is the type of complexity that's the annoying bit.
EDIT: As an example, E2 instances are 20.7% cheaper on-demand, 20.7% cheaper with 1-year commitment, and 20.7% cheaper with 3-year commitment compared to N2D instances. That's wonderful consistency. Then we look at sustained use and it's 0.9% cheaper with no real explanation why. It's a weird pricing artifact that means that you aren't choosing, "this is the correct machine for the price/performance balance I'm looking for" but rather you're balancing three things: price, performance, and billing scenario.
AMD spent less money on TSMC's research. Apple has been bankrolling TSMC to get the latest and greatest process tech.
AMD reached 7nm not because AMD put the R&D research into it... but because they can ride on the coattails of Apple and TSMC's investments.
--------
TSMC and Apple simultaneously benefit: TSMC can spread the risk of the 7nm process to more companies. Apple still gets first-dibs on the technology (but they only need ~6months worth of factory time to build all the chips they need).
Its more surprising that Intel managed to stay ahead of TSMC / Apple for so long. The economics are kind of against Intel here. The more people working together on process tech, the more efficient the results get.
> These other benchmarks are certainly more relevant but none of them jumps out at me as a killer claim. An EPYC 7402 with 50% more cores, drawing 80% more power, and costing 35% more dollars than a Xeon Silver 4216 delivers 24% more pgsql ops per second. What TCO equation do you plug that into? I would describe these results as mixed.
That's some interesting cherry picking. If I may do some of my own...
- The Epyc 7642 is doing 66% more pg sql ops per second than the Silver 4216, but only using an average of 34% more power than the 4216.
- The Xeon Platinum 8253 is consuming about the same amount of power as the Epyc 7402, costs twice as much, and yet the 7402 is performing 34% faster.
The Xeon Silver 4216 is competitive in this one benchmark, and you declare that the results are "mixed". It gets thoroughly destroyed in tons of other benchmarks.
So, yes, if you will only ever run this specific version of MariaDB on this one server, then it might be a toss up... IF you don't benefit from using PCIe 4.0 to access more (or faster) SSDs, and you don't want to have the option of putting in more RAM.
AMD is consistently better in the overwhelming majority of benchmarks here, especially as you get away from the low end. Saying that Intel has one "toss up" victory in the low end category is not exactly a ringing endorsement to pick Intel here.
Threadripper has official support for ECC. Well, "optional" based off of the motherboard's support: https://www.amd.com/en/chipsets/str40
And just picking a random board: https://www.gigabyte.com/Motherboard/TRX40-AORUS-XTREME-rev-... you'll see it listed:
"Support for ECC Un-buffered DIMM 1Rx8/2Rx8 memory modules"
That it must be un-buffered is an annoying market segmentation thing that limits your max RAM in practice, BUT you can at least get ECC up to 256GB with official support and RAM modules that actually exist.
>Quad-Channel DDR4 ECC Memory Support >With the most memory channels you can get on desktop6, the Ryzen™ Threadripper™ processor can support Workstation Standard DDR4 ECC (Error Checking & Correction Mode) Memory to keep you tight, tuned and perfectly in sync.
https://www.amd.com/en/products/ryzen-threadripper
ECC is also supported in the desktop class CPUs and chipsets.
You can be like Digitalocean and just say "You want a CPU core, you get a CPU core, no guarantee what it'll be". Most enterprises won't buy this. But, I think there's some interesting use-cases where even a hyperscale provider targeting enterprises could (and do) utilize this; not on an EC2-like product, but as the infrastructure for something like Lambda, or to run the massive number of internal workloads necessary to power highly-managed cloud workloads.
Note the again as well: GCE originally had it such that N in N1 meant iNtel, and A1 was for AMD (as Joe said publicly here: https://twitter.com/jbeda/status/1159891645531213824). By the time I joined though, we didn’t see the point of the A1 parts, since the Sandybridge’s smoked them.
This has come up a few times, so I wanted to reiterate that these are the Zen2/Rome parts not the first generation “Naples” parts. We didn’t bother launching Naples for GCE, because (as you can see) Rome is a huge step up.
The challenge here is balancing diverse customer workloads against the processor vendors. Historically, at Google, we just bought a single server variant (basically) because almost all code is expected to care primarily about scale-out environments. That made the GCE decision simple: offer the same hardware we build for Google, at great prices.
The problem is that many customers have workloads and applications that they can’t just change. No amount of rational discounting or incentives makes a 2 GHz processor compete with a 4 GHz processor (so now, for GCE, we buy some speedy cores and call that Compute Optimized). Even more strongly, no amount of “you’re doing it wrong” actually is the right answer for “I have a database on-prem that needs several sockets and several TB of memory” (so, Memory Optimized).
There’s an important reason though that we refer to N1, N2, N2D, and E2 as “General purpose”: we think they’re a good balanced configuration, and they’ll continue to be the right default choice (and we default to these in the console). E2 is more like what we do internally at Google, by abstracting away processor choice, and so on. As a nit to your statement above, E2 does flip between Intel and AMD.
You should choose the right thing for your workloads, primarily subject to the Regions you need them in. We’ll keep trying to push for simplicity in our API and offering, but customers really do have a wide range of needs, which imposes at least some minimum amount of complexity. For too long (probably) we attempted to refuse, because of complexity, both for us and customers. Feel free to ignore it though!
ROME was announced in Later 2018, Released on Aug / September 2019. Why did it take another 6 months to roll out an new instance? When I assume you must have had samples since early 2019.
It's long been the "google way" to try and abstract out compute but it's led to an industry full of people trying to follow in their way and overcomplicating what can be solved on one or two machines.
Yeah, it's that "optional" part that is problematic for ECC in particular. But don't let that be a distraction; there are plenty of other enterprisey features in Epyc that are not present in TR, including registered memory support.
Re: 256GB ECC UDIMM on an 8-socket TR board, that's 32GB a DIMM. I guess you can find 32 GB ECC UDIMMs now, but that's pretty recent and expensive.
Then don't make your example be ECC specifically. It's the only thing you listed, I wasn't "distracted" by it. And I also even commented on the lack of registered memory support, so I don't know why you're repeating that back to me?
E2 is just such an amazing idea that feels like it's going to be under-utilized because it isn't cheaper for the sustained-use case. There doesn't seem to be any reason why E2 would be more expensive (to Google) for sustained-use and not for on-demand or committed.
Google Cloud is really nice, but the inconsistent pricing/discounting between the different types seems odd. Like, I'm running something on N1 right now with sustained-use because there's no incentive for me to switch to E2. It feels a bit wasteful since it doesn't get a lot of traffic and would be the perfect VM to steal resources from. However, I'd only get a discount if I did a 1-year commitment. For Google, I'm tying up resources you could put to better use. E2 instances are usually 30% cheaper which would give me a nice incentive to switch to them, but without the sustained-use discount, N2D and N1 instances become the same price. So, I end up tying up hardware that could be used more efficiently.
I was surprised to discover the other day that one of my VPSs had been upgraded from 1 old Xeon 26XX core to 2 EPYC cores. other stats unmetered 10Gb/s up/down, low latency A'dam location, 2GB RAM, SSD.. it even outperformed my i7-8700T in a single-core openSSL benchmark. most importantly it costs €3/mo
I really can't see google competing with that
224 threads = 112 HT cores = 2 x 56 core CPUs. This is 8 cores short of the 64 core flagship. 8 cores == 1 CCX.
It seems exceedingly unlikely that AMD would produce a Rome CPU with 7 out of 8 CCX in perfect health, but have the 8th CCX completely missing (functionally). It seems more likely that the 8th CCX is there with all 8 cores, and that it is reserved for some other type of service. One possibility could be that there are higher guarantees of side channel protection at the CCX boundary, and google intends to use this for secure internal functions or sell to clients who have side channel sensitivity. Another may be that they simply want the hypervisor to have a very fat budget of 8 cores per socket to work with. Considering the amount of potential IO going on, this might be required in some cases.
https://www.scaleway.com/en/virtual-instances/general-purpos...
Does n2d instance is considered safe from CPU vulnerability and safely enable hyper-threading when used with gVisor?
It's fallacious to think that relying on "n" things is strictly safer than "3" things where n is large. That's not quite true due to the significant complexity increases when dealing with large "n" and accompanying overhead.
For web applications (which I suspect the majority of HN readers work on) then sure, but plenty of realtime or safety critical applications are perfectly ok with three-way redundancy.
Also opens avenues for highly paid consultants to dip their beak and promote your products.
Heck, even a machine like that being down for a week is usually still worth it.
eg stackoverflow only has 1 active DB server, and 1 backup.
More seriously, we're adding multi-node stuff for isolation and multi-GPU for performance. Both are quite different... and useful!
"As for 224, we've always reserved threads on each host for I/O and so on. Figure 2 from the Snap paper [1] is probably the best public reference. We also don't make it clear (on purpose) what size the underlying host processors are, though you can clearly guesstimate pretty easily."
Ie. if there's even one thread reserved on the host, your speculation comes to naught. Sorry.
And anyway if you already have all in place to completely rebuild every SPOF machine from scratch in a few hours, go the extra mile and make it an active/passive cluster, even manually switched, and make the downtime a minutes thing.
I agree not everyone can develop like Google, but it’s wrong to say that “it doesn’t work”
569 Euro equals 613.73 United States Dollar
1. n2-standard-16 (Intel Cascade Lake): https://browser.geekbench.com/v5/cpu/1257619
2. n2d-standard-16 (AMD EPYC): https://browser.geekbench.com/v5/cpu/1257340
3. n1-standard-16 (Intel Skylake): https://browser.geekbench.com/v5/cpu/1257420
Intel Cascade Lake is with a very small lead on 1st place. But AMD is with three-year commitment the cheapest option. Great to have a choice. Thx @cvallejo
Single machines just don't fail that often. I managed a database server for an internal tool and the machine failed once in about 10 years. It was commodity hardware, so I just restored the backups to a spare workstation and it was back up in less than 2 hours. 15 people used this service and they could get some work done, without it, so there was less than 30 person-hours of productivity lost. If I spent 30 hours getting failover &c. working for this system over a 10 year period, it would have been more hours lost for the company than the failure caused.
