VMware's in court again. Customer relationships rarely go this wrong

Play stupid games, win stupid prizes.

If you need 40,000 servers to keep your business running (which you don't, your ~3-8 million weekly transactions can be processed on 1 computer, but whatever), hire people that will work on you, and whose paycheck depends on keeping those computers working, to keep those computers working.

Game theory arguments like "they wouldn't screw me over because other people won't want to do business with them" don't work when the other party is trying to maximize quarterly earnings, and their long-term thinking is in the order of ~2 years.

To be fair, and I know nothing about Tesco’s actual stack, a large grocery chain needs to track their contracts with suppliers, track their inventory in each location and in transit, track what goods they want in which locations, understand which larger pallets and big boxes contain which goods, track things prepped in house, and also optimize what to move from where, to where, and when and how. The latter part probably uses some spiffy stack involving something like CPLEX or Gurobi, and it’s not running on their “1 computer” OLTP stack.

That being said, I don’t see what 40k servers is for unless the POS machines are thin clients that use a substantial fraction of a server each.

Most admins are more keen to shift blame than to keep things running. Having another company to point fingers at is more attractive than a proper functioning team.

A long and storied history of nasty surprises has taught me to never underestimate the complexity of an unfamiliar domain.

In that sense I'm surprised it's only 40,000.

Numbers I found vary but Tesco has around 3500 stores in the UK alone alongside other chains they have a hand in. They also have a large online presence, click and collect operations, estates, data collection schemes and a whole logistics network to operate. I'd have actually thought it would be higher than ~11 VMs per store.

If you're doing 10 million transactions per week (which is likely way more than what they're pulling) that's about 16 transactions processed per second. You can add inventory management, payroll management, you can run the company's email server, write all that in JavaScript, and you'll still have room to run a Minecraft server on the same laptop.

My point was not that running all that on one computer is a great idea, just that 40,000 servers for a CRUD application is way past what should be considered reasonable.

But even that's fine. I like computers, you can have 40,000 of them if you want, even if the only reason they exist is some guy's job security. However, you're insane if the guy keeping them running doesn't work for you.

That works out to ~11 computers/licenses per store, which sounds a tad high but also very easy to do if you let new system accrete over time and factor in the need for offline operations and redundancy across regions.

You're suggesting that all Tesco branches should do all the PoS transactions online to a single server?

Tell me you've never designed a system at this scale without telling me you've never designed something at this scale...

The cashier register is probably running on a vmware.

Running it on a single laptop might be an exaggeration, but I can't imagine there's any essential complexity that requires more than a few dozen servers.

No, I did not suggest that, in fact in the very comment you're replying to I said:

  My point was not that running all that on one computer is a great idea...

Regardless, if you want to strawman my passing remark, I'm happy to defend it.

Let's even say my numbers are wildly wrong, and they're processing 100x more transactions than what I claimed (which was already an overestimate). Tell me why you can't process 1600 transactions per second on one computer, especially for a country the size of the UK, where you would expect a ~15ms ping when talking to a server on the other side of the country.

I'd hate to be the lowly, underpaid sysadmin who responded "40,000 servers" when asked the current number of servers, but he meant to respond "4,000 servers". LOL

It seems that you should work for Tesco, they would probably pay you well for reducing 40k VMs to 3.

I would expect outages when talking to a server on the other side of the country, and an outage preventing supermarket customers from checking out or supermarket staff from usefully restocking shelves would be a very expensive mistake.

A good system here will be distributed.

That being said, two servers (for redundancy) physically located at each branch ought to do the trick. Tesco has a bit over 5000 branches [0], so that’s 10k of the 40k VMWare seats right there. Throw in some extra seats so storage and compute can be separated at each location (maybe unnecessary but comes with some benefits) and so that there is still redundancy while a server or two at a site is being re-imaged and 40k seems about in the right ballpark even for a fairly lean implementation.

And, sure, all those on-site servers might be relatively inexpensive industrial units designed to tolerate a toasty, dusty, and occasionally damp closet that looks nothing like a tidy datacenter, but it still makes sense to run something like VMWare on them.

[0] https://www.tescoplc.com/investors/reports-results-and-prese...

You are severely underappreciating how complex a retail organization of Tesco's size is.

I work in this space for a retailer almost the same size as Tesco and when factoring in all the attendant organizations, businesses, and functions it requires, 40k servers does not surprise me at all.

Outside of just the brick-and-mortar stores you have Marketing, Retail, eComm, Merchandising, Strategic Sourcing, FP&A, Finance & Accounting, Asset Protection, Corporate Real Estate, Retail Real Estate, Internal Audit, Supply Chain, Transportation, Business Services, Data Science, etc etc. and IT at every level of those. Each one of these components is large enough to be a medium-to-large sized company in its own right.

You’re ignoring general heavy workloads such as observability. How much telemetry do they gather and analyze for tracking and fraud detection. A quick google on Tesco engineering shows that they process 35k qps against couchbase, and 35 terrabytes of telemetry data per day. They track 150k devices in their ecosystem, which, reading between the lines, would produce the telemetry and require observability, state management, anomaly detection, etc. They have hundreds of thousands of employees using these devices for varying purposes. We’re talking quite a bit of compute, which also requires high availability.

I know nothing about Tesco beyond that quick google search, but I’ve been at several companies where I would read online comments claiming we could reduce our workload to a few servers, and I would think of our tens of thousands of fully loaded machines and roll my eyes.

Yeah, I'm pessimistically sure that there is other stuff, like:

* Checking whether each item scanned has satisfied a logical contract for a discount, some of which may be per-region, per-store, or even per-customer.

* If multiple exclusive coupons or deals are available, resolve the contradiction, preferably in favor of the customer.

* Check if any items or quantities of items require an ID to be shown before proceeding, and record information about the employee authorizing it.

* Update customer "rewards" data and generate any special offers so that you can put it onto their receipt.

And that's not even starting to get into all the other less-customer-synchronous stuff that you still need CPU power somewhere to do. Managing stock levels, orders, deliveries, price changes, anti-"shrinkage", employee shifts, market-research, status and repairs of freezer-units, operational logging and telemetry, every form of reporting/dashboard "strategic insight" stuff beloved by upper management...