Mercury: Ultra-fast language models based on diffusion

> Maybe I've just got unlucky in the past, but in most projects I worked on a lot of developer time was wasted on waiting for PRs to go green.

I don't understand this. Developer time is so much more expensive than machine time. Do companies not just double their CI workers after hearing people complain? It's just a throw-more-resources problem. When I was at Google, it was somewhat common for me to debug non-deterministic bugs such as a missing synchronization or fence causing flakiness; and it was common to just launch 10000 copies of the same test on 10000 machines to find perhaps a single digit number of failures. My current employer has a clunkier implementation of the same thing (no UI), but there's also a single command to launch 1000 test workers to run all tests from your own checkout. The goal is to finish testing a 1M loc codebase in no more than five minutes so that you get quick feedback on your changes.

> make builds fully hermetic (so no inter-run caching)

These are orthogonal. You want maximum deterministic CI steps so that you make builds fully hermetic and cache every single thing.

> I don't understand this. Developer time is so much more expensive than machine time. Do companies not just double their CI workers after hearing people complain? It's just a throw-more-resources problem.

I'd personally agree. But this sounds like the kind of thing that, at many companies, could be a real challenge.

Ultimately, you can measure dollars spent on CI workers. It's much harder and less direct to quantify the cost of not having them (until, for instance, people start taking shortcuts with testing and a regression escapes to production).

That kind of asymmetry tends, unless somebody has a strong overriding vision of where the value really comes from, to result in penny pinching on the wrong things.

I was also at Google for years. Places like that are not even close to representative. They can afford to just-throw-more-resources, they get bulk discounts on hardware and they pay top dollar for engineers.

In more common scenarios that represent 95% of the software industry CI budgets are fixed, clusters are sized to be busy most of the time, and you cannot simply launch 10,000 copies of the same test on 10,000 machines. And even despite that these CI clusters can easily burn through the equivalent of several SWE salaries.

> These are orthogonal. You want maximum deterministic CI steps so that you make builds fully hermetic and cache every single thing.

Again, that's how companies like Google do it. In normal companies, build caching isn't always perfectly reliable, and if CI runs suffer flakes due to caching then eventually some engineer is gonna get mad and convince someone else to turn the caching off. Blaze goes to extreme lengths to ensure this doesn't happen, and Google spends extreme sums of money on helping it do that (e.g. porting third party libraries to use Blaze instead of their own build system).

In companies without money printing machines, they sacrifice caching to get determinism and everything ends up slow.

It's more than that. You can measure salaries too, measurement isn't the issue.

The problem is that if you let people spend the companies money without any checks or balances they'll just blow through unlimited amounts of it. That's why companies always have lots of procedures and policies around expense reporting. There's no upper limit to how much money developers will spend on cloud hardware given the chance, as the example above of casually running a test 10,000 times in parallel demonstrates nicely.

CI doesn't require you to fill out an expense report every time you run a PR thank goodness, but there still has to be a way to limit financial liability. Usually companies do start out by doubling cluster sizes a few times, but each time it buys a few months and then the complaints return. After a few rounds of this managers realize that demand is unlimited and start pushing back on always increasing the budget. Devs get annoyed and spend an afternoon on optimizations, suddenly times are good again.

The meme on HN is that developer time is always more expensive than machine time, but I've been on both sides of this and seen how the budgets work out. It's often not true, especially if you use clouds like Azure which are overloaded and expensive, or have plenty of junior devs, and/or teams outside the US where salaries are lower. There's often a lot of low hanging fruit in test times so it can make sense to optimize, even so, huge waste is still the order of the day.

IME it's less of a "throw more resources" problem and more of a "stop using resources in literally the worst way possible"

CI caching is, apparently, extremely difficult. Why spend a couple of hours learning about your CI caches when you can just download and build the same pinned static library a billion times? The server you're downloading from is (of course) someone else's problem and you don't care about wasting their resources either. The power you're burning by running CI for there hours instead of one is also someone else's problem. Compute time? Someone else's problem. Cloud costs? You bet it's someone else's problem.

Sure, some things you don't want to cache. I always do a 100% clean build when cutting a release or merging to master. But for intermediate commits on a feature branch? Literally no reason not to cache builds the exact same way you do on your local machine.

>Do companies not just double their CI workers after hearing people complain?

They do not.

I don't know if it's a matter of justifying management levels, but these discussions are often drawn out and belabored in my experience. By the time you get approval, or even worse, rejected, for asking for more compute (or whatever the ask is), you've spent way more money on the human resource time than you would ever spend on the requested resources.

Developer time is more expensive than machine time, but at most companies it isn't 10000x more expensive. Google is likely an exception because it pays extremely well and has access to very cheap machines.

Even then, there are other factors:

* You might need commercial licenses. It may be very cheap to run open source code 10000x, but guess how much 10000 Questa licenses cost.

* Moores law is dead Amdahl's law very much isn't. Not everything is embarrassingly parallel.

* Some people care about the environment. I worked at a company that spent 200 CPU hours on every single PR (even to fix typos; I failed to convince them they were insane for not using Bazel or similar). That's a not insignificant amount of CO2.

I have never once been refused by a manager or director when I am explicitly asking for cost approval. The only kind of long and drawn out discussions are unproductive technical decision making. Example: the ask of "let's spend an extra $50,000 worth of compute on CI" is quickly approved but "let's locate the newly approved CI resource to a different data center so that we have CI in multiple DCs" solicits debates that can last weeks.

Most of my experience writing concurrent/parallel code in (mainly) Java has been rewriting half-baked stuff that would need a lot of testing with straightforward reliable and reasonably performant code that uses sound and easy-to-use primitives such as Executors (watch out for teardown though), database transactions, atomic database operations, etc. Drink the Kool Aid and mess around with synchronized or actors or Streams or something and you're looking at a world of hurt.

I've written a limited number of systems that needed tests that probe for race conditions by doing something like having 3000 threads run a random workload for 40 seconds. I'm proud of that "SuperHammer" test on a certain level but boy did I hate having to run it with every build.

This is exactly my experience with asking for more compute at work. We have to prepare loads of written justification, come up with alternatives or optimizations (which we already know won't work), etc. and in the end we choose the slow compute and reduced productivity over the bureaucracy.

And when we manage to make a proper request it ends up being rejected anyways as many other teams are asking for the same thing and "the company has limited resources". Duh.

That's solvable with modern cloud offerings - Provision spot instances for a few minutes and shut them down afterwards. Let the cloud provider deal with demand balancing.

I think the real issue is that developers waiting for PRs to go green are taking a coffee break between tasks, not sitting idly getting annoyed. If that's the case you're cutting into rest time and won't get much value out of optimizing this.

Not really, in most small companies/departments, £100k a month is considered a painful cloud bill and adding more EC2 instances to provide cloud runners can add 10% to that easily.

Many companies are strangely reluctant to spend money on hardware for developers. They might refuse to spend $1,000 on a better laptop to be used for the next three years by an employee, whose time costs them that much money in a single afternoon.

No it is not. Senior management often has a barely disguised contempt for engineering and spending money to do a better job. They listen much more to sales complain.

I’m at Google today and even with all the resources, I am absolutely most bottlenecked by the Presubmit TAP and human review latency. Making CLs in the editor takes me a few hours. Getting them in the system takes days and sometimes weeks.

That depends on the company.

I have faced this at each of the $50B in profit companies I have worked at.

> Moores law is dead Amdahl's law

Yes, but the OP specifically is talking about CI for large numbers of pull requests, which should be very parallelizable (I can imagine exceptions, but only with anti-patterns, e.g. if your test pipeline makes some kind of requests to something that itself isn't scalable).

That's been a pet peeve of mine for so long. (Glad my current employer gets me the best 1.5ℓ machine from Dell every few years!)

On the other hand I've seen many overcapitalized pre-launch startups go for months with a $20,000+ AWS bill without thinking about it then suddenly panic about what they're spending; they'd find tens of XXXXL instances spun up doing nothing, S3 buckets full of hundreds of terabytes of temp files that never got cleared out, etc. With basic due diligence they could have gotten that down to $2k a month, somebody obsessive about cost control could have done even better.

Both companies I've worked in recently have been too paranoid about IP to use the cloud for CI.

Anyway I don't see how that solves any of the issues except maybe cost to some degree (but maybe not; cloud is expensive).

Writing testing infrastructure so that you can just double workers and get a corresponding doubling in productivity is non-trivial. Certainly I've never seen anything like Google's testing infrastructure anywhere else I've worked.

Yeah Google's infrastructure is unique because Blaze is tightly integrated with the remote execution workers and can shard testing work across many machines automatically. Most places can't do that so once you have enough hardware that queue depth isn't too big you can't make anything go faster by adding hardware, you can only try to scale vertically or optimize. But if you're using hosted CI SaaS it's often not always easy to get bigger machines, or the bigger machines are superlinear in cost.

Sorta. For CI/CD you can use spot instances and spin them down outside of business hours, so they can end up being cheaper than buying many really beefy machines and amortizing them over the standard depreciation schedule.

Even Google can not buy more old Intel Macs or Pixel 6s or Samsung S20s to increase their testing on those devices (as an example)

Maybe that affects less devs who don't need to test on actual hardware but plenty of apps do. Pretty much anything that touches a GPU driver for example like a game.

I’m currently at google (opinions not representative of my employer’s etc) and this is true for things that run in a data center but it’s a lot harder for things that need to be tested on physical hardware like parts of Android or CrOS.

Presumably the "days and sometimes weeks" thing is entirely down to human review latency?

Were they running CI on their own physical servers under a desk or in a basement somewhere, or renting their own racks in a data center just for CI?

That’s paranoid to the point of lunacy.

Azure for example has “confidential compute” that encrypts even the memory contents of the VM such that even their own engineers can’t access the contents.

As long as you don’t back up the disks and use HTTPS for pulls, I don’t see a realistic business risk.

If a cloud like Azure or AWS got caught stealing competitor code they’d be sued and immediately lose a huge chunk of their customers.

It makes zero business sense to do so.

PS: Microsoft employees have made public comments saying that they refuse to even look at some open source repository to avoid any risk of accidentally “contaminating” their own code with something that has an incompatible license.

You're confusing throughput and latency. Lengthy CI runs increase the latency of developer output, but they don't significantly reduce overall throughput, given a developer will typically be working on multiple things at once, and can just switch tasks while CI is running. The productivity cost of CI is not zero, but it's way, way less than the raw wallclock time spent per run.

Then also factor in that most developer tasks are not even bottlenecked by CI. They are bottlenecked primarily by code review, and secondarily by deployment.

Actually, OP was talking about the throughput of running on a large number of pull requests and the latency of running on a single pull request. The latter is not necessarily parallelizable.

I don't know about Azure's implementation of confidential compute but GCP's version basically essentially relies on AMD SEV-SVP. Historically there have been vulnerabilities that undermine the confidentiality guarantee.

Mandatory XKCD: https://xkcd.com/538/

Nobody's code is that secret, especially not from a vendor like Microsoft.

Unless all development is done with air-gapped machines, realistic development environments are simultaneously exposed to all of the following "leakage risks" because they're using third-party software, almost certainly including a wide range of software from Microsoft:

- Package managers, including compromised or malicious packages.

    Microsoft owns both NuGet and NPM!

- IDEs and their plugins, the latter especially can be a security risk.

    What developer doesn't use Microsoft VS Code these days?

- CLI and local build tools.

- SCM tools such as GitHub Enterprise (Microsoft again!)

- The CI/CD tooling including third-party tools.

- The operating system itself. Microsoft Windows is still a very popular platform, especially in enterprise environments.

- The OS management tools, anti-virus, monitoring, etc...

And on and on.

Unless you live in a total bubble world with USB sticks used to ferry your dependencies into your windowless facility underground, your code is "exposed" to third parties all of the time.

Worrying about possible vulnerabilities in encrypted VMs in a secure cloud facility is missing the real problem that your developers are probably using their home gaming PC for work because it's 10x faster than the garbage you gave them.

Yes, this happens. All the time. You just don't know because you made the perfect the enemy of the good.

Between Github and Copilot, MS has a copy of all of your code.

Yes and no, I'd estimate 1/3 to 1/2 of that is down to test suites are flaky and time-consuming to run. IIRC shortest build I had was 52m for Android Wear iOS app, easily 3 hours for Android.

> ...your developers are probably using their home gaming PC for work because it's 10x faster than the garbage you gave them...

I went from a waiter to startup owner and then acquirer, then working for Google. No formal education, no "real job" till Google, really. I'm not sure even when I was a waiter I had this...laissez-faire? naive?...sense of how corporate computing worked.

That aside, the whole argument stands on "well, other bad things can happen more easily!", which we agree is true, but also, it isn't an argument against it.

From a Chesterson's Fence view, one man's numbskull insistence on not using AWS that must only be due to pointy-haired boss syndrome, is another's valiant self-hosting-that-saved-7 figures. Hard to say from the bleachers, especially with OP making neither claim.

My personal experience: We run over 1.1m test cases to verify every PR that I submit, and there are more test cases that don't get run on every commit and instead get run daily or on-demand.

At that scale getting quick turnaround is a difficult infrastructure problem, especially if you have individual tests that take multiple seconds or suites that take multiple minutes (we do, and it's hard to actually pull the execution time down on all of them).

I've never personally heard "we don't have the budget" or "we don't have enough machines" as answers for why our CI turnaround isn't 5 minutes, and it doesn't seem to me like the answer is just doubling the core count in every situation.

The scenario I work on daily (a custom multi-platform runtime with its own standard library) does by necessity mean that builds and testing are fairly complex though. I wouldn't be surprised if your assertion (just throw more resources at it) holds for more straightforward apps.

> missing the real problem that your developers are probably using their home gaming PC for work because it's 10x faster than the garbage you gave them.

> Yes, this happens. All the time. You just don't know because you made the perfect the enemy of the good.

That only happens in cowboy coding startups.

In places where security matters (e.g. fintech jobs), they just lock down your PC (no admin rights), encrypt the storage and part of your VPN credentials will be on a part of your storage that you can't access.

My last company was unsure about paying $20/mo to get a Copilot license for all the engineers.

Indeed. You'd think Google would test for how well people will cope with boredom, rather than their bait-and-switch interviews that make it seem like you'll be solving l33tcode every evening.

I've seen people not pay for Slack and just deal with disappearing messages and use Skype (back in the day) for group calls.

There are non-IP reasons to go outside the big clouds for CI. Most places I worked over the years had dedicated hardware for at least some CI jobs because otherwise it's too hard to get repeatable performance numbers. At some point you have an outage in production caused by a new build passing tests but having much lower performance, or performance is a feature of the software being sold, and so people decide they need to track perf with repeatable load tests.

The way Azure implements CC unfortunately lowers a lot of the confidentiality. It's not their fault exactly, more like a common side effect of trying to make CC easy to use. You can certainly use their CC to do secure builds but it would require an absolute expert in CC / RA to get it right. I've done design reviews of such proposals before and there's a lot of subtle details.

Length CI runs do reduce throughput, as working around high CI latencies pushes people towards juggling more PRs at once meaning more merge conflicts to deal with, and increases the cost of a build failing transiently.

And context switching isn't free by any means.

Still, if LLM agents keep improving then the bottleneck of waiting on code review won't exist for the agents themselves, there'll just be a stream of always-green branches waiting for someone to review and merge them. CI costs will still matter though.

As a 35 year old waiter with no formal education, who has also spent the majority of his free time the last 25 years either coding or self-studying to further my coding, I am super interested in your life story. While struggling to scrape by has been "awesome", I'm hoping to one day succeed at making tech my livelihood. Do you have a blog or something? lol

But to go back to the topic: are companies that have such a high level of OpSec actually outfitting devs with garbage, enterprise lease, mid-to-low tier laptops? I only have knowledge from a few friends' experiences, but even guys doing relatively non-hardware intensive workloads are given a Dell XPS or MacBook Pro. I would imagine a fintech would know better AND have the funds to allocate for either of those options

Maybe an in-house SWE at a major bank would end up with that level of OpSec on a mediocre fleet laptop, although I'd hope they'd have managers willing to go to bat for them and an IT department that can accommodate provisioning multiple SKUs depending on an employee's actual computational needs.... perhaps I too have a skewed/naive sense of how the corporate computing world works haha

You think people work on a single issue at a time?

Maybe at Google they can afford that, where I worked at some point I was working 2 or 3 projects switching between issues. Of course all projects were the same tech and mostly the same setup, but business logic and tasks were different.

If I have to wait 2-3 hours I have code to review, bug fixes in different places to implement. Even on a single project if you wait 2 hours till your code lands test env and have nothing else to do someone is mismanaging the process.

It is not anymore, because cloud providers noticed it and started to exploit that thinking.

Cloud is cheap on low end servers. You can heave really cheap setup to start as a gateway drug. Once you turn that knob to full speed it is super expensive.

In my experience, fintech companies (including ones that either belong to or own a bank) follow one of two playbooks:

- Issue high-powered laptops that the developers work on directly, then install so many security suites that Visual Studio takes three minutes to launch. The tech stack is too crusty and convoluted to move to anything else like developer VMs without major breakage. - Rely 100% on Entra ID to protect a tech stack that's either 100% Azure or 99% Azure with the remaining 1% being Citrix. You can dial in with anything that can run a Citrix client or a browser modern enough to run the AVD web client. If they could somehow move the client hardware to the Azure cloud, they would.

I don't really associate fintech with a modern, well-implemented tech stack. Well, I suppose moving everything to the cloud is modern but that doesn't mean it's particularly well done.

This feels incredibly weird. I and my team never wait on CI for very long because we just throw more machines at the problem. Supplying a whole team of SE’s with unlimited CI costs us the equivalent of 1/4th SE’s salary. We don’t use anything but Github’s built-in caching, and yeah, that’s the main thing making CI slower right now. Never more than 5 minutes though. We certainly never wait for machines to free up.

>> Making CLs in the editor takes me a few hours. Getting them in the system takes days and sometimes weeks.

You just need an AI agent to shepherd it through the slow process for you while you work on something else!

> In normal companies, build caching isn't always perfectly reliable

In normal companies you often don't have build & task caching to begin with. Heck, people often don't even know how Docker image layer caching works.

Microsoft, Google, or Amazon don't t care about your fintech code. Other fintechs do.

The threat isn't your cloud provider stealing your code, it's your own staff walking out the door with it and either starting their own firm or giving it to a competitor in exchange for a "job" at 2x their previous salary.

I've seen very high security fintech setups first-hand and I've got friends in the industry, including a friend that simply memorised the core algorithms, walked out, rewrote it from scratch in a few years and is making bank right now.

PS: The TV show Severance is the wet dream of many fintech managers.

Yes my comment explicitly states that the cost is not zero

I believe the solution is to run CI locally, and upload some signed proof that CI completed successfully. Running the same tests locally and then on an ~10 times slower CI build always felts like a ridiculous waste of time to me.

I’m all for n boring tech and all, and leveraging the simplest thing that can work. But arguing against streams? You mean reactive streams?

These have helped me replace megabytes of poorly written concurrent/parallel crap with a few lines of stream orchestration. I find it interesting that our experiences diverge so wildly.

Then again, I’ve had to rewrite some terribly clever code, some downright diabolical, so that could be part of it.

I mean the streams library in JDK 8.

Reactive streams are great. I wrote a data transformation toolkit which used reactive streams for the data plane (passing small RDF documents along the pipes) and the Jena rules engine for the control plane (assembling the reactive stream pipeline and tearing it down)

I later worked with some people who built something with a similar architecture -- their system didn't get the same answer every time because they didn't handle teardown properly but that's a common problem and not hard to fix. Lots of people forget to do it, even with Executors.

While not at Google for myself a lot of the CI test failures just become knock on effects from complex interdependent CI components delivering the whole experience. Oops Artifactory or GitHub rate limited you. Oops the SAST checker from some new vendor just never finished. Even if your code passes locally the added complexity of CI can often be fraught with flaky and confusing errors that are intermittent or run afoul based on environmental problems that particular moment you tried.

I am also at Google, and I can corroborate this experience personally and corroborate this based off of comments teammates make to me, in group settings, and in team retrospectives.

There are a lot of technical challenges in maintaining code health in a monorepo with 100k+ active contributors, so teams and individuals get a lot of plausible excuses for kicking the problem down the road, and truly improving code health is not appropriately incentivized. One common occurrence is a broken monorepo, so one just waits until someone fixes the monorepo, and you retry submitting your code again. It's such a common occurrence that people generally do not investigate brokenness, and maybe the monorepo wasn't broken but your code change actually made things even flakier, but no one would be able to distinguish that from a broken monorepo that eventually got fixed when no one bothers to check anymore.

Data center.

Yeah though not for silicon verification because we always run more tests overnight. They get basically 100% utilisation.

Yeah I actually 100% agree. I think even more important is that the IP isn't even that valuable to competitors. Nobody outside China would touch it for legal reasons, and even in China it's just not that useful without the people that wrote it. Especially given how badly most of my colleagues document their code!

Well how do you think they got to $50B profit? /s

Dude, I've worked at Google.

It's an overpaid overglorified boring job (obv. outside all the 'cool' research-y projects).

> I don't understand this. Developer time is so much more expensive than machine time. Do companies not just double their CI workers after hearing people complain?

They don’t, and it’s annoying. Oh some do for sure but it’s the same with developer hardware. I’ve worked places where developers are waiting 4+ minutes for a build to complete when that could be halved or more by better developer hardware but companies are sometimes incredibly “penny wise and pound foolish”.