What happens to latency if service time is cut in half (2022)

    In mathematical queueing theory, Little's law (also result, theorem, lemma, or formula[1][2]) is a theorem by John Little which states that the long-term average number L of customers in a stationary system is equal to the long-term average effective arrival rate λ multiplied by the average time W that a customer spends in the system. Expressed algebraically the law is

    L=λW

    The relationship is not influenced by the arrival process distribution, the service distribution, the service order, or practically anything else. In most queuing systems, service time is the bottleneck that creates the queue.

https://en.m.wikipedia.org/wiki/Little%27s_law

An extremely useful law to remember. You’d be surprised how much bullshit it can detect!

> An extremely useful law to remember....

Would you be willing to provide an example where you applied Little's Law?

Many electronic queueing systems in hospitals etc. show you how many customers are ahead of you. If you observe the time it takes for a few people to receive service you can estimate how long you'll be stuck waiting. This is a number they rarely show you because it's often not encouraging...

It's the most primitive application but immediately useful for anyone.

I've used similar methods to estimate work lag: https://two-wrongs.com/estimating-work-lag.html

And to argue for concurrency limits in latency-sensitive backend serviced.

Every time I'm in a queue at a store I compute the arrival rate while I wait in line. (Little's Law is one of my very favorite pages in all of Wikipedia!).

Sure! In a professional capacity - our customers were dissatisfied with the length of time and flakiness of a particular workload job[0], and I was party to the discussions on how to solve this problem. Knowing Little's Law allowed me to dissent against the prevailing opinion that we should customise our job processing queue to prioritise these jobs[1], arguing instead to provision more resources (i.e. beefier servers).

The decision was still made to alter the priority. The change went into production and was found to unacceptably degrade the performance of other jobs. Thankfully, one of the engineers who I had convinced that "processing time is the only factor that matters" had spent all their time optimizing the heavy task, to the point where it was no longer a heavy task, thus saving the day.

0. The job was some kind of "export CSV" form, and it somehow involved both 'traversing dozens of highly normalised tables' and 'digging into json blobs stored as text'.

1. I specifically remember one of the arguments was that if you have 3 heavy tasks A B and C, best case is "in parallel" which takes max(A, B, C) time whereas worst case is "sequential" which takes (A) + (B + A) + (C + B + A) time, our current priority approximated the "sequential" scenario, and the priority change would instead approximate the "parallel" scenario. I use scare quotes because I felt it was a resource issue (the "sequential" pattern was a byproduct of the most common way a heavy task got enough resources... which was an earlier heavy task finished and freed up a lot of resources).

Thank you! -- and thank you to the others who shared!

what a predictable result. changing to a priority queue in a heavily utilized system will ALWAYS increase wait time for other tasks if independent tasks.

The ONLY time thats not true is if higher priority tasks would eliminate the need for other tasks.

You are citing amdahl's law: https://en.wikipedia.org/wiki/Amdahl%27s_law

Yes, that sounds a lot like the argument I remember hearing