I'll give a fun example from the past.
I used to work at a company that did auto inspections. (e.x. if you turned a lease in, did a trade in on a used car, private party, etc.)
Because of that, we had a server that contained 'condition reports', as well as the images that went through those condition reports.
Mind you, sometimes condition reports had to be revised. Maybe a photo was bad, maybe the photos were in the wrong order, etc.
It was a perfect storm:
- The Image caching was all inmem
- If an image didn't exist, the server would error with a 500
- IIS was set up such that too many errors caused a recycle
- Some scraper was working off a dataset (that ironically was 'corrected' in an hour or so) but contained an image that did not exist.
- The scraper, instead of eventually 'moving on' would keep retrying the URL.
It was the only time that org had an 'anyone who thinks they can help solve please attend' meeting at the IT level.
> and you would not believe the extent that grocers go to to make price comparison difficult. This thing doesn't make thousands or even hundreds of requests - maybe a few dozen over the course of a day.
Very true. I'm reminded of Oren Eini's tale of building an app to compare grocery prices in Israel, where apparently mandated supermarket chains to publish prices [0]. On top of even the government mandate for data sharing appearing to hit the wrong over/under for formatting, There's the constant issue of 'incomparabilities'.
And it's weird, because it immediately triggered memories of how 20-ish years ago, one of the most accessible Best Buy's was across the street from a Circuit City, but good luck price matching because the stores all happened to sell barely different laptops/desktops (e.x. up the storage but use a lower grade CPU) so that nobody really had to price match.
[0] - https://ayende.com/blog/170978/the-business-process-of-compa...