Most active commenters
  • mrkramer(6)
  • tialaramex(3)
  • foldr(3)

←back to thread

631 points xbryanx | 35 comments | | HN request time: 1.706s | source | bottom
Show context
mike_hearn ◴[] No.44531351[source]
To the NY Times: please don't say they died by suicide. The passive voice makes it sound like some act of God, something regrettable but unavoidable that just somehow happened. It's important not to sugarcoat what happened: the postmasters killed themselves because the British state was imprisoning them for crimes they didn't commit, based on evidence from a buggy financial accounting system. Don't blur the details of what happened by making it sound like a natural disaster.

Horizon is the case that should replace Therac-25 as a study in what can go wrong if software developers screw up. Therac-25 injured/killed six people, Horizon has ruined hundreds of lives and ended dozens. And the horrifying thing is, Horizon wasn't something anyone would have previously identified as safety-critical software. It was just an ordinary point-of-sale and accounting system. The suicides weren't directly caused by the software, but from an out of control justice and social system in which people blindly believed in public institutions that were actually engaged in a massive deep state cover-up.

It is reasonable to blame the suicides on the legal and political system that allowed the Post Office to act in that way, and which put such low quality people in charge. Perhaps also on the software engineer who testified repeatedly under oath that the system worked fine, even as the bug tracker filled up with cases where it didn't. But this is HN, so from a software engineering perspective what can be learned?

Some glitches were of their time and wouldn't occur these days, e.g. malfunctions in resistive touch screens that caused random clicks on POS screens to occur overnight. But most were bugs due to loss of transactionality or lack of proper auditing controls. Think message replays lacking proper idempotency, things like that. Transactions were logged that never really occurred, and when the cash was counted some appeared to be missing, so the Post Office accused the postmasters of stealing from the business. They hadn't done so, but this took place over decades, and decades ago people had more faith in institutions than they do now. And these post offices were often in small villages where the post office was the center of the community, so the false allegations against postmasters were devastating to their social and business lives.

Put simply - check your transactions! And make sure developers can't rewrite databases in prod.

replies(19): >>44531419 #>>44531429 #>>44531441 #>>44531521 #>>44531528 #>>44531663 #>>44531829 #>>44531922 #>>44531950 #>>44531986 #>>44531999 #>>44532057 #>>44532232 #>>44532622 #>>44532757 #>>44532759 #>>44533432 #>>44535081 #>>44538524 #
cedws ◴[] No.44531441[source]
>if software developers screw up

Well, yes, they did screw up, but the fallout was amplified 100x by bad management.

replies(3): >>44531543 #>>44532415 #>>44534970 #
1. mrkramer ◴[] No.44531543[source]
"The Horizon IT system contained "hundreds" of bugs[0]."

If your accounting software has hundreds of bugs then you are really in the deep shit.

[0]https://en.wikipedia.org/wiki/British_Post_Office_scandal#:~...

replies(7): >>44531604 #>>44531684 #>>44531831 #>>44531853 #>>44532648 #>>44533820 #>>44534107 #
2. PUSH_AX ◴[] No.44531604[source]
Well not really, no one should be committing suicide due to a buggy system. If you know the details of the case it was widespread but the post office decided to gaslight everyone and put people in debt and prison. That’s what caused this, the bugs were just a catalyst for shitty humans to do shitty things
replies(1): >>44531657 #
3. mrkramer ◴[] No.44531657[source]
Yea management failed but wouldn't the most logical thing be to call in computer forensics experts and quality test the software, reverse engineering it and try to catch the bugs. This wasn't the classic case of financial fraud, this was all about faulty software.
replies(2): >>44531701 #>>44531845 #
4. voxic11 ◴[] No.44531684[source]
But it was the decision to gaslight and charge the postmasters with crimes that caused the suicides, not the bugs in the code. If they had just admitted that the accounting issues were due to bugs in the system then I really doubt anyone would have committed suicide.
replies(1): >>44531737 #
5. voxic11 ◴[] No.44531701{3}[source]
The Post Office management knew about the bugs but didn't want to take the blame for the accounting issues they caused (since it was management that purchased and approved the software some blame would have fallen on them).
replies(1): >>44531759 #
6. ◴[] No.44531737[source]
7. mrkramer ◴[] No.44531759{4}[source]
Fujitsu was all to blame, after all they created and maintained the software. It just blows my mind why would courts pursue the individuals and not the creator of the software, when they realized that this mess was widespread and not isolated.
replies(1): >>44531971 #
8. tialaramex ◴[] No.44531831[source]
So long as the jury understands this, it's all fine.

If you're on trial for doing X and your jury is told by a prosecution witness "mrkramer did X" and under cross they admit that's based on computer records which are often bogus, inconsistent, total nonsense, it doesn't take the world's best defence lawyer to secure an "innocent" verdict. That's not a fun experience, but it probably won't drive you to suicide.

One of the many interlocking failures here is that the Post Office, historically a government function, was allowed to prosecute people.

Suppose I work not for the Post Office (by this point a private company which is just owned in full by the government) but for say, an Asda, next door. I'm the most senior member of staff on weekends, so I have keys, I accept deliveries, all that stuff. Asda's crap computer system says I accepted £25000 of Amazon Gift Cards which it says came on a truck from the depot on Saturday. I never saw them, I deny it, there are no Gift Cards in stock at our store.

Asda can't prosecute me. They could try to sue, but more likely they'd call the police. If the police think I stole these Amazon cards, they give the file to a Crown Prosecutor, who works for the government to prosecute criminals. They don't work for Asda and they're looking at a bunch of "tests" which decide whether it makes sense to prosecute people.

https://www.cps.gov.uk/about-cps/how-we-make-our-decisions

But because the Sub-postmasters worked under contract to the Post Office, it could and did in many cases just prosecute them, it was empowered to do that. That's an obvious mistake, in many of these cases if you show a copper, let alone a CPS lawyer your laughable "case" that although this buggy garbage is often wrong you think there's signs of theft, they'll tell you that you can't imprison people on this basis, piss off.

A worse failure is that Post Office people were allowed to lie to a court about how reliable this information was, and indeed they repeatedly lied in later cases where it's directly about the earlier lying. That's the point where it undoubtedly goes from "Why were supposedly incompetent morons given this important job?" where maybe they're morons or maybe they're liars, to "Lying to a court is wrong, send them to jail".

replies(4): >>44532139 #>>44532231 #>>44532264 #>>44534251 #
9. noisy_boy ◴[] No.44531845{3}[source]
> Yea management failed but wouldn't the most logical thing be to call in computer forensics experts

Yea and who is responsible for engaging them?

replies(1): >>44531918 #
10. mrkramer ◴[] No.44531918{4}[source]
I meant courts should've called in multiple expert witnesses and even computer forensics companies. This case looks like government or in this case courts colluded with British Post Office.
replies(1): >>44534179 #
11. kelnos ◴[] No.44531932[source]
Where are you getting the idea that anyone suggested this?

At any rate, it was the persecuted postal workers who committed suicide, not the software developers.

12. mrkramer ◴[] No.44531938[source]
I meant top management is in deep shit if their finical departments run low quality buggy accounting software not the staff. Or in this case post office branches run the buggy software. All in all, decentralized nature of post office system was the thing that drove everything to this madness.
13. blibble ◴[] No.44531971{5}[source]
because UK law says (said?) the computer can't be wrong

and the post office management had no interest in proving otherwise

they should be going after the management

replies(1): >>44532058 #
14. albedoa ◴[] No.44531991[source]
This is not at all what happened or what anyone suggested, yikes.
15. foldr ◴[] No.44532058{6}[source]
UK law said that there was a presumption that computer systems were working correctly unless there was evidence to the contrary. That’s not inherently nuts. It makes roughly as much sense as assuming that, say, a dishwasher is in working order unless there’s evidence to the contrary. This presumption in and of itself could just as well aid a person’s defense as hinder it (e.g. if they have an alibi based on computer records).

In this case it should have been very easy to provide evidence to override the presumption that the Horizon system was working correctly. That this didn’t happen seems to have resulted from a combination of bad lawyering and shameless mendacity on the part of Fujitsu and the Post Office.

Don’t get me wrong — the whole thing is a giant scandal. I’m just not sure if this particular presumption of UK law is the appropriate scapegoat.

replies(1): >>44532195 #
16. pcthrowaway ◴[] No.44532139[source]
> If you're on trial for doing X and your jury is told by a prosecution witness "mrkramer did X" and under cross they admit that's based on computer records which are often bogus, inconsistent, total nonsense, it doesn't take the world's best defence lawyer to secure an "innocent" verdict. That's not a fun experience, but it probably won't drive you to suicide.

I imagine digital records are involved in nearly every trial at this point. Good luck getting this point admitted by the justice system.

replies(1): >>44533498 #
17. mrkramer ◴[] No.44532195{7}[source]
>UK law said that there was a presumption that computer systems were working correctly unless there was evidence to the contrary.

Defense had to prove that only one Horizon/Fujitsu accounting software was buggy and the whole prosecution falls apart e.g. If John's Horizon/Fujitsu accounting software has bugs then Peter's Horizon/Fujitsu accounting software most probably has bugs too.

replies(3): >>44532240 #>>44532474 #>>44533507 #
18. cameronh90 ◴[] No.44532231[source]
> Asda can't prosecute me.

They can, actually. Anyone in the UK can launch a private prosecution. It's rare because it's expensive and the CPS can (and often do) take over any private prosecution then drop it.

Nevertheless, the power exists and has been intentionally protected by parliament. I think most would agree it needs reform, however.

replies(2): >>44533532 #>>44535079 #
19. foldr ◴[] No.44532240{8}[source]
In principle, yes. It may be that the bar was set too high and that there needs to be some clarification of exactly what the presumption means.

I’d argue that some kind of weak presumption along these lines clearly makes sense and is probably universal across legal systems. For example, suppose the police find that X has an incriminating email from Y after searching X’s laptop. Are they required to prove that GMail doesn’t have a bug causing it to corrupt email contents or send emails to the wrong recipients? Presumably not.

20. pjc50 ◴[] No.44532242[source]
The suicide victims weren't programmers, they were postmasters, who had been falsely accused of fraud by the software.
replies(1): >>44538356 #
21. DaveLond ◴[] No.44532264[source]
It's worse than that - in UK law you cannot question the evidence produced by a computer unless you can prove the computer is not operating correctly - it's an inversion of the normal burden of proof.

They've started the process of thinking about if that law makes sense given this case: https://www.gov.uk/government/calls-for-evidence/use-of-evid...

replies(2): >>44532698 #>>44532871 #
22. mike_hearn ◴[] No.44532474{8}[source]
IIRC one issue was that every time someone advanced the theory something was wrong with Horizon, the Post Office kept claiming that nobody else was experiencing any issues. They also lied under oath, claiming no bugs that could cause such situations were known. Given this most the of defence lawyers abandoned that line of inquiry (they were nothing special, seeing as village postmasters aren't rich).
23. ptero ◴[] No.44532648[source]
Every system has bugs, even deployed, high visibility accounting systems. Debian stable, which I personally view as the gold standard for a robust general purpose OS, has hundreds of bugs.

That is not to say that bugs are good. They are bad and should be squashed. But the Horizon failure, IMO, is with the management, that pretended that the system was bug free and, faced with the evidence to the contrary, put the blame on postmasters. My 2c.

24. foldr ◴[] No.44532698{3}[source]
It’s only an inversion of the usual burden of proof if you assume that evidence from a computer can only ever be used to aid the prosecution. It can also be used to aid the defense, in which case this presumption makes it harder to convict someone, not easier.
25. petercooper ◴[] No.44532871{3}[source]
A juror can, and should IMHO, however consider that evidence based entirely upon computer records may potentially be erroneous and therefore unable to secure proof 'beyond a reasonable doubt'. If I were a juror, I'd default to non-guilty if a case were based entirely upon the results of an algorithm or computerised records because they introduce doubt.
26. tialaramex ◴[] No.44533498{3}[source]
There are plenty of examples, Light Blue Touchpaper talks about this a bunch. You do have a problem that courts will believe technicians very broadly unless somebody competent is cross-examining to highlight where the limits of their evidence are. So your defence will need to hire such an expert and your legal team need to get the judge to understand why everybody is going to listen to nerd stuff for however long when they thought this was a case about, say, theft.
27. buzer ◴[] No.44533507{8}[source]
Proving bugs can be pretty hard if you don't have access to software & source code. That is similar to the US, courts usually won't give you access to source code to verify if software is operating correctly, you generally only get cross examine the company representative & person who performed the test. DNA tests are one good example.

One case where defense did get access to the code (FST developed by NYC) led to discoveries (https://www.propublica.org/article/federal-judge-unseals-new...) that led to it being retired from use.

28. tialaramex ◴[] No.44533532{3}[source]
This is Technically Correct, which is, I admit, the Best Kind of Correct, but in practical terms it won't happen.

[Edited: Got the Futurama quote wrong, fixed that]

29. wat10000 ◴[] No.44533820[source]
I'd be shocked if any piece of software large enough to qualify as an "accounting system" didn't contain at least hundreds of bugs. We're just not that good at building software. Especially if you consider that the system encompasses all of the dependencies, so you should count bugs in the OS, CPU, any relevant firmware, etc.
30. mr_toad ◴[] No.44534107[source]
If any large system wasn’t constantly logging errors I’d immediately assume there was something wrong with the error logging system. Only trivial software is bug free.
31. mr_toad ◴[] No.44534179{5}[source]
> I meant courts should've called in multiple expert witnesses and even computer forensics companies.

UK courts don’t (can’t) do that, that’s up to the plaintiffs or defendants.

32. ◴[] No.44534251[source]
33. carstout ◴[] No.44535079{3}[source]
Unfortunately the "its rare" isnt true. it is more common now than it was back in the horizon days. It also isnt necessarily expensive since you can apply for costs with the default being for it to be paid (unless good reason not to). As such whilst its not an option for the average person who cant afford the upfront cost it is very practical for large businesses especially if they engage in it often and hence can stand up a department for it.

Its one of the offerings from TM-Eye aka one of the "private police forces". https://tm-eye.co.uk/what-we-do/private-prosecutions/

It is an actual example of a two tier justice system since those who can afford the private prosecution skip the queue for the public system but will still normally have the taxpayer pay for it.

There is currently a consultation underway as per below article which, incidentally, mentions a more recent dubious example of private prosecutions which got slapped down.

https://www.gov.uk/government/consultations/oversight-and-re...

34. nirui ◴[] No.44538356{3}[source]
Damn! That's dark. I should've read the article first, but I did not. Sorry.

I kinda understand the false guilt these postmasters must have felt when they were wrongfully accused. These people should not be dead like that, those who puts them into that living hell should.

Oddly though, the justice of this world usually don't work like that. Usually, it's the people at the lowest level who suffer the worst fate/abuse, simply because they are the most defenseless, while the people "on the top" pets themselves for "resolving" the problem they created.

It's a odd place to mention, but from one of Donald Trump's story (The Verrazano-Narrows Bridge one where he noticed the unnoticed engineer) I've learned it is important that one must actively take what they deserved (recognition in his story, justice in this case), and at least don't be silent when other people is stealing it away. This is "a nasty world", and if you want to make things right, you must "Fight! Fight! Fight!" and never give up.

replies(1): >>44539898 #
35. pjc50 ◴[] No.44539898{4}[source]
> Damn! That's dark. I should've read the article first, but I did not. Sorry.

Now you will remember to at least skim the article before posting.