Most active commenters
  • skissane(4)
  • immibis(4)
  • jacquesm(3)

←back to thread

341 points hlandau | 26 comments | | HN request time: 1.913s | source | bottom
Show context
abigail95 ◴[] No.37962300[source]
> What would a perfect attacker do?

If you had physical access to the computer, some sort of bus interception to exfiltrate data from the machine.

replies(2): >>37962373 #>>37963714 #
1. whalesalad ◴[] No.37962373[source]
extremely difficult to get physical access in a datacenter
replies(5): >>37962416 #>>37962421 #>>37962655 #>>37963217 #>>37963509 #
2. sebstefan ◴[] No.37962416[source]
Sure but this is the German police and more generally nation states, not only they can, they don't even need to they just ask
replies(1): >>37965648 #
3. LeoPanthera ◴[] No.37962421[source]
I would suggest that if you are the police, you can break into a datacenter with a flash of a badge. I can't imagine many would attempt to stop you.
replies(3): >>37962600 #>>37963230 #>>37964311 #
4. mjevans ◴[] No.37962600[source]
I would hope they at least:

* Require a copy of the badge number, and verify that this officer is assigned and expected to be at this business right now.

* Require them to sign into and out of the site.

* Annotate which systems / compromises are in place.

- That all of the above MIGHT be sealed under a court order; I would hope any such order has an automatic 'sunset' date, and possibly renewal upon review by a different judge.

replies(3): >>37962663 #>>37963155 #>>37974369 #
5. mike_d ◴[] No.37962655[source]
The organization conducting the MitM likely has physical access to the machine already. The original post indicates the link on the network interface went down for 19 seconds, indicating a device was placed in front of the server.
6. LeoPanthera ◴[] No.37962663{3}[source]
Those are some very optimistic hopes!
7. skissane ◴[] No.37963155{3}[source]
A business can request visiting law enforcement to do all those things, and hopefully law enforcement complies. However, if they refuse to comply, realistically you just have to let them in anyway. Document their non-compliance and provide it to your lawyers, who can decide what action to take (lodge a formal complaint to the law enforcement agency, apply to a judge for an injunction to compel their compliance, etc)

Well, that’s true in countries like Germany or the US. I suspect in somewhere like Russia or China, formal complaints are unlikely to achieve anything except invite government retaliation.

replies(2): >>37964602 #>>37968642 #
8. jacquesm ◴[] No.37963217[source]
All the people working in the datacenter have that level of physical access.

Unless they are very closely supervised they can do a lot of damage without anybody being the wiser until they get caught. I've been in (nominally very secure) DCs on behalf of customers and I've seen:

- unlocked racks

- doors open

- temporary network cables and keyboards, monitors and mice attached to running systems

- systems logged in left unattended

- floor panels raised up and left open unattended exposing cabling

- meet-me rooms with interfaces exposed (gear in racks without doors)

DC personnel tends to trust each other, and they probably shouldn't. But it's hard to be part of a closely knit crew for a long time without getting into a 'get stuff done' mode where protocol and rules are there in principle but less so in practice because it is seen as an efficiency penalty. It's another instance of the 'normalization of deviation' phenomenon.

replies(1): >>37963645 #
9. jacquesm ◴[] No.37963230[source]
I highly doubt it is that simple for LE to enter a DC without a warrant signed by a judge, but insiders have all of that access and plants in DCs can and do happen.

I was present when Dutch LE seized a bunch of servers on behalf of an FBI liaison officer in NL and everything went 'by the book', there is no way an LE officer without a signed order from a judge would have been granted access.

10. OmarAssadi ◴[] No.37963509[source]
Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine.

And outside of nation-state requests, even ignoring the fact that someone could probably pay-off an employee, I think ease would depend a lot on the datacenter and target; judging by the awesome and hilarious story behind the Fremont Cabal accidentally becoming an internet exchange by essentially having some dude barely secretly slipping unauthorized cables into the raceways [1], I figure there are a lot of places where if your target is simply renting a couple rack units or single rack rather than an entire locked cage, you can probably get physical access by doing the same.

Also, a lot of Deviant Ollam's stories about industrial security and the dozens of ways he's broken into utility companies, server rooms, etc — mostly just by being confident, looking the part, bad doors [2], and badge cloning [3] — don't give me a ton of confidence that someone with skill couldn't feasibly either get direct access to servers they shouldn't, or at the very least, access to an important part of the supply chain for their target.

And speaking of supply chain, my processor died recently, so I ordered a brand-new in box replacement Ryzen, and when it arrived last night, out of curiosity, I wanted to see if I could get the CPU out of the box without breaking the tamper-evident authenticity seal...

... and about fifteen minutes later, after borrowing a syringe and hypodermic needle from my mom, a little bit of isopropyl alcohol, a blade from my safety razor, and a quick look at a video from LockPickingLawyer [4] and a couple from datagram at DEFCON's Tamper-Evident Village [5][6], I had the CPU out, put my old one for now, and re-applied the sticker with no visible damage to the box or seal.

All I had to do was tip it upside down at about 90-degrees, douse a little bit of the alcohol under the top of the seal, let gravity do most of the work, and then carefully lift the seal with the razor. After that, I just lightly squeezed the box to make the front tab come as forward as possible, and then carefully pushed the ear flaps down to prevent tearing, and then I was in.

I've seen others demonstrate it on older AMD boxes that had flexible cardboard in-place of the cooler, allowing them to pull the cardboard to make enough room for tools to get out the CPU without even touching the seal [7]. But in my case, it was a newer box with hard plastic inside where the cooler would've been, so that's why I went for the seal instead.

No surprise to me now that counterfeiting is rampant on Amazon, with people returning the box after putting in either random junk, dead Athlons covered by a counterfeit serial-matching IHS, or the cheapest socket-compatible CPU after deluding both and swapping the IHS.

I figure with a bit of practice and better tools, like Teflon spudgers and syringes, it'd be significantly easier to get past 99% of tamper-resistant/tamper-evident seals and into boxes you're not supposed to be without risking damage, and then you can intercept a package, compromise something critical, like the server BMC or firmware, reseal everything, and be on your way.

And given the relatively recent scare with loads of servers, including Dell and others, being shipped with "AMERICAN MEGATRANDS" labels on their BMC boards, with no one noticing until a YouTube commenter pointed it out during a teardown by ServeTheHome, I think it's totally feasible for an enemy to just compromise the entire physical supply-chain of a company, datacenter, or whatever else [8].

[1]: Oxide's On the Metal: Kenneth Finnegan - https://oxide.computer/podcasts/on-the-metal/kenneth-finnega...

[2]: Deviant Ollam @ Shakacon: The Search for the Perfect Door - https://www.youtube.com/watch?v=4YYvBLAF4T8

[3]: Deviant Ollam / Modern Rogue: Getting an RFID Implant - https://youtu.be/SZiRISGdQ4g?t=277

[4]: LockPickingLawyer: Did I Cheat On This Challenge? (Tamper-Sealed Abus) - https://www.youtube.com/watch?v=xUJtqvYDnkg

[5]: DEFCON 19: Introduction to Tamper Evident Devices - https://www.youtube.com/watch?v=W07ZpEv9Sog

[6]: DEFCON 30: Tamper Evident Village - https://www.youtube.com/watch?v=slhdowWjSuU

[7]: cycurious: How Counterfeiters replace CPU in Sealed Retail Box - https://www.youtube.com/watch?v=Bni8bgGlXDE

[8]: ServeTheHome: Dude this should NOT be in a Dell Switch… or HPE Supercomputer - https://www.servethehome.com/dude-dell-hpe-ami-american-mega...

replies(2): >>37965269 #>>37974377 #
11. manxman ◴[] No.37963645[source]
Agree re: everything you said but wanted to add datadentre security staff are some of the most interesting characters I’ve encountered. Not sure I sleep as well at night after seeing what I saw.
replies(1): >>37964756 #
12. bsder ◴[] No.37964311[source]
> I can't imagine many would attempt to stop you.

You would be 99% wrong. Even if law enforcment presented proper paperwork, every colo I have ever used would call and verify the paperwork. They might not call me, but they sure as hell would call their own lawyers. Once law enforcement is on the other side of the cage, important customers who pay real money could get compromised.

There is a massive difference between getting physical access to your server in a data center and coughing up everything about your server by simply emailing a minion in a cloud provider.

13. leosarev ◴[] No.37964602{4}[source]
Latter is not correct. It's well known difference in Russia between companies that willingly cooperate with government agencies informally, and those who just provide information upon formal request according to law.
replies(1): >>37971737 #
14. OmarAssadi ◴[] No.37964756{3}[source]
Do tell, please; stories about "interesting characters" are often the best.
replies(1): >>37971886 #
15. ThePowerOfFuet ◴[] No.37965269[source]
Next time you try that trick, use heptane (sold commercially in North America as "Un-du") instead of isopropanol. Hot knife meets butter... and it evaporates cleanly and the sticker retains most of its stickiness!
16. Beijinger ◴[] No.37965648[source]
While the rule of law in Germany is much worse than most people think, it is not so bad as you assume. I doubt that Hetzner would give in to a police request. A court order, yes. But not to a police request. This does not mean, the police won't try it: https://www.dw.com/de/e-mail-firma-kritisiert-ermittler/a-18...

Unfortunately can't find the original post. Every idiot police officer thought he has a right to just email them to handle over data :-)

17. chatmasta ◴[] No.37968642{4}[source]
> realistically you just have to let them in anyway

No, you don't. If they have a warrant then you need to let them in for the purposes specified in the warrant. Otherwise you're free to tell them to piss off. Unfortunately you're also free to acquiesce to any of their demands.

This kind of passive, default-compliant attitude from service providers, while understandable from a "path of least resistance" standpoint, is exactly the kind of behavior that allows the third party doctrine to circumvent so many of our basic rights. As a service provider, often the more difficult path is to challenge authority, rather than to cooperate with it. And unfortunately that means that most service providers will simply cooperate.

replies(2): >>37971696 #>>37974552 #
18. skissane ◴[] No.37971696{5}[source]
> No, you don't. If they have a warrant then you need to let them in for the purposes specified in the warrant. Otherwise you're free to tell them to piss off.

Any lawyer will tell you - if law enforcement attempts a warrant-less search, you tell them you do not consent to it, but you do not attempt to physically stop them from performing it. Tell them they are unwelcome and to come back with a warrant, but if they insist on entering in spite of that, you let them in.

replies(1): >>37974559 #
19. skissane ◴[] No.37971737{5}[source]
How do you know for sure the people who “just provide information upon formal request according to law” aren’t covertly engaging in informal cooperation?

If one morning the CEO gets an unexpected visit at home from a group of FSB agents asking for some favours, is the CEO going to say “no”? And if the CEO says “yes”, are you going to hear about it, or are they going to let the CEO continue that pretence?

Western CEOs don’t have the same worry about “accidentally” falling out of hospital windows.

replies(1): >>38035621 #
20. jacquesm ◴[] No.37971886{4}[source]
GP may well be under NDA and easy to identify.
21. immibis ◴[] No.37974369{3}[source]
You would expect that at AWS, but Hetzner is a low-cost operation.
22. immibis ◴[] No.37974377[source]
> Assuming this was done at the government's request, I assume Hetzner is more than willing to comply with a court order mandating they allow them to monitor and physically size a machine.

Amusingly, I read about an incident like this on one of those forums (probably ServeTheHome). It apparently happens so often that Hetzner's control panel has a special state for it. Server status: "seized by law enforcement" and the power-on button is disabled.

23. immibis ◴[] No.37974552{5}[source]
Non-compliance with a law enforcement order is a good way to get shot (in America) or arrested (in most countries) even if there is no legal basis for the order.
24. immibis ◴[] No.37974559{6}[source]
"Letting them in" is another way of saying you consent. Don't "let" them in... just don't physically stop them coming in.
replies(1): >>37979279 #
25. skissane ◴[] No.37979279{7}[source]
If you unlock a door for someone but simultaneously say “I don’t consent to you passing through it”, the first act does not cancel out the second. Whereas, if you don’t unlock it, if they really want to go in they’ll knock it down, causing damage in the process. Unlocking it for them is about avoiding damage to property, it is not a form of consent if accompanied by a clear verbal refusal of consent
26. leosarev ◴[] No.38035621{6}[source]
Actually, that's happen to some people that I know.

Roem.ru site (small but ifluential at time) recieved official, but illegal request from high level FSB agent to disclose commentators identities. They send formal complaint to a FSB own security and to public prosecutor office. Former officially warned FSB to stoppes illegal actions.

Funny thing: 7 years later FSB agent was convicted for being CIA asset.