←back to thread

364 points adtac | 1 comments | | HN request time: 0.307s | source

Hey HN, we built Subtrace (https://subtrace.dev) to let you see all incoming and outgoing requests in your backend server—like Wireshark, but for Docker containers. It comes with a Chrome DevTools-like interface. Check out this video: https://www.youtube.com/watch?v=OsGa6ZwVxdA, and see our docs for examples: https://docs.subtrace.dev.

Subtrace lets you see every request with full payload, headers, status code, and latency details. Tools like Sentry and OpenTelemetry often leave out these crucial details, making prod debugging slow and annoying. Most of the time, all I want to see are the headers and JSON payload of real backend requests, but it's impossible to do that in today's tools without excessive logging, which just makes everything slower and more annoying.

Subtrace shows you every backend request flowing through your system. You can use simple filters to search for the requests you care about and inspect their details.

Internally, Subtrace intercepts all network-related Linux syscalls using Seccomp BPF so that it can act as a proxy for all incoming and outgoing TCP connections. It then parses HTTP requests out of the proxied TCP stream and sends them to the browser over WebSocket. The Chrome DevTools Network tab is already ubiquitous for viewing HTTP requests in the frontend, so we repurposed it to work in the browser like any other app (we were surprised that it's just a bunch of TypeScript).

Setup is just one command for any Linux program written in any language.

You can use Subtrace by adding a `subtrace run` prefix to your backend server startup command. No signup required. Try for yourself: https://docs.subtrace.dev

Show context
smw ◴[] No.43104857[source]
Can it decrypt tls? Perhaps by hooking the calls to common libraries?
replies(1): >>43105380 #
adtac ◴[] No.43105380[source]
Yes, but we've managed to do it automatically without any library/language specific hooks! It's probably one of my favourite things in Subtrace :)

We generate an ephemeral TLS root CA certificate and inject it into the system store. The generated certificate is entirely in-memory and never leaves the machine. To make this work without root privileges, we intercept the open(2) syscall to see if it's /etc/ssl/certs/ca-certificates.crt (or equivalent). If so, we append the ephemeral root CA to the list of actual CA certificates; if not, we let the kernel handle the file open like usual. This way, none of the other programs are affected, so only the program you start with `subtrace run` sees and trusts the ephemeral root CA.

After we get the program to trust the ephemeral root CA, we can proxy outgoing TLS connections through Subtrace transparently but also read the cleartext bytes.

All of this is fully automatic, of course.

replies(6): >>43105428 #>>43105948 #>>43105989 #>>43106699 #>>43107159 #>>43111647 #
memhole ◴[] No.43105989[source]
Up front, this is not my area of expertise. You would still not want to run this on the same server as your containers since someone could inject their own cert? I think it allows someone to decrypt traffic that isn't just proxied to Subtrace?

Edit: I've reviewed the docs and it looks like you do run it on the same server. For clarity, I've used Sentry before.

replies(1): >>43106088 #
adtac ◴[] No.43106088[source]
Subtrace proxies the program's connection using a regular TLS connection to the upstream server. For example, if you do `subtrace run -- curl https://example.com`, curl thinks it's talking to example.com over TLS, but it's really talking to Subtrace locally. Since we injected the ephemeral root CA into the system store, curl will trust the valid TLS certificate that Subtrace presents for example.com. From within the same server, Subtrace will handle the actual TLS connection to upstream example.com. That upstream connection is undecipherable to outsiders.

Everything is exactly as secure as before Subtrace. In other words, using Subtrace doesn't make the NSA's job any easier ;)

replies(1): >>43106156 #
1. memhole ◴[] No.43106156[source]
That's helpful! Thanks for clarifying. I'll have to check it out.