←back to thread

364 points adtac | 4 comments | | HN request time: 0.882s | source

Hey HN, we built Subtrace (https://subtrace.dev) to let you see all incoming and outgoing requests in your backend server—like Wireshark, but for Docker containers. It comes with a Chrome DevTools-like interface. Check out this video: https://www.youtube.com/watch?v=OsGa6ZwVxdA, and see our docs for examples: https://docs.subtrace.dev.

Subtrace lets you see every request with full payload, headers, status code, and latency details. Tools like Sentry and OpenTelemetry often leave out these crucial details, making prod debugging slow and annoying. Most of the time, all I want to see are the headers and JSON payload of real backend requests, but it's impossible to do that in today's tools without excessive logging, which just makes everything slower and more annoying.

Subtrace shows you every backend request flowing through your system. You can use simple filters to search for the requests you care about and inspect their details.

Internally, Subtrace intercepts all network-related Linux syscalls using Seccomp BPF so that it can act as a proxy for all incoming and outgoing TCP connections. It then parses HTTP requests out of the proxied TCP stream and sends them to the browser over WebSocket. The Chrome DevTools Network tab is already ubiquitous for viewing HTTP requests in the frontend, so we repurposed it to work in the browser like any other app (we were surprised that it's just a bunch of TypeScript).

Setup is just one command for any Linux program written in any language.

You can use Subtrace by adding a `subtrace run` prefix to your backend server startup command. No signup required. Try for yourself: https://docs.subtrace.dev

1. jgauth ◴[] No.43106111[source]
Looks like it is for http requests only? If so, wireshark is not an apt comparison.
replies(1): >>43106370 #
2. adtac ◴[] No.43106370[source]
For now, yes :)

Since we operate at the TCP level, we can actually handle pretty much any protocol. I have an implementation of a postgres handler in my git stash that intercepts and shows the SQL queries executed + the resulting rows alongside the HTTP request that triggered it (I still need to do some robustness and correctness testing before it's ready to merge). With a handful of other protocols like MySQL, Mongo, Redis, Kafka, or even FTP lol, I think Subtrace can cover most practical dev workloads.

Btw Subtrace can already record .pcap files today since it's just a simple TCP stream proxy, but raw network packet captures are mostly only useful when you're implementing new protocols, which 99% of the people using Docker containers today aren't doing. It's also a solved problem because you can just run `apt-get install tcpdump` inside the container.

Automatic tracing for app-level protocols that is easy to setup, works everywhere, lightweight for prod, fast to search, and can show the data in a clean interface is still insanely difficult today. That's the problem Subtrace is trying to solve.

replies(1): >>43107097 #
3. sophacles ◴[] No.43107097[source]
It's a pretty cool looking product. It's not wireshark, it's not close to wireshark just because it can capture some tcp pcaps, and there are more protocols relevant to container networking than a handful of TCP app-level protocols.

When I came in I was hoping to see a product that actually was for container networking, not just app data flows. Again - this is a neat tool, and probably incredibly useful for people developing way up the stack like that, but a lot of us live below the bottom of a "full-stack developer's" stack. Some features I would expect in a "wireshark for Docker containers":

* Ability to inspect DNS traffic

* Ability to trace packets the enter the conainter network stack (e.g. the packet(s) generated when the server calls send() ) into the virtual interface for the namespace and through the host machine. Ideally with multiple observation points and correlation of packets in the overlay/underlay contexts (decrypting from local keys whenever possible).

* Ability to see where a packet dies between the app and exit NIC on the host. Including things like "packets delivered to this container even though the dest IP/subnet isn't in this container"

* Similar to the previous point: ability to track packets through all the NAT steps containers introduce.

* See arp traffic on virtual interfaces.

* Ability to observe the TLS handshake and gather all the parameters of the connection.

* Packet dissection and protocol session tracing for all the tunnels.

* Bonus points if you can capture weird teleports caused by ebpf programs in the network path.

I expect this because that's how I use wireshark_+ bpftrace in container environments for the most part. I've also used it to debug while implementing protocols, but that's a less common use case of packet dissection and tracing in wireshark.

What you've built is cool, and I can see it expanding in a lot of directions very, very, usefully. I just really dislike something calling itself a wireshark while not really helping with networking (and in fact - the networking has to work reasonably well for this tool to be effective).

replies(1): >>43107396 #
4. adtac ◴[] No.43107396{3}[source]
That's totally fair, I can see why Wireshark wouldn't be the most accurate description to someone working on those kinds of problems. And fwiw, I wish my problems (before Subtrace) were cool enough to need to whip out Wireshark for packet-level inspection :)