Httptap: View HTTP/HTTPS requests made by any Linux program

Why not use eBPF instead? Then you could see all http requests from all processes at once, including ones that are already running. Plus you wouldn't need to bother with TLS at all, just hook on e.g. write(2).

Wouldn't this require root? A big "selling point" of httptap seems to be that precisely it doesn't require root.

Anyway the more options we have, the better.

Presumably eBPF requires root privs?

I'm having a hard time coming up with a use case where I want to use a tool like that but I'm also lacking root privileges

Inside most production environments. I could use this today inside a Pod that isn't allowed root privs.

In production environments that won't give you root access, you won't be exec'ing inside of a pod if you aren't an operator or sysadmin.

No, you are wrong. I would. The pod would be mine though.

How would hooking on write(2) solve TLS? You'll be able to read and modify the ciphertext, but the process will never call write(2) with the plaintext bytes, so you can't actually read the HTTP request. You'll just see the encrypted bytes that go on the wire, but so does the NSA :)

You need the kind of CA certificate trick that httptap uses. It comes with its own set of caveats (e.g. certificate pinning), but it can be made to work reliably in most practical scenarios.

I've spent an unjustifiable amount of time thinking about this specific problem building Subtrace [1], so I'm genuinely very interested in a simpler / more elegant approach.

[1] https://github.com/subtrace/subtrace

Unfortunately TLS happens inside the the application, not in the kernel, so using eBPF to hook syscalls to write won't help with TLS decryption.

In my particular case, I am an operator and sys admin, but I don't give myself root privileges without having to go through some serious hoops, which I only jump through if I really truly need it. If I want root, I have to actually change the kubernetes manifest yaml to allow elevation to root privileges. That's not something that can be done without getting others involved for code reviews and what not.

However, even in the case of general developers, it isn't true. Companies do restrict exec abilities, but we don't. Many startups are the same, because developers are expected to also troubleshoot and debug production issues. If you don't allow shells in pods, you are really binding the hands of your devs.

To be clear, I am not disagreeing with you. You are correct in many cases. But there are a number of exceptions in my experience.

It is quite simple to use eBPF with uprobes to hook library calls, for example: https://github.com/iovisor/bcc/blob/master/tools/sslsniff.py

The downside is this doesn't work with anything not using OpenSSL, there are projects like https://github.com/gojue/ecapture which have interceptors for many common libraries, but the downside is that needs different code for each library.

I think providing a TLS certificate is fine for the use cases of the tool; most tools won't be doing certificate pinning, but ecapture does support Android where this is more likely.

This won't work in most cases inside a Kubernetes pod, as the default seccomp policies don't allow creating namespaces within them. You can obviously relax the seccomp policies, but at that point you can also just give yourself the capabilities.

There are eBPF tools which will work, for example https://inspektor-gadget.io/docs/latest/gadgets/trace_ssl

Using uprobes to hook the SSL library, would it be possible to filter content by inspecting and modifying eg the decrypted HTTP response ?

absolutely

But read and write syscalls are used by the application to do I/O on the sockets before/after the encryption, which can be intercepted. Or you can attach uprobes directly to the TLS library's own functions.

eBPF TLS tracing: The Past, Present and Future https://blog.px.dev/ebpf-tls-tracing-past-present-future/

I believe that's how https://github.com/gojue/ecapture works. I don't know the details, but it seems to work!

My understanding is that typically a TLS library provides a socket interface for the application to write() to, which can be intercepted by an eBPF program.

Author here :). Happy to answer any questions on this TLS tracing stuff.

Yep, that's correct. It uses eBPF upprobes to attach to the SSL_write/SSL_read functions.