←back to thread

Curl-impersonate: Special build of curl that can impersonate the major browsers

(github.com)
545 points mmh0000 | 4 comments | 03 Apr 25 15:24 UTC | HN request time: 0.633s | source
Show context
jchw ◴[03 Apr 25 16:40 UTC] No.43572243[source]▶
I'm rooting for Ladybird to gain traction in the future. Currently, it is using cURL proper for networking. That is probably going to have some challenges (I think cURL is still limited in some ways, e.g. I don't think it can do WebSockets over h2 yet) but on the other hand, having a rising browser engine might eventually remove this avenue for fingerprinting since legitimate traffic will have the same fingerprint as stock cURL.
replies(6): >>43572413 #>>43573011 #>>43574225 #>>43576912 #>>43580376 #>>43583469 #
nonrandomstring ◴[03 Apr 25 19:25 UTC] No.43574225[source]▶
When I spoke to these guys [0] we touched on those quirks and foibles that make a signature (including TCP stack stuff beyond control of any userspace app).

I love this curl, but I worry that if a component takes on the role of deception in order to "keep up" it accumulates a legacy of hard to maintain "compatibility" baggage.

Ideally it should just say... "hey I'm curl, let me in"

The problem of course lies with a server that is picky about dress codes, and that problem in turn is caused by crooks sneaking in disguise, so it's rather a circular chicken and egg thing.

[0] https://cybershow.uk/episodes.php?id=39

replies(2): >>43574560 #>>43575789 #
1. thaumasiotes ◴[03 Apr 25 21:38 UTC] No.43575789[source]▶
> Ideally it should just say... "hey I'm curl, let me in"

What? Ideally it should just say "GET /path/to/page".

Sending a user agent is a bad idea. That shouldn't be happening at all, from any source.

replies(2): >>43578281 #>>43579324 #
2. Tor3 ◴[04 Apr 25 04:17 UTC] No.43578281[source]▶
Since the first browser appeared I've always meant that sending a user agent id was a really bad idea. It breaks with the fundamental idea of the web protocol, that it's the server's responsibility to provide data and it's the client's responsibility to present it to the user. The server does not need to know anything about the client. Including user agent in this whole thing was a huge mistake as it allowed web site designers to code for specific quirks in browsers. I can to some extent accept a capability list from the client, but I'm not so sure even that is necessary.
3. nonrandomstring ◴[04 Apr 25 07:35 UTC] No.43579324[source]▶
Absolutely, yes! A protocol should not be tied to client details. Where did "User Agent" strings even come from?
replies(1): >>43579741 #
4. darrenf ◴[04 Apr 25 08:47 UTC] No.43579741[source]▶
They're in the HTTP/1.0 spec. https://www.rfc-editor.org/rfc/rfc1945#section-10.15

10.15 User-Agent

   The User-Agent request-header field contains information about the
   user agent originating the request. This is for statistical purposes,
   the tracing of protocol violations, and automated recognition of user
   agents for the sake of tailoring responses to avoid particular user
   agent limitations.