←back to thread

Curl-impersonate: Special build of curl that can impersonate the major browsers

(github.com)
545 points mmh0000 | 8 comments | 03 Apr 25 15:24 UTC | HN request time: 0.001s | source | bottom
Show context
davidsojevic ◴[03 Apr 25 23:26 UTC] No.43576680[source]▶
There's a fork of this that has some great improvements over to the top of the original and it is also actively maintained: https://github.com/lexiforest/curl-impersonate

There's also Python bindings for the fork for anyone who uses Python: https://github.com/lexiforest/curl_cffi

replies(3): >>43578994 #>>43580445 #>>43583614 #
nyanpasu64 ◴[04 Apr 25 06:37 UTC] No.43578994[source]▶
I suppose it does make sense that a "make curl look like a browser" program would get sponsored by "bypass bot detection" services...
replies(1): >>43581862 #
ImHereToVote ◴[04 Apr 25 13:14 UTC] No.43581862[source]▶
Easy. Just make a small fragment shader to produce a token in your client. No bot is going to waste GPU resources to compile your shader.
replies(4): >>43582452 #>>43583572 #>>43584954 #>>43601084 #
1. kelsey978126 ◴[04 Apr 25 13:56 UTC] No.43582452[source]▶
Why do people even think this? Bots almost always just use headful instrumented browsers now. if a human sitting at a keyboard can load the content, so can a bot.
replies(2): >>43582619 #>>43582724 #
2. simpaticoder ◴[04 Apr 25 14:07 UTC] No.43582619[source]▶
Security measures never prevent all abuse. They raise the cost of abuse above an acceptable threshold. Many things work like this. Cleaning doesn't eliminate dirt, it dilutes the dirt below an acceptable threshold. Same for "repairing" and "defects", and some other pairs of things that escape me atm.
replies(1): >>43585637 #
3. ImHereToVote ◴[04 Apr 25 14:13 UTC] No.43582724[source]▶
We are talking about Curl bots here. How is what you are saying relevant?
replies(1): >>43583229 #
4. cAtte_ ◴[04 Apr 25 14:47 UTC] No.43583229[source]▶
no, nyanpasu64's comment extended the discussion to general bot detection
5. abofh ◴[04 Apr 25 17:46 UTC] No.43585637[source]▶
That's the same argument as CAPTCHA's - as far as I know there are no bots protesting them making their lives harder, but as a human - my life is much harder than it needs to be because things need me to prove I'm a human.

Clean for data ingestion usually means complicated for data creation - optimizing for the advertisers has material cash value downstream, but customers are upstream, and making it harder is material too.

replies(1): >>43585970 #
6. ImHereToVote ◴[04 Apr 25 18:17 UTC] No.43585970{3}[source]▶
What is so hard about running a fragment shader after the site has loaded?
replies(1): >>43586278 #
7. abofh ◴[04 Apr 25 18:46 UTC] No.43586278{4}[source]▶
I have to assume /s, but lacking that -- Why can't you just allow `curl`? You need a human for advertising dollars or a poor mechanism of rate limiting. I want to use your service. If you're buying me a fragment shader, I guess that's fine, but I'm feeding it to the dogs, not plugging in your rando hardware in to my web-browser.
replies(1): >>43591452 #
8. ImHereToVote ◴[05 Apr 25 06:49 UTC] No.43591452{5}[source]▶
I just want to limit my server to usual human users. If you have JavaScript disabled, you won't be missed. Sorry.