←back to thread

422 points simedw | 2 comments | | HN request time: 0.454s | source
Show context
098799 ◴[] No.44434263[source]
You could also use headless selenium under the hood and pipe to the model the entire Dom of the document after the JavaScript was loaded. Of course it would make it much slower but also would amend the main worry people have which is many websites will flat out not show anything in the initial GET request.
replies(1): >>44434271 #
1. busssard ◴[] No.44434271[source]
can you flesh this out a tiny bit? because for indy-crawlers the javascript rendering is the main problem.
replies(1): >>44435515 #
2. 098799 ◴[] No.44435515[source]
Here's a sketch: https://chatgpt.com/share/68640b97-9a48-8007-a27c-fdf85ff412... -- selenium drives your actual browser under the hood.