←back to thread

“I'm basically giving myself a permanent vacation from being BDFL”

(mail.python.org)
2024 points randlet | 9 comments | 12 Jul 18 15:22 UTC | HN request time: 2.748s | source | bottom
Show context
adamnemecek ◴[12 Jul 18 16:16 UTC] No.17516031[source]▶
Even though I write python only if it’s going to be like 200 loc tops, it’s insane how gracefully the language has aged. It came out in 91, thus predating for example Java. Fun fact, I believe the very first google scraper was written in it.

At the same time I can’t wait for the next language to replace it.

replies(4): >>17516304 #>>17516428 #>>17516829 #>>17518450 #
petters ◴[12 Jul 18 17:34 UTC] No.17516829[source]▶
Wasn't the first Google scraper written in Java? I saw an old Usenet post by Page from about 1997 asking about user agents in Java.

Perhaps that was the second scraper.

replies(1): >>17516900 #
1. adamnemecek ◴[12 Jul 18 17:42 UTC] No.17516900[source]▶
> Perhaps that was the second scraper.

I believe that is the case yes. However correct me if m wrong.

replies(1): >>17517318 #
2. secure ◴[12 Jul 18 18:29 UTC] No.17517318[source]▶
Yes, Larry Page’s first crawler was in Java, and Scott Hassan rewrote it in Python.

From https://www.vanityfair.com/news/2018/07/valley-of-genius-exc...:

> Scott Hassan: In the fall of ’95, for some reason, I started hanging out with Larry in his office. . . . At the time, Larry was trying to download a hundred pages simultaneously. And I was fixing some of the bugs that he was having with Java itself, and this went on for weeks, if not months. And I remember thinking, Wow, this is insane!, because I was spending a lot of time fixing this underlying tool. And so one weekend, I just took all his code, I took his whole entire thing, and threw it all out, and rewrote the thing that he’s been working on for months very quickly—over a weekend—because I was just sick and tired of it. I knew I could get the thing working if I used a language I knew very well, called Python. I wrote it in such a way that it could download 32,000 pages simultaneously. So Larry went from barely downloading a 100, to doing 32,000 [pages] simultaneously on a single machine.

replies(1): >>17519862 #
3. cutler ◴[13 Jul 18 01:15 UTC] No.17519862[source]▶
32,000 pages simultaneously - in 1995? Async has only recently been added to Python, no?
replies(2): >>17519885 #>>17520105 #
4. boyter ◴[13 Jul 18 01:22 UTC] No.17519885{3}[source]▶
Using threads maybe? Id be curious to see how you would achieve that in Python easily back in 1995.
5. mathgenius ◴[13 Jul 18 02:30 UTC] No.17520105{3}[source]▶
Using select: https://en.wikipedia.org/wiki/Select_(Unix)
replies(3): >>17520187 #>>17520198 #>>17521231 #
6. gcb0 ◴[13 Jul 18 02:54 UTC] No.17520187{4}[source]▶
is that a fact or are you guessing?
7. adamnemecek ◴[13 Jul 18 02:57 UTC] No.17520198{4}[source]▶
select doesnt scale well.
8. scandox ◴[13 Jul 18 08:01 UTC] No.17521231{4}[source]▶
Doesn't select have a maximum of 1024 file descriptors it can handle at any one time? Or some such?
replies(1): >>17531379 #
9. __sr__ ◴[14 Jul 18 17:46 UTC] No.17531379{5}[source]▶
By default, yes. But it can be increased. See limit(1) and ulimit(1).