←back to thread

The man who killed Google Search?

(www.wheresyoured.at)
1884 points elorant | 2 comments | 23 Apr 24 16:43 UTC | HN request time: 0s | source
Show context
ot1138 ◴[23 Apr 24 18:11 UTC] No.40135156[source]▶
Phenomenal article, very entertaining and aligns with my experience as a prominent search "outsider" (I founded the first search intelligence service back in 2004, which was later acquired by WPP. Do I have some stories).

The engineers at Google were wonderful to work with up to 2010. It was like a switch flipped mid-2011 and they became actively hostile to any third party efforts to monitor what they were doing. To put it another way, this would like NBC trying to sue Nielsen from gathering ratings data. Absurd.

Fortunately, the roadblocks thrown up against us were half-hearted ones and easily circumvented. Nevertheless, I had learned an important lesson about placing reliance for one's life work on a faceless mega tech corporation.

It was not soon after when Google eliminated "Don't Be Evil" from the mission statement. At least they were somewhat self aware, I suppose.

replies(6): >>40135263 #>>40135980 #>>40136233 #>>40138006 #>>40142360 #>>40146753 #
1. arromatic ◴[23 Apr 24 18:19 UTC] No.40135263[source]▶
1. Do you know what caused it ? 2. How did the hostility look like ? 3. How did you circumvented them ? 4. What did your search service do ?
replies(1): >>40136382 #
2. ot1138 ◴[23 Apr 24 19:47 UTC] No.40136382[source]▶
I don't know what caused it but I suspected at the time, and still do, that it was simply business people getting more involved in order to drive growth.

The hostility was simply this. One day we had a dedicated high level Google engineer helping us out and giving us guidance (and even special tags) to get the data we needed in a cost effective manner for both Google and us. The next day, he was gone and we received demands to know exactly what we were doing, why and even sensitive information about our business. After several months of such probing, we were summarily told that the access we had was revoked and that there was no recourse.

We circumvented by setting up thousands of unique IP addresses in 50+ countries throughout the world and pointing our spiders at Google through them (same as they do to everyone else). These were throttled to maintain very low usage rates and stay off the radar. We continually refilled our queues with untouched IPs in case any were ever blacklisted (which happened occasionally).

As for what we did, we sampled ads for every keyword under the sun, aggregated and analyzed them to find out what was working and what wasn't. This even led to methods for estimating advertiser budgets. At one point, we had virtually every Google advertiser and their ongoing monthly spend, keywords and ad copy in our database. Highly valuable to smart marketers who were looking for an edge.