Foursquare Open Source Places: A new foundational dataset

A nice exploration, this is a link to the announcement itself from the article:

https://location.foursquare.com/resources/blog/products/four...

What is the motivation of Foursquare to open source their places data?

Looking forward the taking a peek. Both at the foursquare data and datasette .. which I feel I've seen spruiked so much I want the tshirt.

As an aside a place in my (small, regional) city is listed in the sample gist. Feel like I should win something.

Largely, improve the data quality from contributions.

Incentivise integrations to gain market share, I assume? Maybe also grow developer knowledge and trust? I honestly haven't thought about 4S in a long time, although I am outside the US so maybe that's why?

Cynically, a desperate attempt at restoring its own relevance, which it hasn't had since 2011.

Not that I'm complaining; it's always nice to have more options.

There are a few reasons.

Firstly, the published reason is they intend to let people contribute and maintain the dataset. This is a good open data mindset to allow a dataset to maintain its health and grow. It would make the dataset sustainable.

An unpublished reason is that it could be a sign that things within Foursquare are winding down. Making the dataset sustainable reduces internal costs and increases the value to themselves and others. (like the reasons to open source software)

Foursquare City Guide is shutting down, and recently laid off 105 (25%) employees. I would imagine releasing data before things stop is a good ethical move.

https://foursquare.com/city-guide-sunset/

https://techcrunch.com/2024/05/23/foursquare-just-laid-off-1...

"The letter does, however, address which units were impacted and, relatedly, the divisions that Foursquare plans to ditch, including Visits, OCF, and Foursquare City Guide.

According to Little’s letter, Foursquare is also pausing work on a number of other initiatives, including "Mobile Developers Tools, Geode, and the current version of FSQ Insights."

I used to work for Nokia Maps (before it was renamed Here Maps) working on their places registry, which was a proprietary data set compiled from various sources of data with a similar amount of places. It was really messy data. Lots of duplicates, lots of stale data, lots of misleading data (e.g. hotels claiming to be right on the beach that in reality were a few kilometers away), etc. Part of my job was cleaning that data up and making sense of it. The registry was used to power Nokia's address and places search and became hugely popular when Nokia started bundling free maps and navigation with all their phones. Up until then you had to pay for in car navigation.

Foursquare at the time was an app for checking into places and becoming the mayor. Which was hugely popular with several of my colleagues that would obsessively checking to places to claim mayorship of places they liked. To ensure Foursquare had good content, they came up with a clever solution: just ask users that are there. This community driven curation was pretty smart. They got users to categorize and correct information.

Later when doing my first startup, I tried to work with open place data. It was a mess. There were bits and pieces available but nothing great. I actually crawled Foursquare and a few other websites in HTML form at some point to get at their ~30M places (at the time). In the end, we never launched anything and there were obvious IP issues with that data set that would have prevented us from using that commercially.

Anyway, Foursquare got all corporate and the whole checking in thing was spun off into a separate app and the main app became a glorified restaurant recommendation thingy which never really managed to stand out from the field of other recommendation thingies. And a data set + API you could license. Because that was still pretty good.

More recently, other companies partnered up to form Overture, which is a competing open data set that Bing, Amazon, and others are contributing to. Overture is kind of solving the same problem.

I'm guessing that's a big contributing reason for Foursquare to be open sourcing their data. It would be interesting to see if their data is going to be contributed to Overture or whether they are somehow against that. Last I checked, Overture still had lots of data issues. Combining the two data sets would probably help.

There’s also Overture Maps with its 3.7B features:

https://overturemaps.org/

Foursquare could conceivably become a member and donate their data.

Because they're interesting?

Because his articles are really popular with people here. he does lots of interesting stuff, and most of it is open source.

You will notice only a few of his posts are posted here by him - lots of different people.

Everyone can share their blog posts here, and it will only show up on the front page if enough people think it's interesting enough to be there.

Apparently more people (Including myself) think it's interesting vs. uninteresting.

I remember the good old time of Foursquare. What caused its downfall? The checkin thing was quite nice, I even remember some places offering discounts or having special offers to checked in visitors or mayors. Its one of those apps that could still have a vibrant community 10 years later.

I wonder if splitting the app was cause or symptom of downfall.

Nice. Works with clickhouse local cli too.

./clickhouse local -q "select count(*) from s3('s3://fsq-os-places-us-east-1/release/dt=2024-11-19/places/parquet/places-00000.snappy.parquet');"

prints: 4180424

Both are open source. However, Foursquare open source places seems to be Apache 2 licensed where overture is CDLA Permissive 2.0. So not sure if you can port from one to another. It does my head in trying to work out the kind of licensing issues.

I'd imagine foursquare is much better on 'places' like this - overture has other things like building footprints etc.

Why should one contribute to this dataset instead of OpenStreetMap?

Not enough people wanted to check in. Those that did were all of the same demographic. So lots of check-ins at bars / restaurants, none at "boring" shops., doctor's office, etc. Then, like any fad, checking in was no longer fun or cool.

I remember a significant redesign and a lot of features moved to Swarm, which had a different kind of social push. I remember the app losing its usefulness to me and Swarm not being something I wanted to use. Details are hazy, though, it's been a long while.

Neat! Wondering if it's possible to reuse this is OSM?

I was an early user of Foursquare and also tried to contribute fixes back when the Foursquare Android app was still open source(!).

From a end-user's perspective, it was perfect! It was gamified just enough to be engaging but not so much that it would become annoying or a grind. It was tied to the real-world via restaurants and other places offering special discounts when you "checked-in" or became the mayor of a place. It had people contributing essentially-real-time updates to metadata of all places (opening hours, location, description, phone numbers, images, etc) - way before Google Maps user contributions were really a thing. It had a social network aspect to it that actually worked etc etc..

It pains me that Foursquare, the company, failed to figure out how to actually monetize any of this and keep the thing going that made them popular.. so they grasped as straws and did the whole app split thing (WTH, really) and the community just faded out as far as I can tell.

"Places Pro" and "Premium Data" are not open source.

Dennis crowley who led that effort had a difficult time getting his marauder map like vision of the places+people going, not for lack of trying. He was at a NYC event describing his experience. First he tried dogeball which google bought and did not do much with it. He lefy and started FSQ. He mentioned that eventually FSQ pivoted to enterprise as the swarm/checkin like features were not making business sense, it was also copied by FB, Yelp which is when traffic moved away from FSQ's app to those. But FSQ always had a higher committment to location privacy and still does(remember cambridge analytica and all those NYT articles on location tracking)

Dennis is trying something new now, which is true(to him) and to the original vision of Marauder's map and FSQ. He has been on the dogeball train since a long time and hats off to his commitment.

It's a bit bothersome they nudge you to install and use aws tools to download the data, instead of providing a plain CSV or JSON downloadable dump instead.

It's S3 so this works to list the bucket contents:

    curl -s "https://fsq-os-places-us-east-1.s3.amazonaws.com/?list-type=2&prefix=release/dt=2024-11-19/places/parquet/&delimiter=/"

And you can download a file called places-00000.snappy.parquet like this:

    curl -O "https://fsq-os-places-us-east-1.s3.amazonaws.com/release/dt=2024-11-19/places/parquet/places-00000.snappy.parquet"

To be honest, this problem is unsolvable. The only API that's closest is Google Maps, I've just accepted it and gone all-in with them.

I mean not really, why would you want to direct download a CSV or JSON dump this big