Reflections on Palantir

When I interned at Palantir (summer 2014) their business was mostly in data ingestion, visualization, and correlation.

A typical workflow for a Palantir customer was that Palantir would come in and dump a ton of data out of old crufty databases and into Palantir's datastore. Then, they'd establish connections between that data. This is all sounds kind of hand-wavy, but the gist of it is that a lot of government agencies have data that lives in separate databases and they can't easily correlate data between those two databases. Once the data was in Palantir's system, they could do queries against all their data, and make connections and correlations that they wouldn't otherwise be able to find when the data was previously siloed.

One of the sample use cases was identifying people filling prescriptions for schedule II drugs multiple times on the same day, and correlating that with pharmacies run by people connected to known drug traffickers. Previously, this was hard to do because the database of prescription purchases was disconnected from the database of drug convictions.

So it’s hygiene and structure

That, and a really powerful visualization suite. In the example I gave above, you could plot the prescription purchases on a map and see that people were driving along the highway and hitting up pharmacies along the interstate. Better yet, you could drop into Google Street view in front of one of the pharmacies, and look at it from the street level and see that it doesn't even have signage indicating it's a pharmacy.

IIRC part of it is that the software itself can make connections between separate data sets. You're not just ingesting data about purchasing information and drug convictions and so on, you're getting automatic relationship detection. For example, figuring out that the cust_ss_num field in one dataset correlates to the conv_ssn_full field in another dataset, and knowing that those fields are the "SSN" field from a third dataset, and being able to automatically give you a view where those three datasets are correlated. This saves people having to go through every data set and manually map each field to each other equivalent field in each other related dataset.

I could be mistaken, but I think this is how it was explained to me originally.

That makes sense and sounds really useful

In many of the enterprise orgs I've worked in, the two tech teams that are chronically understaffed are 1) info sec, 2) DBA/ data architecture/ data science. I'm lumping those 3 together on purpose, because they're always understaffed and typically not empowered to build anything.

So if they are dumping data out of old crufty databases and into Palantir's datastore, which one is the active database going forward? In 2024.

You're right to group Data teams together. They seem to share a common plight.

In my experience, internal employees outside Data have a funny relationship with Data. They hate to manage it but they love to blame it, especially in analytical / decision-making scenarios. Teams that "own" the data usually get the blame, on top of having to deal with a mass of rotting pipes and noncompliant teams, while also losing out on credit when non-Data teams report big wins.

Based on what the GP says, it sounds like Palantir knows how to exploit common internal politics around Data. They build up technical & social expertise in ETL'ing disparate data sources, and they can avoid blame by being hired by executives as an external third party.

…particularly to German chancellors in the 1930s.

People dismiss this type of work as no big deal, but in my experience this is the actual hard work of producing something useful for companies, and what 90% of SaaS resellers will never be able to deliver on.

Yes, it is very hard. But does Palantir succeed? Or do they like some other companies just trick customers with big wallets to buy?

I used it quite a bit early on during military operations. The ability to see the timing component was key; not only would you plot the purchase locations, but you could play the timeframe of records, work out the timing so you knew the order in which they visited the locations, where they must have stopped for gas along the route. In a classic workflow, you'd then investigate the gas stations, attach them to the event with confidence intervals, pull CCTV footage, see if you can get a payment receipt, and enter all of that data back into palantir. A few days of doing this, and you can build up all a map of every aspect of the drug run; the who what when where and why. It's a fantastic organization system.

To me it seems they do https://logicmag.io/commons/enter-the-dragnet/

Building a panopticon is always justified as a way to fight crime and then becomes a way to control the population. Tracking women getting Plan B, tracking people buying birth control, etc.

I appreciate the technical achievements here. However, I wonder how long before it’s standard practice to track all peoples movement, not just those suspected of a crime. I know of at least one YouTube channel that is always recording all traffic camera streams in Washington so there must be some State entities doing the same. Back in 2020 there was a twitch channel that would play a 9x9 grid of all the livestream footage from the George Floyd protests. I’m sure an archive of that exists somewhere on a LE server.

We used them at a bulge bracket investment bank and they failed miserably...

nsa is storing everything

Visualization is a fancy package. Nobody looks at visualizations, but that's what makes people buy.

So basically data warehousing, and making it possible to do joins?

Super boring, but super important stuff, which I've seen neglected at far too many places I've worked.

Sounds like data engineering with a dash of ML.

The impression I get from their involvement at one company I know of is that it’s very much the latter. I was pretty surprised to see them behaving and performing about the same as any parasitic enterprise software vendor with an integration services arm. One wonders how different they really are, and if maybe they just have very good PR and marketing.

Chalk it up as yet another case of some famous one-would-suppose impressive entity, or strata of a company hierarchy, or whatever, turning out to be pretty average, or even below average. You’d think I’d stop being surprised by now.

Then again, maybe I was just seeing their B-team.

This is exactly what I thought TFA was getting at when it brought up politics being a problem at companies and in sectors Palantir engages with, but instead it went a much more general direction.

This is a good post to explain the value proposition. It sounds like "Big Data" from the 1990s, but a very good salesperson was able to infiltrate some US gov orgs to sell the idea.

I’ve known companies to spend stupid amounts of money on fake, fancy “war rooms” they staff with people doing nothing useful, filled with “big board” style maps and shit, big graphs and visualizations that aren’t used anywhere else, just as a sales tool. Walk the visiting CEO through, let them pretend what they’re involved in is way cooler and more interesting and important than it really is, and I guess that assists sales so much that such endeavors make way more money than they cost.

I connect this with comments I heard from several major management consulting firm folks stating bluntly that the best way to communicate effectively with execs is to approach them like young children.

Life is super weird. Who knew imaginative play would be such a big thing for “serious” adults? I’d never have imagined, but it’s kinda everywhere.

All of them, plus whatever next vendor they “migrate” to in three years (I’m being generous).

He talks about it a little:

> Why is data integration so hard? The data is often in different formats that aren’t easily analyzed by computers – PDFs, notebooks, Excel files (my god, so many Excel files) and so on. But often what really gets in the way is organizational politics: a team, or group, controls a key data source, the reason for their existence is that they are the gatekeepers to that data source, and they typically justify their existence in a corporation by being the gatekeepers of that data source (and, often, providing analyses of that data). [3] This politics can be a formidable obstacle to overcome, and in some cases led to hilarious outcomes – you’d have a company buying an 8-12 week pilot, and we’d spend all 8-12 weeks just getting data access, and the final week scrambling to have something to demo.

I think he's seen more companies without talented Data experts than companies with that talent.

This makes it sound like their actual product is to be a bureaucratic/departmental door rammer?

Because the ostensible product, at least in the ‘pilot’, produced in just a single week, seems like it is pretty much guaranteed to be bad.

It’s also something you’ll never do in-house because there’s too many politics involved in getting everyone to give up their data.

Then when the CEO hires Palantir suddenly everyone has to.

Huh, I feel like we have the opposite issue. We have all those teams and I’m not sure what they’re actually doing.

This all just sounds like any other consulting company, really?

They should help the business with the evidence to make all kinds of decisions, and in a platform-team kind of way help you self serve data needed to make decisions in your team.

My sense working at an adjacent company and having talked with folks there is that they are more successful with their government projects than their corporate ones.

I’ve heard you often get the A Team coming up with the plan and making the sale and then the B Team doing the actual implementation which surprise! doesn’t live up to the A Team hype. Not specific to Palantir.

Never seen people internally guard their data that much.

But who is going to do the heavy lift? who is going to get billed for that? who is paying for the cloud space, or licenses? absolute holy war.

no problems getting people into the data lakes, but if you want us to do anything useful with it you gotta pay / get people / get resources. but like, you want me to approve the read access or pull request? no problem, have at it.

I used to get paid to develop those war room monitoring solutions. literally just crafting dashboards that no one would ever look at directly, but just sorta had around.

> execs is to approach them like young children.

lots of images. bright colors. no more than 3 bulletpoints per slide. no more than 4 minutes to get to the point, and be unambiguous about what and why.

> approach them like young children

To take a generous go at this - my guess is that they have multiple urgent issues they're dealing with at any one time, and so the cognitive bandwidth they're able to dedicate to 'random presentation number 3 for the day' is quite low

But I do agree that a lot of day-to-day work is play acting at being cooler than our actual work.

Perhaps with the additional level of security clearance that government entities are able to hand over very sensitive data sets.