New details emerge about Palantir’s custom software for spy agencies

A new report by The Intercept provides some compelling specifics on collaboration between Palo Alto’s secretive big data shop Palantir and the intelligence agencies that it keeps on its client roster.

While some broad strokes are known about Palantir’s relationship with spy agencies, the finer points seldom see daylight. That guarantee of secrecy no doubt aids the company’s growth into 2017, though its cozy relationship with a presidency keen on implementing aggressive policies with big opportunities for big data certainly can’t hurt, either.

Palantir’s work with state-sponsored spying appears to date back to 2008, when the company demoed its software to the Government Communications Headquarters (GCHQ), the British counterpart to the National Security Agency (NSA). According to classified internal documents quoted in the report, Palantir made quite the impression (“We were very impressed. You need to see it to believe it.”)


Image via The Intercept

Apparently, beyond kicking in early money, intelligence agencies actually worked directly with the company to develop custom software “through [an] iterative collaboration between Palantir computer scientists and analysts from various intelligence agencies over the course of nearly three years.”

The report goes into more detail on the development of two particular software systems, one called Kite and one known as XKEYSCORE Helper. Through Kite, implemented by GCHQ, Palantir offered deep customization:

“Out of the box, Kite was able to handle a variety of types of data (including dates, images, geolocations, etc.), but GCHQ was free to extend it by writing custom fields for complicated types of data the agency might need to analyze. The import tools were designed to handle a variety of use cases, including static data sets, databases that were updated frequently, and data stores controlled by third parties to which GCHQ was able to gain access.”

Designed in apparent collaboration with Palantir, XKEYSCORE Helper sought to make data obtained through the NSA’s muscular XKEYSCORE program more digestible and offered a way to port it into Palantir’s more analyst-friendly interface:

“This collaborative environment also produced a piece of software called “XKEYSCORE Helper,” a tool programmed with Palantir (and thoroughly stamped with its logo) that allowed analysts to essentially import data from the NSA’s pipeline, investigate and visualize it through Palantir, and then presumably pass it to fellow analysts or Five Eyes intelligence partners.”

Palantir, founded in 2004 by Alex Karp and Peter Thiel, remains something of an enduring mystery in Silicon Valley. In 2013, TechCrunch obtained a closely guarded list of the company’s clients, revealing relationships with the CIA, DHS, FBI, NSA, Marine Corps and Air Force, among others. Though the company continues to grow, much of what it does remains under wraps, or understood mostly in abstract terms.

While the picture is far from complete, every detail that emerges about Palantir’s mission, software implementation and client base helps piece together the puzzle of perhaps the most politically relevant company in technology right now.