I don’t much care for modern “yellow peril” stories. Such stories rely on the fact that China is a foreign country with a billion and a half people in it that still reflexively scares Americans.
It makes it very easy to write alarmist “trend pieces” where a few headlines get blown up into a story about how a whole country is filled with drivers intentionally killing pedestrians they hit to avoid lawsuits or whatnot.
Even so, I approve for the most part of the coverage of China’s Social Credit System, with the government paying tech companies to scour through everything they can find of a citizen’s financial and professional history, use secret-sauce algorithms to turn that history into a flat numerical score, and then make that score available to the general public for the purpose of everyone being able to judge everyone else.
Sure, the coverage is to a degree exaggerated. There are no plans, yet, to use SCS comprehensively crunch the numbers on how liked or disliked you are on social media, how loyal or disloyal you are to the government, how of your friends are the “right” or “wrong” people, etc. But the technology to do so clearly exists and companies are experimenting with “social network credit scores” in many places, including China.
The Chinese government isn’t using Big Data algorithms to automatically detect dissident thought before it happens a la Minority Report, as far as we know, but scouring through your entire financial, professional and legal history to tell all future employers whether you can be trusted with a job is surely scary enough.
China’s SCS is only a mild extrapolation of where our society’s lust for Big Data is already taking us. We’re scared at the thought of that kind of power being in the hands of a foreign authoritarian government, without noticing that that’s power already held by unaccountable private actors here in the “free” West.
Interestingly, China and the United States have this in common–they’re both in favor of relatively unrestricted data-hoovering of their citizens done to build Big Data model of those citizens to predict their behavior and control it with incentives. It’s just that China’s oligopoly of companies doing the data-mining are acting under the explicit direction of the state, whereas in the United States it’s more often the state acting under the implicit direction of the oligopoly.
Complaints about FICO scoring and the concept of a “credit rating” aren’t new. I’ve been following one of my favorite bloggers, Fred Clark, fighting the good fight against the Experian/Equifax/Transunion protection racket for years.
It’s bad enough that, for as long as I’ve been alive, shady corporations have had the power to dig up dirt on me and then charge me for the privilege of seeing or challenging any of that dirt before they use it to possibly ruin my life.
Up till now we’ve had regulations restricting what kind of information can go into a credit report and what kind of information can come out of it.
SCS proposes to ignore such petty consumer protections, vacuuming up everything from points on your driver’s license to negative performance reviews at your last job and spitting out a numerical score that anyone at all can look up — not just potential landlords and employers but potential romantic partners, roommates, neighbors or random strangers who want to attack your reputation.
SCS is drawing attention because it’s being pushed by the same people who erected the Great Firewall of China, people whom the Internet can mobilize against as enemies of freedom, enemies of the future, enemies of the West, etc.
But make no mistake about it. The oppressive world that SCS implies, the Orwellian nightmare of being constantly surveilled and constantly judged, is not an attack on the Internet as it currently exists. It’s the Internet’s logical endpoint. What data-broker sites like Spokeo clumsily try to pretend to do, China simply wants to do for real.
As with so many other complex metaphors from the past, Foucault’s idea of the “Panopticon” in Discipline and Punish has become depressingly easy to explain because the Internet has made it depressingly literal.
Foucault said you can build a prison without walls just by letting your prisoner know he’s always being watched. In real life in 2015, you can totally control someone’s behavior by training them to tweet or instagram every tiny thing they do and see how many likes it gets.
Foucault said you can have a jail without any jailers as long as all the prisoners know there might be someone watching at all times, even if they can’t see who it is. In real life in 2015, we can all be each other’s wardens as members of the amorphous mob that hands out likes and dislikes.
Indeed the only way to not feel totally helpless in the face of the invisible mob’s upvotes and downvotes is to aggressively participate in voting yourself–and so like the crabs in the bucket we all cooperate in crowdsourced self-imprisonment.
Does this sound overwrought? Melodramatic? Does it trigger an urge in you to say “It’s just online BS, just don’t read the comments”?
Well, consider that even though New York City can enact a law against employers pulling credit checks against you, no one can do anything to stop a hiring manager from Google-stalking you.
Consider hiring managers outright admitting that finding negative remarks about you on social media from people they don’t know can still sink your chances of a job offer. Consider the stories of people like Lindsey Stone losing their jobs after a tasteless photo went massively viral. Consider organized attempts to leak information from people’s private lives–nude photos from a romantic relationship, a past career in sex work–in order to poison their reputation in their future careers.
Even though New York City can enact a law against employers pulling credit checks against you, no one can do anything to stop a hiring manager from Google-stalking you.
Look at the world of Yelp reviews, which makes drama over hits to one’s credit rating look tame. Look at the cutthroat warfare of retaliating to bad reviews with bad reviews we see small businesses engaging in. Look at how one possibly faked review can lead to an explosion of one-star reviews and negative press.
Look at how the fear people have of a damaged credit rating leads to profits for companies that sell access to your credit report–now look at the far greater fear companies have of Yelp’s power over them, leading to allegations of “Yelp mafia” tactics of filtering negative reviews in return for cash.
Look at the panic and uproar that erupted over the announcement of Peeple, the “Yelp for People” app, an attempt to bootstrap a version of SCS using purely user-generated content. At this point it almost doesn’t matter if the proposed clearinghouse for people’s “ratings” of other people they know comes out or not.
The idea is simple enough and the reasons it’s frightening are obvious enough that it was already thoroughly mocked in the “Meow Meow Beanz” episode of Community, and yet that didn’t stop Nicole McCullough and Julia Cordray from moving ahead with Peeple.
Even if Peeple’s founders bow to public pressure and scrap the project or try to radically rework it to be purely “positive,” nothing stops the next set of founders from creating a more “neutral” or outright malicious version of Peeple to feed our dark appetite for saying nasty things about each other behind each other’s backs.
It’s the same logic, after all, behind apps like Yik Yak or the subculture around comment sections and chans. No matter how many times we write think-pieces exhorting each other to pay no attention to the voice of the anonymous mob we still find ourselves jerked around by it.
Human nature is to be deeply invested in knowing what “people” in general think, in getting some reified abstract consensus of the population as a whole as a neat tidy number.
Human nature is to be deeply invested in knowing what “people” in general think, in getting some reified abstract consensus of the population as a whole as a neat tidy number. We feel like we’re validated if we get to vote on what “we” as a whole think and we treat that consensus as more valid than any individual opinion, no matter how flawed that concept may be.
Our technology is happy to enable this obsession, aggregating and averaging and quantifying to our heart’s content, turning our collective opinion of restaurants and films and consumer products into tidy integers.
Cory Doctorow, tech prophet of our times, predicted all this back in 2003, with his novel Down and Out in the Magic Kingdom that introduced the concept of “Whuffie.” Doctorow imagined a post-scarcity world where everyone was guaranteed basic food and shelter.
Rather than ownership of material goods, economic life revolved entirely around a “reputation currency” called Whuffie, which, like favs on Twitter or likes on Facebook or stars on Yelp, represented nothing but what other people thought of you.
The Whuffie system promised to deliver us from the arbitrariness of who gets rich and who gets poor in the real-money economy and instead focus on what really mattered, which was your social standing among your peers.
It sounded appealing enough to some people that at TechCrunch 50 in 2009 the Whuffie Bank nonprofit was founded to try to make reputation currency into a reality.
Now, in 2015, the Whuffie Bank is dead and every attempt to implement something like Whuffie–Klout, Peeple, SCS–feels less like a dream and more like a nightmare. Doctorow admits that he got the term “Whuffie” from local slang at his high school for teenage “brownie points.”
Anyone who’s been in high school knows that the way the popularity game works is capricious and often vicious. That the process of scoring “points” is not in any sense meritocratic and can easily be gamed by those unscrupulous and skilled enough to do so, and that if anything a “popularity economy” is one where the rich get richer and the poor get poorer far more reliably than even the real money economy.
Anyone who’s been in high school knows that the way the popularity game works is capricious and often vicious.
One of my favorite responses to Doctorow’s Whuffie is Ben Rosenbaum’s 2010 short story “The Guy Who Worked For Money,” positing that in the reputation economy of the future there will be some of us who long for the good old days of good old fashioned alienated labor and trading toil for coin because for some of us that might be less stressful and scary than spending all our time worrying what everyone thinks of us.
The “guy who works for money” in Rosenbaum’s story does so by fleeing the ubiquitous reputation web in Europe for the un-networked hinterland of “China and the ’stans.”
In the real world of 2015, China is embracing the reputation-based web faster than Europe and “Facebook Zero” is working overtime to get the social web embedded everywhere else. Before long there may be nowhere left to flee.
Sarah Jeong’s The Internet of Garbage argues that despite all the appealing metaphors about size and space and bigness we use about the Internet–the “electronic frontier” and so on–the Internet’s real effect has been to make our world smaller.
The time when you could sell your house and drive to another state to start a new life with new friends and a new reputation is over.
Now, wherever you go and whatever you do, your reputation is what comes up on the first page of Google when someone looks up your name. Now we all live in the same small town, or high school, or prison, depending on your perspective.
I’m high-profile enough that I’ve had the experience of people coordinating to broadcast smears about me and push them up my Google search results. I haven’t gotten it nearly as badly as other people, some of whom have seen their entire careers implode over essentially nothing because they pissed off the wrong dedicated hate mob.
But even that’s not really what concerns me. When quasi-famous people get slimed by other quasi-famous people there’s at least an argument to be had, specific points to contest, evidence to grapple with.
What about when our existing paranoia over how one late payment affects our credit spills over into paranoia over how one errant tweet or photo or comment might give our Whuffie a hit? What about when you don’t even have a list of Google search results to pore over, just an opaque number that jumps upward or downward as unpredictably as the stock market?
Every day we take another step toward that future. The whole world of social networking and the drive to monetize social networking by pulling useful data out of it is inexorably pulling us toward that future. And every step we take in the other direction–the EU’s “right to be forgotten,” one city trying to limit the overuse of credit scores in decision-making, our transient moment of outrage over Peeple–seems laughably weak in comparison.
The EU is stepping up their dissent against the US-led ideal that more access to everyone’s data will always make for a better world. Critics of the EU’s increasingly restrictive privacy policies worry that it will split the Internet into pieces, that it will throw a roadblock the wonderfully economically productive process of shrinking the world into a global village.
Perhaps we tech leaders in the United States need to think about the fact that in opposing such measures, we’re standing against our fellow First World democracies and standing with the authoritarian bogeyman of China’s one-party state.
It’s easy to get called a Luddite these days for advocating any proposal to limit the growth of the social web. But even if there is no stopping or even slowing this train we’re on, it’s at least worth thinking about the downsides of the destination. Even if we have to imagine it happening in Communist China before we get it.