Apple still has work to do on privacy

There’s no doubt that Apple’s self-polished reputation for privacy and security has taken a bit of a battering recently.

On the security front, Google researchers just disclosed a major flaw in the iPhone, finding a number of malicious websites that could hack into a victim’s device by exploiting a set of previously undisclosed software bugs. When visited, the sites infected iPhones with an implant designed to harvest personal data — such as location, contacts and messages.

As flaws go, it looks like a very bad one. And when security fails so spectacularly, all those shiny privacy promises naturally go straight out the window.

And while that particular cold-sweat-inducing iPhone security snafu has now been patched, it does raise questions about what else might be lurking out there. More broadly, it also tests the generally held assumption that iPhones are superior to Android devices when it comes to security.

Are we really so sure that thesis holds?

But imagine for a second you could unlink security considerations and purely focus on privacy. Wouldn’t Apple have a robust claim there?

On the surface, the notion of Apple having a stronger claim to privacy versus Google — an adtech giant that makes its money by pervasively profiling internet users, whereas Apple sells premium hardware and services (including essentially now ‘privacy as a service‘) — seems a safe (or, well, safer) assumption. Or at least, until iOS security fails spectacularly and leaks users’ privacy anyway. Then of course affected iOS users can just kiss their privacy goodbye. That’s why this is a thought experiment.

But even directly on privacy, Apple is running into problems, too.

To wit: Siri, its nearly decade-old voice assistant technology, now sits under a penetrating spotlight — having been revealed to contain a not-so-private ‘mechanical turk’ layer of actual humans paid to listen to the stuff people tell it. (Or indeed the personal stuff Siri accidentally records.)

In recent months very similar privacy concerns have struck rival voice assistants, too — including AIs operated by Amazon, Facebook, Google and Microsoft, all of whom, it has become clear, also rely on human ears and brains to make their AIs less stupid.

It’s also true that — prior to this unmasking by journalists — none of these tech companies made any effort to make sure users of their products were aware that behind AI’s fancy facade human ears are listening.

Facebook’s bleated excuse about having human workers ‘enhance’ a voice-to-text transcription service, which it offers as a feature to Messenger users, is that it’s just standard industry practice. (Translation: Everyone spies.)

Setting what are (generally) terrible industry privacy standards aside for a moment, the Siri rat king revelations do remain especially awkward for Apple. Because, unlike the rest, it sells a substantive promise of privacy — not bogus marketing claims of ‘respecting’ privacy (whilst simultaneously and systematically mining personal stuff for profile and profit).

Apple still makes most of its money from selling expensive hardware. So it’s no accident that in recent years its marketing has often very boldly associated buying Apple with ‘buying privacy’ — implying there’s more to the ‘Apple premium’ than high sheen and expensive branding.

The plication is you’re getting privacy peace of mind, so it really is supposed to be better than that. A lot better.

After all, this is also a company that’s made a public show of supporting privacy as “a fundamental human right”. It has also said it has a “responsibility” to develop software which respects that right — an onus to “achieve both great artificial intelligence and great privacy standards,” as CEO Tim Cook put it in a speech at a major conference of data protection experts in Brussels last year.

Yet, at the coal-face of trying to make Siri pass as intelligent, Apple appears to have forgotten its pledge to put ‘human values at the heart of engineering’ — unless that particular talking point was actually a coded disclosure for (also) having human ears hard at work inside its listening machines.

Respecting people’s privacy certainly doesn’t mean hiding the fact your system doesn’t respect every person’s privacy. Really the golden rule when it comes to other people’s information is don’t ever just do — always ask if it’s okay first.

It’s both surprising and not surprising that Apple so spectacularly failed to ask in this case. While Apple talks big on privacy, as a company it affords itself administrator privileges over users; taking decisions on their behalf and exercising a high degree of control over products in order, it would say, to deliver the best experience.

It’s an approach that is naturally criticized across the OS aisle for control freakery and serving up hermetically sealed products. But, on the flip side, it can deliver mainstream user benefits — especially when combined with Apple’s historical design flair for simplifying technical processes and making them more widely accessible. Even if, over time, some complexity has crept back, and some of its more ‘courageous’ design decisions haven’t been a delight to every Apple user.

There are other risks and pitfalls, too. The Apple power dynamic always works by keeping users at least slightly in the dark. It operates with an assumption that Apple knows best that risks veering into arrogance and insensitivity — which can in turn be expressed as a rigid operational style in which Apple’s decisions discount minority interests.

The problem where Siri is concerned is that Apple allowed its desire to produce a better product experience for all users to override the privacy rights of some of the users. That’s not right.

And while all of big tech is apparently right there where AI and privacy is concerned, Apple has actually come out slightly worse than some of the usual data-mining tech giants in this instance — because it did not even disclose the existence of what it euphemistically refers to as an audio ‘grading’ program to users. Not even deep in the Siri Terms and Conditions. It kept the fact that some users’ audio is listened to by contractors conveniently to itself.

Apple’s choice of term — “grading” — makes it sound like something farm workers do with vegetables, when it actually refers to humans eavesdropping on goings-on inside the homes of owners of its expensive devices. (So ‘WTAF’ might be a more appropriate moniker.)

Privacy isn’t respected in obscurity either.

Apple has been careful to state that <0.2% of Siri recordings passed through its audio review process. But that looks like an attempt to shrink the perceived size of the scandal. It may sound statistically insignificant but still translates into (potentially) tens of millions of snippets listened to without people’s consent — given Siri has 15 billion requests per month.

In any case, when you’re talking about privacy it does not make it okay for any of the people whose privacy has been torched to tell them they are in a tiny minority. In fact, they might even be more pissed to hear that.

Privacy is personal and individual. Which is why choices about privacy must always be passed back to the user.

But — and now to Apple’s claimed reform for Siri ‘grading’ ‚ it has said it will do this in future.

Its initial response to the Guardian‘s exposé of sensitive Siri snippets being listened to by contractors was to suspend audio reviews entirely. It followed up with an announcement of reform — saying it would change how the program works. Users would by default be opted out and would have to actively choose to opt in. When the program relaunches in the fall, it will also bring the eavesdropping in-house.

So no more remote contract workers who might leak awkward details of less-than-best privacy practice to the press, risking tainting the claims it makes for its premium brand.

In a blog post about the changes — to which Apple has applied title-spin straight out of the standard big tech crisis PR playbook (hence we read: “Improving Siri’s privacy protections”, not: ‘sorry we really messed up Siri privacy’) — it apologized for not living up to its own “high ideals” but also expends most of its energy trying to put the cork back in the ‘we’re really great about privacy’ bottle, including dedicating the entire second section of the post to promoting “How Siri protects your privacy”:

Apple is committed to putting the customer at the center of everything we do, which includes protecting their privacy,” it finishes up, seeking to reboot its privacy cred. “We created Siri to help them get things done, faster and easier, without compromising their right to privacy. We are grateful to our users for their passion for Siri, and for pushing us to constantly improve.

The changes Apple says it has been ‘pushed’ to make to Siri’s mechanical turk — no audio data retained by default (though it might still review computer-generated transcripts without user consent); an opt in for audio reviews should the user wish to contribute to Siri improvement which can be revoked at any time; opt in audio samples exclusively reviewed by Apple employees in-house; and a pledge that staff will “work to delete any recording which is determined to be an inadvertent trigger of Siri” — represent a reset to a privacy baseline that Apple users would be forgiven for thinking they should have been enjoying already.

Yes, the switch to asking for and storing consent to review Siri users’ audio is certainly welcome — but the episode remains an awkward stumble for Apple. A glitch in its highly polished privacy ideals.

However, the Siri privacy story doesn’t end there.

More problems are looming for Apple at the intersection of AI and privacy rights, plural, as its paternalistic approach toward users appears to be on a collision course with the full package of rights enjoyed by Europeans under the bloc’s comprehensive data protection framework.

Apple’s treatment of Siri data is the target of (at least) two EU privacy complaints, one of which dates back to 2017, and the other filed since the EU’s General Data Protection Regulation (GDPR) came into force last year. They assert that Apple is failing to respect European Siri users’ fundamental data access rights by not fulfilling requests to let them access Siri audio data it holds on them.

Even though Apple claims to now be deleting data from Siri on behalf of users including with all historically retained audio snippets, that’s not the same as letting individuals control what happens with their data.

Sure, EU users might want to delete it all. But, equally, they might want to review their Siri recordings first and/or download them before hitting the bin button.

This should be possible under EU privacy law because it places a strong emphasis on data access rights in order to provide individuals with an important layer of control over information held on them.

Article 12 of the EU’s Data Protection Directive (which predates GDPR) includes rights of rectification, erasure and blocking of data processing — with an emphasis on enabling citizens to act themselves to correct incomplete and inaccurate data held on them (the wrong information linked to a person’s identify can of course be harmful in myriad ways).

While the GDPR expands these core data access rights. Article 15 says data subjects can also request a copy of the data held on them. And Article 20 — which confers a ‘right to data portability’ — says data must be provided in a structured, commonly used and machine-readable format to facilitate easier transfers to another data controller.

In a nutshell, EU privacy law is designed to give users both privacy and control over their personal data. With the potential added benefit of fostering competition by supporting service switching, in the case of portability.

Where this loops back to Apple and Siri is that even under the reformed grading program for Siri, Apple is only saying users can be confident it will ensure the confidentiality of their information. (And they still have to trust that Apple staffers won’t leak their stuff.)

Apple is still not letting Siri users have access to their recordings so they could control the data themselves. Essentially it’s saying ‘trust us to control it for you’. That’s just not how data access rights work.

Two of the European privacy researchers behind the Siri complaints, Michael Veale and Jef Ausloos, also co-authored a research paper last year in which they discuss instances where they see a clash between data protection by design (also an imperative in EU privacy law) and data subject access rights.

Siri is cited in the paper as a case study as a result of Apple denying their data access request — with the company citing ‘privacy-by-design’ as its reason for not providing their data. (The paper points out that Google, with a similar voice assistant service, does offer a tool that let users access and manage their voice and audio data.)

In the paper the researchers identify what they describe as “significant conceptual flaws” in Apple’s claims of ‘privacy by design’, and pick apart its justifications for not being able to provide data — such as saying it uses random identifiers, rather than the usual Apple ID, to link Siri data to a device. They point out though that such identifiers are still persistent. (So, essentially, Apple could link the data to a user but has chosen not to build a system to do so — and “refusing to build a database retrieval tool is no basis on which to refuse data subject rights”.)

They also suggest that for Siri users the risk of re-identification should their data leak is likely to be persistent. (“[A] significant body of research has demonstrated that individuals can be re-identified and clustered by voiceprints alone, which have such re-identification potential that they are being used and proposed for biometric authentication. Apple themselves even possess several patents in this area from their own in-house research activities. Even based on text transcripts without the voice data, researchers have demonstrated attacks that can re-identify or cluster individuals stylometrically, based on the words and grammar they use.”)

They also argue that Siri is likely to record not just personal data but sensitive personal data, including special category data such as political opinions, given the always listening, trigger-word design of the technology. Which clearly dials up the privacy risk.

“Without guarantees that private and re-identifiable parts of a conversation have been redacted, which seem technically difficult, if not currently impossible, to provide, little assurance can be given,” they conclude of Apple’s ‘privacy by design’ claim.

TechCrunch has confirmed that both Siri data access complaints are with Ireland’s Data Protection Commission. But despite the regulator having had years to look at one of them — in the case of Veale’s 2017 complaint — their current status remains unclear.

Asked for a progress update, the watchdog told us: “The DPC is currently engaging with Apple in relation to the complaints it has received in relation to data access rights for users to Siri data”.

In Veale’s view, Apple’s problem is that while it may be very comfortable with privacy as confidentiality it’s totally out of its comfort zone with privacy as control. Yet that is the long-standing legal standard in Europe. So it has work to do.

“Apple stores voice recordings and estimated transcripts alongside a device identify that is generated when Siri is turned on for the first time. It should be trivially easy to prove that the phone or device belongs to you, transmit the device [identity] back to Apple, and take control of the data they hold — which may even be you dictating a message that would otherwise be end-to-end encrypted. Apple has always denied this right, but for a truly absurd and flabbergasting reason. Initially stated in correspondence to me, and later confirmed with them, Apple denied that any of this data falls within the material scope of the GDPR,” Veale told TechCrunch.

“In a herculean leap of logic, Apple claims that they do not store this data in a “filing system” — an exemption that actually does not apply to any automated processing of data, but only applies to paper records. They effectively say they do not wish to build in a feature to access data, and that wish is itself an expression of their dedication to privacy. But their dedication to privacy is as confidentiality, not, as data protection imagine that, as control. They do not agree with the premise that control of data that is directly identifiable is a good thing for users, and wish to ignore the law’s existence entirely.”

“Privacy as confidentiality is good thing,” he added. “Companies can and should do more with less data. But what we see here is a system where users’ data can be trivially identified back to them. It’s not good privacy engineering at all. The privacy engineering Apple is relying on is their own good will not to identify users, not cryptographic black magic. That’s not privacy engineering at all, that’s marketing. And if you can identify users, you have to give them access to their data. The law applies — yes Apple, also to you.”

It’s clear that cloth-eared AIs aren’t automagically intelligent enough to distinguish between personal stuff humans intend them to hear and personal stuff they’re overhearing and therefore shouldn’t be hearing at all. But, either way, all these recordings are personal data. Trying to claim otherwise is just specious.

It follows then that Apple’s reform of Siri privacy needs to go further.

Acting like a control freak over other people’s data isn’t a good look nor a position Apple should be seeking to sustain.

And if in future — triggered by EU privacy complaints — it ends up having to expand the rights it gives European users over their Siri snippets, it will be another kind of awkward stumble should Apple be shown choosing to intentionally withhold the same controls for Siri users more broadly, including for users in the US. That would constitute another glitch in its privacy-respect matrix.

“Apple currently provide data protection rights in a portal that is attached to your user ID. The Siri device identifier is not attached to your user ID. To attach it would require an extra step. It seems to me that Apple does not find it elegant to follow the law if it means they can’t elegantly display rights in a single location. But the law’s the law, and a slightly psychopathic approach to design does not change that,” Veale adds.

We put a number of questions to Apple about data access rights for Siri recordings but the company did not respond to our requests for comment.

Zooming back out, what about iOS vs Android security? Should iPhone users be looking a little more critically at the sheen on Apple’s superior security claims too?

Well, that depends.

“This problem is really nuanced because there are many facets of security,” says Dr Lukasz Olejnik, an independent cybersecurity and privacy researcher, and research associate at Oxford University’s Center for Technology and Global Affairs, who says the perception of iPhone as more secure than Android stems in part from the historical difficulty of getting updates to Android devices, as a result of greater platform fragmentation, in addition to vulnerabilities in the past.

“Limiting to this view, of iOS/iPhone being a more controlled and sealed ecosystem, it was easier to design software and hardware simultaneously, including for issuing updates,” he says. But he also argues Android has improved.

“While iOS is indeed a secure ecosystem, recently Android has also improved greatly — particularly when used in conjunction with [Google’s] Pixel hardware. Today, the situation is that both operating systems seem to stand pretty well when it comes to security.”

“It is true that iOS exploits may be officially more expensive [to carry out] than Android but as we have seen this is very relative and depends also on the number of targets of interest you can reach with a single exploit kit,” he adds. “The relative risk to certain users also varies, as well as the attacker intention and motivation. This is perfectly highlighted by the recent disclosure where a multi-year expenditure was made to target particular groups of users.

“The fact of the matter is that today all systems have vulnerabilities, and all can be hacked. But it may also be economically easier to target monocultures. When you have the same software installed on basically all the servers — think the old OpenSSL cases — targeting it makes sense, and can yield a good return on investment.

“Accidentally, some potentially high-value targets may use iPhone. And going along this high-level reasoning that prioritizes attacker intent, if the targets are iPhone users, it should not be surprising that the attacker will need to go after iPhone.”

The latest grave iPhone exploit — gleefully unearthed by Google researchers — “vividly highlights” the fact economic motivations for penetrating platforms are complex, says Olejnik.

“Keeping this all in mind the importance of overall platform security to raise the bar for attackers cannot be understated,” he adds.

So Google needling Apple over iPhone malware might be just the competitive push the company also needs to be kept on its security toes.