An app being made by DeepMind, the Google-owned AI company, working in collaboration with the NHS Royal Free Trust in London and being used to help identify hospital patients who might be at risk of acute kidney disease (AKI) is not currently in use, TechCrunch has learned.
The collaboration between the tech giant and a portion of the UK’s publicly funded health service has drawn criticism for the breadth of patient data being used to power an app which targets a single medical condition.
DeepMind and the Royal Free have also been criticized for not approaching the UK’s medicines and healthcare devices regulator, the MHRA, prior to using the Streams app in hospitals. The MHRA is responsible for standards of safety, quality and efficacy for healthcare products, which can include software apps.
It has emerged that DeepMind and the Royal Free Trust are now in discussions with the MHRA over whether the Streams app needs to be registered as a medical device.
“We have been in contact with Google since May 4 and are currently in discussions with them about whether or not their app needs to be registered as a device,” a spokesman for the MHRA told TechCrunch.
The spokesman said the project is not currently under formal investigation by the MHRA.
“We’re talking to them about what might be required and what they are doing. I would not call it an investigation,” he added. “There are just various technicalities with apps. It’s not necessarily as clear cut as say medicine so we’re just trying to be clear about what they’re doing, whether or not that constituted being an app.”
DeepMind announced the collaboration to develop an app with the Royal Free NHS Trust back in February. The MHRA was not informed of their plans at this point. However both DeepMind and the Royal Free assert there was no requirement for them to gain prior approval to develop and pilot the app because they say they have not conducted any “clinical trials/investigations”.
A Royal Free spokesman rather says they have carried out small “user tests” of the app.
We’ve asked the MHRA at what point a pilot of a product would be considered to constitute a clinical trial or investigation in its view and will update this post with any response.
The MHRA’s normal procedures mean it can issue a letter of ‘No Objection’ after reviewing an application to run a clinical investigation into a medical device — assuming it does not have any concerns about the proposal. The Streams app has not yet gone through this review process.
With anything with regulation if someone has a discussion with us first then that helps the process. That would be the case with anything.
Asked whether it is normal procedure for product makers to approach the regulator prior to starting a trial, the MHRA spokesman said: “With anything with regulation if someone has a discussion with us first then that helps the process. That would be the case with anything.”
Separately, the UK’s data protection watchdog, the ICO, confirmed to TechCrunch it has received a “small number of complaints” about the Streams app, and is currently looking into it.
“We are aware of this story and are making enquiries. Any organisation processing or using people’s sensitive personal information must do so in accordance with Data Protection Act,” said a spokeswoman.
Direct patient care vs secondary use
Another criticism of the Streams app project has centered on the patient data that is being processed. At the time of the project launch it was also not clear how much data was being passed to the Google-owned company as part of the Streams app project.
However earlier this month, New Scientist obtained a copy of the data-sharing agreement between DeepMind and the Royal Free — which revealed that rather than only getting access to data from patients directly affected by AKI, the agreement in fact shared all hospital admissions data, extending back a full five years. The catchment area for the three London hospitals covers some 1.6 million people.
DeepMind asserts that access to all patient data across the three hospitals is necessary for the app’s predictive function to work. It also claims it is not engaged in research, and says the Streams app is being used for direct patient care — an important distinction because additional regulatory and ethical approvals would likely be necessary if the Google-owned company was performing research on the data-set. Or applying any machine learning algorithms to the data, which it says it is not (although DeepMind co-founder Mustafa Suleyman has suggested that’s something it might like to do in future).
That said, it is clear at this point that the vast majority of the Royal Free patients whose data is being passed to DeepMind via this collaboration have not had, and will never have, AKI. It’s this secondary usage scenario of the data-sharing agreement that has drawn specific criticism from patient data privacy groups, among others, given that the data in question is personally identifiable — which normally, under NHS regulations, can only be shared with third parties with implied consent if it is to be used for direct patient care. i.e. if the person whose data is being shared will directly benefit from the sharing.
With the Streams app it might well be the case that, for example, a patient who lives outside the Trust’s catchment area yet who was rushed to one of the hospital’s A&E departments after an accident ends up having their data shared with the Google-owned company yet will never themselves be in a direct patient care relationship with the doctors who are using the app.
“Direct care is between a clinician and a patient. In this case, the patient who has a blood test, and the clinician who reviews the results. That is not for concern, and DeepMind has the ability to access whatever data needed for that clinical review as part of direct care. But that is on an individual patient basis, not in bulk,” says Sam Smith of patient privacy group MedConfidential.
“What happened though, was they got all SUS [secondary use service] data from the hospital for the last five years plus monthly updates, including data on patients who never had a blood test when they were there, and who will never return to the hospital. What is the direct care relationship for those patients to have their data used by google? I’ve been asking Google that question for a fortnight, and they can’t answer it. Because there isn’t one.”
“Additionally, and separately, what Google refer to as “development work”, is by definition, not direct care. It is entirely fine that Google wanted to use live data to train their decision tree processes; but that process is not direct care. It is a secondary use,” he adds. “Development work is not direct care.
“They would be able to keep some information around those whose data is displayed in the app, etc, but how long it is kept for, what it is used for, etc, would need to be written down somewhere. Denying they need to do it suggests that piece of paper doesn’t exist.”
Again DeepMind and the Royal Free rebut these criticisms, claiming all the data is being used for direct patient care — and therefore that no additional consent or regulatory/ethical approvals are required for the app to be used.
“We believe we have complied with all relevant policies and regulations relating to the collection and processing of patient data,” a spokesman for the Royal Free said in a statement. “Throughout the NHS, patient data is routinely collected and processed by IT companies for the purpose of direct patient care under the principle of implied consent. Our agreement with DeepMind is our standard third-party data sharing agreement, with the trust being the data controller and DeepMind being the data processor.”
A DeepMind spokesperson added in a statement: “We are working with clinicians at the Royal Free to understand how technology can best help clinicians recognise patient deterioration — in this case acute kidney injury (AKI). We have, and will always, hold ourselves to the highest possible standards of patient data protection. Section 251 assent is not required in this case. All the identifiable data under this agreement can only ever be used to assist clinicians with direct patient care and can never be used for research. We and our partners at the Royal Free are in touch with MHRA regarding our development work.”
“Three user tests”
So what then is meant by “development work”? The Royal Free spokesman told TechCrunch that in total three small “user tests” of Streams have been run so far, with each lasting between two and six days, and with a maximum of six clinicians using the app during each test.
The spokesman declined to specify how many patients have been involved in the tests — although given that all three hospitals’ patients data is being fed into the algorithm powering the app then, in theory, all current and past patients (extending back five years) of the hospitals are in some sense ‘involved’ in these tests because their data is being used by the app. In all likelihood the vast majority of these people will be unaware their data is being used for this purpose.
It is also not clear what criteria DeepMind/the Royal Free are using to evaluate their “user tests” of the Streams app. Nor which outside body — if any — is reviewing the tests.
The Royal Free spokesman declined to answer these specific questions, pointing to an online Q&A that was published on the same day the MHRA contacted Google to discuss the app.
In this Q&A the Trust asserts that “a range of patient data must be analysed” in order to “provide diagnostic support and track patient outcomes” — as its explanation for why the data of a person who is currently not an in-patient is being used in the Streams app.
“All data is shared with the purpose of improving patient safety and care,” it adds. “Historical data is used to analyse trends and detect historical tests and diagnoses that may affect patient care.”
Another interesting question here is what exactly is the role of DeepMind in the project? The design of the app was at least partially outsourced (described as ‘co-designed by’) to London based app design studio ustwo, while the algorithm being used to process patients’ data was, we are told, developed by the NHS. So why is a company famed for its artificial intelligence algorithms being engaged to act as, effectively, a project manager for a healthcare app?
In the Q&A the Royal Free says it approached DeepMind “with the aim of developing an app that improves the detection of acute kidney injury (AKI) by immediately reviewing blood test results for signs of deterioration and sending an alert and the results to the most appropriate clinician via a dedicated handheld device”.
It does not provide any more details on why it specifically chose to work with the Google-owned company.
“AKI affects more than one in six in-patients and can lead to prolonged hospital stays, admission to critical care units and, in some cases, death. The Streams app improves the detection of AKI by immediately reviewing blood test results for signs of deterioration and sending an alert and the results to the most appropriate clinician,” it adds.
Asked for his personal views on the data-sharing agreement between the Trust and DeepMind, the Royal Free’s Caldicott Guardian, who is responsible for patient confidentiality and enabling appropriate information sharing, said he is unable to comment without being given approval to do so by the Trust’s communications department. He added that he had “looked into this extensively” — but no further details about that scrutiny have been forthcoming.
The Royal Free spokesman confirmed that the data-sharing agreement between the Trust and DeepMind was signed on behalf of the Trust by its data protection officer, Subir Mondal.