Media & Entertainment

UK outs extremism blocking tool and could force tech firms to use it


The UK government’s pressure on tech giants to do more about online extremism just got weaponized. The Home Secretary has today announced a machine learning tool, developed with public money by a local AI firm, which the government says can automatically detect propaganda produced by the Islamic State terror group with “an extremely high degree of accuracy”.

The technology is billed as working across different types of video-streaming and download platforms in real-time, and is intended to be integrated into the upload process — as the government wants the majority of video propaganda to be blocked before it’s uploaded to the Internet.

So yes this is content moderation via pre-filtering — which is something the European Commission has also been pushing for. Though it’s a highly controversial approach, with plenty of critics. Supporters of free speech frequently describe the concept as ‘censorship machines’, for instance.

Last fall the UK government said it wanted tech firms to radically shrink the time it takes them to eject extremist content from the Internet — from an average of 36 hours to just two. It’s now evident how it believes it can force tech firms to step on the gas: By commissioning its own machine learning tool to demonstrate what’s possible and try to shame the industry into action.

TechCrunch understands the government acted after becoming frustrated with the response from platforms such as YouTube. It paid private sector firm, ASI Data Science, £600,000 in public funds to develop the tool — which is billed as using “advanced machine learning” to analyze the audio and visuals of videos to “determine whether it could be Daesh propaganda”.

Specifically, the Home Office is claiming the tool automatically detects 94% of Daesh propaganda with 99.995% accuracy — which, on that specific sub-set of extremist content and assuming those figures stand up to real-world usage at scale, would give it a false positive rate of 0.005%.

For example, the government says if the tool analyzed one million “randomly selected videos” only 50 of them would require “additional human review”.

However, on a mainstream platform like Facebook, which has around 2BN users who could easily be posting a billion pieces of content per day, the tool could falsely flag (and presumably unfairly block) some 50,000 pieces of content daily.

And that’s just for IS extremist content. What about other flavors of terrorist content, such as Far Right extremism, say? It’s not at all clear at this point whether — if the model was trained on a different, perhaps less formulaic type of extremist propaganda — the tool would have the same (or worse) accuracy rates.

Criticism of the government’s approach has, unsurprisingly, been swift and shrill…

The Home Office is not publicly detailing the methodology behind the model, which it says was trained on more than 1,000 Islamic State videos, but says it will be sharing it with smaller companies in order to help combat “the abuse of their platforms by terrorists and their supporters”.

So while much of the government anti-online-extremism rhetoric has been directed at Big Tech thus far, smaller platforms are clearly a rising concern.

It notes, for example, that IS is now using more platforms to spread propaganda — citing its own research which shows the group using 145 platforms from July until the end of the year that it had not used before.

In all, it says IS supporters used more than 400 unique online platforms to spread propaganda in 2017 — which it says highlights the importance of technology “that can be applied across different platforms”.

Home Secretary Amber Rudd also told the BBC she is not ruling out forcing tech firms to use the tool. So there’s at least an implied threat to encourage action across the board — though at this point she’s pretty clearly hoping to get voluntary cooperation from Big Tech, including to help prevent extremist propaganda simply being displaced from their platforms onto smaller entities which don’t have the same level of resources to throw at the problem.

The Home Office specifically name-checks video-sharing site Vimeo; anonymous blogging platform (built by messaging platform Telegram); and file storage and sharing app pCloud as smaller platforms it’s concerned about.

Discussing the extremism-blocking tool, Rudd told the BBC: “It’s a very convincing example that you can have the information that you need to make sure that this material doesn’t go online in the first place.

“We’re not going to rule out taking legislative action if we need to do it, but I remain convinced that the best way to take real action, to have the best outcomes, is to have an industry-led forum like the one we’ve got. This has to be in conjunction, though, of larger companies working with smaller companies.”

“We have to stay ahead. We have to have the right investment. We have to have the right technology. But most of all we have to have industry on our side — with industry on our side, and none of them want their platforms to be the place where terrorists go, with industry on side, acknowledging that, listening to us, engaging with them, we can make sure that we stay ahead of the terrorists and keep people safe,” she added.

Last summer, tech giants including Google, Facebook and Twitter formed the catchily entitled Global Internet Forum to Counter Terrorism (Gifct) to collaborate on engineering solutions to combat online extremism, such as sharing content classification techniques and effective reporting methods for users.

They also said they intended to share best practice on counterspeech initiatives — a preferred approach vs pre-filtering, from their point of view, not least because their businesses are fueled by user generated content. And more not less content is always generally going to be preferable so far as their bottom lines are concerned.

Rudd is in Silicon Valley this week for another round of meeting with social media giants to discuss tackling terrorist content online — including getting their reactions to her home-backed tool, and to solicit help with supporting smaller platforms in also ejecting terrorist content. Though what, practically, she or any tech giant can do to urge co-operation from smaller platforms — which are often based outside the UK and the US, and thus can’t easily be pressured with legislative or any other types of threats — seems a moot point. (Though ISP-level blocking might be one possibility the government is entertaining.)

Responding to her announcements today, a Facebook spokesperson told us: “We share the goals of the Home Office to find and remove extremist content as quickly as possible, and invest heavily in staff and in technology to help us do this. Our approach is working — 99% of ISIS and Al Qaeda-related content we remove is found by our automated systems. But there is no easy technical fix to fight online extremism.

“We need strong partnerships between policymakers, counter speech experts, civil society, NGOs and other companies. We welcome the progress made by the Home Office and ASI Data Science and look forward to working with them and the Global Internet Forum to Counter Terrorism to continue tackling this global threat.”

A Twitter spokesman declined to comment, but pointed to the company’s most recent Transparency Report — which showed a big reduction in received reports of terrorist content on its platform (something the company credits to the effectiveness of its in-house tech tools at identifying and blocking extremist accounts and tweets).

At the time of writing Google had not responded to a request for comment.

More TechCrunch

Zen Educate, an online marketplace that connects schools with teachers, has raised $37 million in a Series B round of funding. The raise comes amid a growing teacher shortage crisis…

Zen Educate raises $37M and acquires Aquinas Education as it tries to address the teacher shortage

“When I heard the released demo, I was shocked, angered and in disbelief that Mr. Altman would pursue a voice that sounded so eerily similar to mine.”

Scarlett Johansson says that OpenAI approached her to use her voice

A new self-driving truck — manufactured by Volvo and loaded with autonomous vehicle tech developed by Aurora Innovation — could be on public highways as early as this summer.  The…

Aurora and Volvo unveil self-driving truck designed for a driverless future

The European venture capital firm raised its fourth fund as fund as climate tech “comes of age.”

ETF Partners raises €284M for climate startups that will be effective quickly — not 20 years down the road

Copilot, Microsoft’s brand of generative AI, will soon be far more deeply integrated into the Windows 11 experience.

Microsoft wants to make Windows an AI operating system, launches Copilot+ PCs

Hello and welcome back to TechCrunch Space. For those who haven’t heard, the first crewed launch of Boeing’s Starliner capsule has been pushed back yet again to no earlier than…

TechCrunch Space: Star(side)liner

When I attended Automate in Chicago a few weeks back, multiple people thanked me for TechCrunch’s semi-regular robotics job report. It’s always edifying to get that feedback in person. While…

These 81 robotics companies are hiring

The top vehicle safety regulator in the U.S. has launched a formal probe into an April crash involving the all-electric VinFast VF8 SUV that claimed the lives of a family…

VinFast crash that killed family of four now under federal investigation

When putting a video portal in a public park in the middle of New York City, some inappropriate behavior will likely occur. The Portal, the vision of Lithuanian artist and…

NYC-Dublin real-time video portal reopens with some fixes to prevent inappropriate behavior

Longtime New York-based seed investor, Contour Venture Partners, is making progress on its latest flagship fund after lowering its target. The firm closed on $42 million, raised from 64 backers,…

Contour Venture Partners, an early investor in Datadog and Movable Ink, lowers the target for its fifth fund

Meta’s Oversight Board has now extended its scope to include the company’s newest platform, Instagram Threads, and has begun hearing cases from Threads.

Meta’s Oversight Board takes its first Threads case

The company says it’s refocusing and prioritizing fewer initiatives that will have the biggest impact on customers and add value to the business.

SeekOut, a recruiting startup last valued at $1.2 billion, lays off 30% of its workforce

The U.K.’s self-proclaimed “world-leading” regulations for self-driving cars are now official, after the Automated Vehicles (AV) Act received royal assent — the final rubber stamp any legislation must go through…

UK’s autonomous vehicle legislation becomes law, paving the way for first driverless cars by 2026

ChatGPT, OpenAI’s text-generating AI chatbot, has taken the world by storm. What started as a tool to hyper-charge productivity through writing essays and code with short text prompts has evolved…

ChatGPT: Everything you need to know about the AI-powered chatbot

SoLo Funds CEO Travis Holoway: “Regulators seem driven by press releases when they should be motivated by true consumer protection and empowering equitable solutions.”

Fintech lender SoLo Funds is being sued again by the government over its lending practices

Hard tech startups generate a lot of buzz, but there’s a growing cohort of companies building digital tools squarely focused on making hard tech development faster, more efficient and —…

Rollup wants to be the hardware engineer’s workhorse

TechCrunch Disrupt 2024 is not just about groundbreaking innovations, insightful panels, and visionary speakers — it’s also about listening to YOU, the audience, and what you feel is top of…

Disrupt Audience Choice vote closes Friday

Google says the new SDK would help Google expand on its core mission of connecting the right audience to the right content at the right time.

Google is launching a new Android feature to drive users back into their installed apps

Jolla has taken the official wraps off the first version of its personal server-based AI assistant in the making. The reborn startup is building a privacy-focused AI device — aka…

Jolla debuts privacy-focused AI hardware

The ChatGPT mobile app’s net revenue first jumped 22% on the day of the GPT-4o launch and continued to grow in the following days.

ChatGPT’s mobile app revenue saw its biggest spike yet following GPT-4o launch

Dating app maker Bumble has acquired Geneva, an online platform built around forming real-world groups and clubs. The company said that the deal is designed to help it expand its…

Bumble buys community building app Geneva to expand further into friendships

CyberArk — one of the army of larger security companies founded out of Israel — is acquiring Venafi, a specialist in machine identity, for $1.54 billion. 

CyberArk snaps up Venafi for $1.54B to ramp up in machine-to-machine security

Founder-market fit is one of the most crucial factors in a startup’s success, and operators (someone involved in the day-to-day operations of a startup) turned founders have an almost unfair advantage…

OpenseedVC, which backs operators in Africa and Europe starting their companies, reaches first close of $10M fund

A Singapore High Court has effectively approved Pine Labs’ request to shift its operations to India.

Pine Labs gets Singapore court approval to shift base to India

The AI Safety Institute, a U.K. body that aims to assess and address risks in AI platforms, has said it will open a second location in San Francisco. 

UK opens office in San Francisco to tackle AI risk

Companies are always looking for an edge, and searching for ways to encourage their employees to innovate. One way to do that is by running an internal hackathon around a…

Why companies are turning to internal hackathons

Featured Article

I’m rooting for Melinda French Gates to fix tech’s broken ‘brilliant jerk’ culture

Women in tech still face a shocking level of mistreatment at work. Melinda French Gates is one of the few working to change that.

2 days ago
I’m rooting for Melinda French Gates to fix tech’s  broken ‘brilliant jerk’ culture

Blue Origin has successfully completed its NS-25 mission, resuming crewed flights for the first time in nearly two years. The mission brought six tourist crew members to the edge of…

Blue Origin successfully launches its first crewed mission since 2022

Creative Artists Agency (CAA), one of the top entertainment and sports talent agencies, is hoping to be at the forefront of AI protection services for celebrities in Hollywood. With many…

Hollywood agency CAA aims to help stars manage their own AI likenesses

Expedia says Rathi Murthy and Sreenivas Rachamadugu, respectively its CTO and senior vice president of core services product & engineering, are no longer employed at the travel booking company. In…

Expedia says two execs dismissed after ‘violation of company policy’