Featured Article

Covariant is building ChatGPT for robots

The UC Berkeley spinout says its new AI platform can help robots think more like people


Covariant's AI platform
Image Credits: Covariant

Covariant this week announced the launch of RFM-1 (Robotics Foundation Model 1). Peter Chen, the co-founder and CEO of the UC Berkeley artificial intelligence spinout tells TechCrunch the platform, “is basically a large language model (LLM), but for robot language.”

RFM-1 is the result of, among other things, a massive trove of data collected from the deployment of Covariant’s Brain AI platform. With customer consent, the startup has been building the robot equivalent of an LLM database.

“The vision of RFM-1 is to power the billions of robots to come,” Chen says. “We at Covariant have already deployed lots of robots at warehouses with success. But that is not the limit of where we want to get to. We really want to power robots in manufacturing, food processing, recycling, agriculture, the service industry and even into people’s homes.”

The platform launches as more robotics firms are discussing the future of “general purpose” systems. The sudden onslaught of humanoid robotics firms like Agility, Figure, 1X and Apptronik has played a pivotal role in that conversation. The form factor is particularly suited to adaptability (much like the humans on which it’s modeled), though the robustness of on-board AI/software systems is another question entirely.

For now, Covariant’s software is largely deployed on industrial robotic arms doing a variety of familiar warehouse tasks, including jobs like bin picking. It isn’t currently deployed on humanoids, though the company is promising some level of hardware agnosticism.

“We do like a lot of the work that is happening in the more general purpose robot hardware space,” says Chen. “Coupling the intelligence inflection point with the hardware inflection point is where we will see even more explosion of robot applications. But a lot of those are not fully there yet, especially on the hardware side. It’s very hard to go beyond the staged video. How many people have interacted with a humanoid in person? That tells you the degree of maturity.”

Image Credits: Covariant

Covariant doesn’t, however, shy away from human comparisons when it comes to the role RFM-1 plays in robots’ decision-making processes. Per its press material, the platform, “provides robots the human-like ability to reason, representing the first time Generative AI has successfully given commercial robots a deeper understanding of language and the physical world.”

This is one of those realms where we have to be careful with claims, both in terms of comparisons to abstract — or even philosophical — concepts and their actual real-world efficacy over time. “Human-like ability to reason” is a broad-sweeping concept that means a lot of different things to a lot of different people. Here the notion applies to the system’s ability to process real-world data and determine the best course of action to execute the task at hand.

This is a departure from traditional robotic systems that are a programmed to one job repeatedly, ad infinitum. Such single-purpose robots have thrived in highly structured environments, starting with automotive assembly lines. As long as there are minimal changes to the task at hand, a robot arm can do its work over and over again, unimpeded, until it’s time to call it a day and collect the golden pocket watch for its years of loyal service.

Things can break down quickly, however, with even the smallest deviations. Say the object isn’t placed exactly right on the conveyor belt, or there’s been an adjustment to lighting that impacts on-board cameras. These sorts of differences can have a huge impact on the robot’s ability to execute. Now imagine trying to get that robot to work with a new part, new material or even do a wholly different task. That’s even harder.

This is the point where programmers traditionally step in. The robot must be reprogrammed. More often than not, someone from outside the factory floor enters the picture. This is a big drain of resources and time. If you want to avoid this, one of two things needs to happen: 1) People working on the floor need to learn code or 2) You need a new, more natural method for interacting with the robot.

While it would be great to do the former, it seems unlikely that companies will be willing to invest the money and wait the necessary time. The latter is precisely what Covariant is attempting to do with RFM-1. “ChatGPT for robots” isn’t a perfect analogy, but it’s a reasonable shorthand (especially in light of the founders’ connection to OpenAI).

From the customer’s point of view, the platform presents as a text field, much like the current iteration of consumer-facing generative AI. Input a text command like, “pick up the apple” by typing or voice, and the system uses its training data (shape, color, size, etc.) to identify the object in front of it that most closely matches that description.

RFM-1 then generates video outcomes — in essence simulations — to determine the best course of action using past training. This last bit is similar to how our brains work out the potential outcomes of an action prior to executing.

During a live demo, the system reacts to inputs like “pick up the red object” and even the more semantically complex, “pick up what you put on your feet before you put on your shoes,” which caused the robot to correctly pick up the apple and a pair of socks, respectively.

A lot of big ideas are tossed around when discussing the system’s promise. At the very least, Covariant has an impressive pedigree among its founders. Chen studied AI at Berkeley under Pieter Abbeel, his Covariant co-founder and chief scientist. Abbeel also became an early OpenAI employee in 2016, a month after Chen joined the ChatGPT firm. Covariant was founded the following year.

Chen says the company expects the new RFM-1 platform will work with a “majority” of the hardware on which Covariant software is already deployed.

More TechCrunch

Zen Educate, an online marketplace that connects schools with teachers, has raised $37 million in a Series B round of funding. The raise comes amid a growing teacher shortage crisis…

Zen Educate raises $37M and acquires Aquinas Education as it tries to address the teacher shortage

“When I heard the released demo, I was shocked, angered and in disbelief that Mr. Altman would pursue a voice that sounded so eerily similar to mine.”

Scarlett Johansson says that OpenAI approached her to use her voice

A new self-driving truck — manufactured by Volvo and loaded with autonomous vehicle tech developed by Aurora Innovation — could be on public highways as early as this summer.  The…

Aurora and Volvo unveil self-driving truck designed for a driverless future

The European venture capital firm raised its fourth fund as fund as climate tech “comes of age.”

ETF Partners raises €284M for climate startups that will be effective quickly — not 20 years down the road

Copilot, Microsoft’s brand of generative AI, will soon be far more deeply integrated into the Windows 11 experience.

Microsoft wants to make Windows an AI operating system, launches Copilot+ PCs

Hello and welcome back to TechCrunch Space. For those who haven’t heard, the first crewed launch of Boeing’s Starliner capsule has been pushed back yet again to no earlier than…

TechCrunch Space: Star(side)liner

When I attended Automate in Chicago a few weeks back, multiple people thanked me for TechCrunch’s semi-regular robotics job report. It’s always edifying to get that feedback in person. While…

These 81 robotics companies are hiring

The top vehicle safety regulator in the U.S. has launched a formal probe into an April crash involving the all-electric VinFast VF8 SUV that claimed the lives of a family…

VinFast crash that killed family of four now under federal investigation

When putting a video portal in a public park in the middle of New York City, some inappropriate behavior will likely occur. The Portal, the vision of Lithuanian artist and…

NYC-Dublin real-time video portal reopens with some fixes to prevent inappropriate behavior

Longtime New York-based seed investor, Contour Venture Partners, is making progress on its latest flagship fund after lowering its target. The firm closed on $42 million, raised from 64 backers,…

Contour Venture Partners, an early investor in Datadog and Movable Ink, lowers the target for its fifth fund

Meta’s Oversight Board has now extended its scope to include the company’s newest platform, Instagram Threads, and has begun hearing cases from Threads.

Meta’s Oversight Board takes its first Threads case

The company says it’s refocusing and prioritizing fewer initiatives that will have the biggest impact on customers and add value to the business.

SeekOut, a recruiting startup last valued at $1.2 billion, lays off 30% of its workforce

The U.K.’s self-proclaimed “world-leading” regulations for self-driving cars are now official, after the Automated Vehicles (AV) Act received royal assent — the final rubber stamp any legislation must go through…

UK’s autonomous vehicle legislation becomes law, paving the way for first driverless cars by 2026

ChatGPT, OpenAI’s text-generating AI chatbot, has taken the world by storm. What started as a tool to hyper-charge productivity through writing essays and code with short text prompts has evolved…

ChatGPT: Everything you need to know about the AI-powered chatbot

SoLo Funds CEO Travis Holoway: “Regulators seem driven by press releases when they should be motivated by true consumer protection and empowering equitable solutions.”

Fintech lender SoLo Funds is being sued again by the government over its lending practices

Hard tech startups generate a lot of buzz, but there’s a growing cohort of companies building digital tools squarely focused on making hard tech development faster, more efficient and —…

Rollup wants to be the hardware engineer’s workhorse

TechCrunch Disrupt 2024 is not just about groundbreaking innovations, insightful panels, and visionary speakers — it’s also about listening to YOU, the audience, and what you feel is top of…

Disrupt Audience Choice vote closes Friday

Google says the new SDK would help Google expand on its core mission of connecting the right audience to the right content at the right time.

Google is launching a new Android feature to drive users back into their installed apps

Jolla has taken the official wraps off the first version of its personal server-based AI assistant in the making. The reborn startup is building a privacy-focused AI device — aka…

Jolla debuts privacy-focused AI hardware

The ChatGPT mobile app’s net revenue first jumped 22% on the day of the GPT-4o launch and continued to grow in the following days.

ChatGPT’s mobile app revenue saw its biggest spike yet following GPT-4o launch

Dating app maker Bumble has acquired Geneva, an online platform built around forming real-world groups and clubs. The company said that the deal is designed to help it expand its…

Bumble buys community building app Geneva to expand further into friendships

CyberArk — one of the army of larger security companies founded out of Israel — is acquiring Venafi, a specialist in machine identity, for $1.54 billion. 

CyberArk snaps up Venafi for $1.54B to ramp up in machine-to-machine security

Founder-market fit is one of the most crucial factors in a startup’s success, and operators (someone involved in the day-to-day operations of a startup) turned founders have an almost unfair advantage…

OpenseedVC, which backs operators in Africa and Europe starting their companies, reaches first close of $10M fund

A Singapore High Court has effectively approved Pine Labs’ request to shift its operations to India.

Pine Labs gets Singapore court approval to shift base to India

The AI Safety Institute, a U.K. body that aims to assess and address risks in AI platforms, has said it will open a second location in San Francisco. 

UK opens office in San Francisco to tackle AI risk

Companies are always looking for an edge, and searching for ways to encourage their employees to innovate. One way to do that is by running an internal hackathon around a…

Why companies are turning to internal hackathons

Featured Article

I’m rooting for Melinda French Gates to fix tech’s broken ‘brilliant jerk’ culture

Women in tech still face a shocking level of mistreatment at work. Melinda French Gates is one of the few working to change that.

2 days ago
I’m rooting for Melinda French Gates to fix tech’s  broken ‘brilliant jerk’ culture

Blue Origin has successfully completed its NS-25 mission, resuming crewed flights for the first time in nearly two years. The mission brought six tourist crew members to the edge of…

Blue Origin successfully launches its first crewed mission since 2022

Creative Artists Agency (CAA), one of the top entertainment and sports talent agencies, is hoping to be at the forefront of AI protection services for celebrities in Hollywood. With many…

Hollywood agency CAA aims to help stars manage their own AI likenesses

Expedia says Rathi Murthy and Sreenivas Rachamadugu, respectively its CTO and senior vice president of core services product & engineering, are no longer employed at the travel booking company. In…

Expedia says two execs dismissed after ‘violation of company policy’