Embodied AI, superintelligence and the master algorithm

Superintelligence, roughly defined as an AI algorithm that can solve all problems better than people, will be a watershed for humanity and tech.

Even the best human experts have trouble making predictions about highly probabilistic, wicked problems. And yet those wicked problems surround us. We are all living through immense change in complex systems that impact the climate, public health, geopolitics and basic needs served by the supply chain.

Just determining the best way to distribute COVID-19 vaccines without the help of an algorithm is practically impossible. We need to get smarter in how we solve these problems — fast.

Superintelligence, if achieved, would help us make better predictions about challenges like natural disasters, building resilient supply chains or geopolitical conflict, and come up with better strategies to solve them. The last decade has shown how much AI can improve the accuracy of our predictions. That’s why there is an international race among corporations and governments around superintelligence.

Highly credible think tanks like Deepmind and OpenAI say that the path to superintelligence is visible. Last month, Deepmind said reinforcement learning (RL) could get us there, and RL is at the heart of embodied AI.

What is embodied AI?

Embodied AI is AI that controls a physical “thing,” like a robot arm or an autonomous vehicle. It is able to move through the world and affect a physical environment with its actions, similar to the way a person does. In contrast, most predictive models live in the cloud doing things such as classifying text or images, steering flows of bits without ever moving a body through three-dimensional space.

For those who work in software, including AI researchers, it is too easy to forget the body. But any superintelligent algorithm needs to control a body because so many of the problems we confront as humans are physical. Firestorms, coronaviruses and supply chain breakdowns need solutions that aren’t just digital.

All the crazy Boston Dynamics videos of robots jumping, dancing, balancing and running are examples of embodied AI. They show how far we’ve come from early breakthroughs in dynamic robot balancing made by Trevor Blackwell and Anybots more than a decade ago. The field is moving fast and, in this revolution, you can dance.

What’s blocked embodied AI up until now?

Challenge 1: One of the challenges when controlling machines with AI is the high dimensionality of the world — the sheer range of things that can come at you.

What does high dimensionality mean? One way to think about it is: It’s the number of signals you have to pay attention to in order to get what you want. In chess, you only have to pay attention to the pieces on the board. That’s relatively low dimensionality. The weather doesn’t matter. The pieces are fixed. The pawns will not sprout wings and fly.

But what if you are building agricultural robots to solve a farming problem? You have to pay attention to what’s happening in the field. But you also have to know how to respond to a thousand different weather types and keep in mind that some things, like locusts, do sprout wings before they come for you. If we cannot solve the basic problem of producing food for ourselves, we will not make a superintelligence. Embodied AI is the gate it must pass through.

Challenge 2: It’s hard to know what worked. Sometimes our environment only reveals the consequences of our choices many years later. Like people, AI agents don’t learn as well if they lack feedback, which is the case when you can’t see the results of what you do. Humans have developed culture, principles and platitudes to pass long-term lessons down to short-lived individuals. Robots don’t have culture outside their software. So how do they learn the long-term effects of decisions made in a fast-paced environment?

Challenge 3: It’s hard to learn how to handle situations that have never occurred before. To navigate the evolving present, we have to extract principles and theories from our experience about how the world works and apply those theories to new situations. A lot of machine learning is constrained to learn from the narrow corridor of history, i.e., the data that was collected. That is why machine learning algorithms that learn from the past are doomed to repeat it.

But we are much closer to making embodied AI work

There are three positive indicators that we are on the road to making embodied AI work: advances in deep reinforcement learning, more compute on the edge, and AI training based on simulations and historical data.

Advances in deep RL

Advances in deep RL, a family of AI algorithms that is often used in robotics, have moved us much closer to solving embodied AI because deep RL solves sequential decision-making problems where long moments separate actions from the outcomes they produce.

In the last few years, the advances in RL have been staggering thanks to think tanks such as DeepMind and OpenAI, robotics-focused startups like Covariant, and AI-driven companies like Tesla.

Kodiak Robotics’ founder says tight focus on autonomous trucks is working

When Peter Thiel and Elon Musk backed DeepMind it was still working on Atari games, which it beat with an RL algorithm called deep Q learning. That work would ultimately lead to Google’s acquisition of the company, as well as Musk’s subsequent initiatives to warn the world of the dangers of AI, and his creation of OpenAI with Sam Altman.

Researchers such as Pieter Abbeel, Peter Chen and Rocky Duan, after spending time at OpenAI, would go on to found Covariant (formerly known as “Embodied Intelligence”) and use deep RL to solve some of the hardest problems in robotics, which they are now bringing to manufacturing and industrial control.

DeepMind’s claims about superintelligence in May, and its recent achievements in protein folding with AlphaFold, are more signals that deep RL is the most promising sector of AI right now.

Faster compute

In addition to algorithmic breakthroughs, embodied AI is benefiting from faster, cheaper and more compact compute on the edge, as companies like Nvidia, AMD, Qualcomm and Intel are making it possible to process more information from sensors locally to keep latency low and ensure a fast response.

Simulation training

Finally, one underreported aspect of reinforcement learning is that it is mostly trained in simulations. That is, it escapes the narrow corridor of history and trains on what might happen in the future. All those video games? Simulations. Protein folding? A molecular simulation. Simulations allow these algorithms to live a thousand lives, experiencing things we have never seen. That’s the real secret: They don’t just contain more intelligence, they contain it because they have lived longer than us, using parallel compute.

What will take us from potential to reality in the next 18 months?

A few years ago, prominent researchers were still claiming that RL “didn’t work.” We have seen case after case where it does. Things are moving fast. What the 1990s was for the internet, the 2020s will be for AI and robotics.

We already have the algorithms and the compute for those systems. Now we need the data and the bandwidth. In the next 18 months, we see advances on three fronts:

1. Deep RL needs to get the right observations from the world, and IoT data is what it will use to do so. To get that data, machines have to be wired with sensors. In a sense, deep RL is the real AI of Things, or AIoT. Companies like PTC, Siemens, ABB and Rockwell Automation are helping the major players of the manufacturing industry wire up their physical plants and gather the data they need to monitor their operations. A lot of that is what they call Industrial IoT, or IIoT. Unicorns such as Samsara have developed a single pane of glass to track that data.
2. Deep RL needs higher bandwidth to move data and decisions between the sensors and the compute in the cloud. That is the purpose of 5G, and we will increasingly see private 5G networks implemented in manufacturing logistics facilities that rely more and more on robots to get the work done. Those 5G implementations are on the way.
3. As RL agents gain more experience both from simulations and embedded experience in machines themselves, such as autonomous vehicles and robot arms, they will perform better and better against benchmarks. That flywheel of performance will accelerate a broader replatforming of IoT workloads to the clouds.

In the next year and a half, we’re going to see increasing adoption of these technologies, which will trigger a broader industry shift, much as Tesla triggered the transition to EVs. By exposing AI to more real-world data and challenges, by getting it into more robot bodies, we can accelerate both the digital transformation of industry and the intelligence gains of AI itself.

How we built an AI unicorn in 6 years