Self-Driving Cars And The Kobayashi Maru


Image Credits: Rihardzz (opens in a new window) / Shutterstock (opens in a new window)

Andrew Heikkila


Andrew Heikkila is a tech enthusiast and writer from Boise, Idaho.

More posts from Andrew Heikkila

In 1966, Gene Roddenberry’s Star Trek would boldly go where no man had gone before, telling the tale of Captain Kirk and his crew as they explored the galaxy while taking on myriad sci-fi adventures.

In the opening scene of the franchise’s 1982 motion picture, Star Trek II: The Wrath of Khan, the U.S.S. Enterprise responds to a distress call from another ship, the Kobayashi Maru. Stranded in an area of space that the Enterprise can’t enter without risking interstellar war, the limping ship has almost 400 souls on board and is quickly losing life support. These people are going to die without help; the captain has an impossible choice to make.

The scene is later shown to be an unwinnable simulation, created as part of a training scenario. Deciding to not aid the Kobayashi Maru results in the death of its crew and passengers. However, acting to help the stranded ship will trigger conflict and result in the death and destruction of the Enterprise. The theme of a no-win scenario is prevalent throughout the rest of the film, and many Star Trek fans have colloquially come to call “damned if you do, damned if you don’t” situations by the name of the ship: Kobayashi Maru.

The idea of the no-win situation has gotten more attention over the last couple of years, as Google has been making strides with the driverless vehicle and Apple is rumored to be getting into the same market. But how does the Kobayashi Maru relate to self-driving automobiles?

Imagine you are driving down the road and you suddenly find yourself boxed in. In front of you is a large semi-truck with heavy crates on the back, to your right is a person on a motorcycle and to your left is a big SUV. All of a sudden, one of the crates falls off the back of the semi, directly in your path. What do you do?

If you swerve to the right, you’ll live, but the move would probably end up costing the person on the motorcycle their life. If you swerve left, you’ll collide with the SUV and possibly kill both yourself and its inhabitants — but there’s still a chance you’ll all survive the incident (albeit sustaining injury) because of the SUV’s high safety ratings. If you don’t swerve either way, you won’t injure anybody, but you’re definitely going to wreck and possibly die. So what should a driver do in this situation? What is the right answer?

This scenario comes from TEDEd, and is meant simply to illustrate that there is no right answer, especially in a scenario where there is little time to think. Each choice has a negative consequence, and the driver simply has to determine which option is, in their mind, the lesser of the evils.

Unfortunately, a person’s reactions in situations like these are more instinctual than they are based on decision or logic, simply because humans can’t process information that fast. Computers, on the other hand, can.

The driverless car as an invention has the potential to prevent approximately 1.3 million deaths annually, as well as between 20 and 50 million injuries, according to ASIRT. They are able to network with other smart cars and stop lights so that 151 million Americans can get to work faster and more safely. Because machines don’t blink. They don’t sleep or get drowsy. Machines don’t get drunk and drive.

In the only accident to date involving a self-driving car, it was determined humans were at fault, not machines — and yet, therein lies the problem. Accidents will happen, and a computer must be programmed to react in those situations, sometimes when death is inevitable. In those instances, it’s succinct to say that we’ll have to program computers to kill.

Let’s take a look at another scenario. There is a thought experiment called the trolley problem that asks you to imagine a runaway trolley headed for a group of five people tied up in its path. You’re standing near a lever, however, that will send the trolley to a different set of tracks if you flip it — the only problem is that there is a person tied up on those tracks, as well. You have two options: Do nothing, letting the trolley kill all five people on the main track, or flip the switch and send the trolley to the side track where it will kill one person.

In the most recent iteration of this problem, facilitated by researchers at Michigan State University, 147 subjects were given 3D headsets so they could actually experience this dilemma in an environment as close to reality as possible. Ninety percent of the participants flipped the switch, saving five people to kill one. This isn’t that surprising, as most people would say that five lives saved over one is ethically the right choice — but what happens when we switch the problem up a little bit?

Let’s say there is no side track the trolley will divert to if you flip the switch; instead, you’re standing next to a person large enough to stop the vehicle. The only caveat is that you must push him onto the track. The second variation of the problem produces different results, because there is a perceived difference between killing somebody and letting them die. The trend you come across is that not as many people would choose to kill the large man, even if it meant saving more lives overall, because they don’t want to be held personally responsible for his death.

Here’s a third scenario: What if you were the large person that could stop the trolley via self-sacrifice? Even better, what if your self-driving car turns a corner only to see a crowd of five people standing in the road? Your car either can hit them, sparing your own life, or the onboard AI can run your car off of the road, killing you and saving five lives.

If you answered that flipping the switch in the first iteration of the trolley problem was the right choice, because one death is better than five, then logically you would agree that your self-sacrifice is necessary to save the lives of the five people in the road ahead of you, right? Interestingly, if you defy the framework of logic and would rather choose self-preservation in this scenario, you’re actually in the majority.

Jean-Francois Bonnefon and the Toulouse School of Economics in France concluded from their own studies that these types of logical fallacies run rampant. As such, they believe it will be interesting to watch public opinion inevitably play a role in deciding how the ethics of AI works. Says Bonnefon and company: “[Participants of our study] actually wished others to cruise in utilitarian autonomous vehicles, more than they wanted to buy utilitarian autonomous vehicles themselves.”

Essentially, the problem is that people actually want driverless cars to sacrifice the occupant in favor of saving a higher number of lives — but only if they don’t have to drive one themselves. Unfortunately, the biggest catch-22 is that people won’t buy autonomous vehicles if they’re designed to kill their passengers, meaning that the status quo allowing split-second human decisions will continue to define accidents and reactions around the world. If we never legalize self-driving cars, our own human driving will continue to contribute to more than a million deaths globally.

Employing a fourth scenario, ran a reader poll that showed similar results trend toward self-preservation: You’re driving through a tunnel and a child appears at the opening and trips, blocking your exit. You can’t stop, so you’re left with the choice of swerving into a wall to save the child, or running over the child to save yourself. Of 110 people polled, 64 percent said they would continue straight and kill the child.

When asked which entities should determine how an autonomous car responds to the tunnel problem, 44 percent of respondents thought it should be the passenger of the vehicle, while 33 percent thought it should be lawmakers. Twelve percent thought the manufacturers or designers should be burdened with that choice; 11 percent responded “other.”

Determining who will control these “ethical settings” that guide no-win responses is a huge problem that self-driving cars are going to have to face in terms of liability. Because if a car will have to be programmed to choose between two lives, that means whoever decides how the algorithm is going to function is also possibly condemning to death either bystanders or passengers.

This type of “predetermined” action, an algorithm that chooses to spare children over adults, for example, would almost vicariously put the programmer in the driver’s seat, lending truth to the Department of Public Safety’s comment that a self-driven car will always have a “determinable human operator.”

Insurance companies are going to have to wrestle with that one, because in any instance, somebody will be liable if an autonomous vehicle gets into a wreck. If you get to decide on your car’s ethics settings and decide to continue straight and kill the child in the tunnel situation, does that make you liable for that child’s death? If it’s left up to the auto company, will they be liable?

The repercussions of these decisions extend much further into the future than anybody is able to foresee. As artificial intelligence advances, it may very well use the programmable ethics settings found in self-driving cars as a platform to build upon. Isaac Asimov once suggested there should be three laws of robotics that govern AI:

  • A robot may not injure a human being or, through inaction, allow a human being to come to harm.
  • A robot must obey orders given it by human beings except where such orders would conflict with the first law.
  • A robot must protect its own existence as long as such protection does not conflict with the first or second laws.

Obviously, the first law doesn’t work in this context, and is in danger of being trampled by militaries the world over searching for autonomous soldiers and vehicles (such as assault drones). Weapons aside, there are obvious reasons explored above that an AI would inevitably have to break the First Law of Robotics when faced with a Kobayashi Maru, and however we decide they should respond may constitute some kind of basis for how AI develops and writes its own ethical programming in the future.

If we determined today that favoring “quantity of lives” is the sole rule to follow for self-driving cars, for example, a much more developed, Skynet-esque AI of the future might calculate that citizens of industrialized countries are making the world uninhabitable for a majority of people and their many generations of offspring. Ethically, that AI could justify eradicating a large swath of the population so that an even larger percent can live.

Of course, there are much more immediate concerns that we’ll have to deal with in response to the self-driving car. What is going to happen to everybody in the trucking industry? Or to cab and Uber drivers? What happens if somebody remotely hijacks your car via the Internet of Things and crashes it with you in it?

The autonomous vehicle is still in the very early stages of development, but the way we decide to build its AI will set precedents. Unfortunately, we’re a species that still fights wars over land and money, that murders over passion and justifies the actions of the wicked.

On the other hand, we do have the capacity to love and sacrifice self for causes greater than our own. Sometimes it seems like we’re these creatures trying to program ethics into machines, when, in reality, we barely seem to understand or practice ethical behavior ourselves.

Nevertheless, we have an opportunity here to discuss these ethics and decide what type of character we want to define humankind, collectively, when faced with a Kobayashi Maru. Only once we’ve done our soul-searching and overcome that obstacle will we be able to follow in the footsteps of Captain Kirk, and bravely go where no man has gone before.

More TechCrunch

Ahead of the AI safety summit kicking off in Seoul, South Korea later this week, its co-host the United Kingdom is expanding its own efforts in the field. The AI…

UK opens office in San Francisco to tackle AI risk

Companies are always looking for an edge, and searching for ways to encourage their employees to innovate. One way to do that is by running an internal hackathon around a…

Why companies are turning to internal hackathons

Featured Article

I’m rooting for Melinda French Gates to fix tech’s broken ‘brilliant jerk’ culture

Women in tech still face a shocking level of mistreatment at work. Melinda French Gates is one of the few working to change that.

12 hours ago
I’m rooting for Melinda French Gates to fix tech’s  broken ‘brilliant jerk’ culture

Blue Origin has successfully completed its NS-25 mission, resuming crewed flights for the first time in nearly two years. The mission brought six tourist crew members to the edge of…

Blue Origin successfully launches its first crewed mission since 2022

Creative Artists Agency (CAA), one of the top entertainment and sports talent agencies, is hoping to be at the forefront of AI protection services for celebrities in Hollywood. With many…

Hollywood agency CAA aims to help stars manage their own AI likenesses

Expedia says Rathi Murthy and Sreenivas Rachamadugu, respectively its CTO and senior vice president of core services product & engineering, are no longer employed at the travel booking company. In…

Expedia says two execs dismissed after ‘violation of company policy’

Welcome back to TechCrunch’s Week in Review. This week had two major events from OpenAI and Google. OpenAI’s spring update event saw the reveal of its new model, GPT-4o, which…

OpenAI and Google lay out their competing AI visions

When Jeffrey Wang posted to X asking if anyone wanted to go in on an order of fancy-but-affordable office nap pods, he didn’t expect the post to go viral.

With AI startups booming, nap pods and Silicon Valley hustle culture are back

OpenAI’s Superalignment team, responsible for developing ways to govern and steer “superintelligent” AI systems, was promised 20% of the company’s compute resources, according to a person from that team. But…

OpenAI created a team to control ‘superintelligent’ AI — then let it wither, source says

A new crop of early-stage startups — along with some recent VC investments — illustrates a niche emerging in the autonomous vehicle technology sector. Unlike the companies bringing robotaxis to…

VCs and the military are fueling self-driving startups that don’t need roads

When the founders of Sagetap, Sahil Khanna and Kevin Hughes, started working at early-stage enterprise software startups, they were surprised to find that the companies they worked at were trying…

Deal Dive: Sagetap looks to bring enterprise software sales into the 21st century

Keeping up with an industry as fast-moving as AI is a tall order. So until an AI can do it for you, here’s a handy roundup of recent stories in the world…

This Week in AI: OpenAI moves away from safety

After Apple loosened its App Store guidelines to permit game emulators, the retro game emulator Delta — an app 10 years in the making — hit the top of the…

Adobe comes after indie game emulator Delta for copying its logo

Meta is once again taking on its competitors by developing a feature that borrows concepts from others — in this case, BeReal and Snapchat. The company is developing a feature…

Meta’s latest experiment borrows from BeReal’s and Snapchat’s core ideas

Welcome to Startups Weekly! We’ve been drowning in AI news this week, with Google’s I/O setting the pace. And Elon Musk rages against the machine.

Startups Weekly: It’s the dawning of the age of AI — plus,  Musk is raging against the machine

IndieBio’s Bay Area incubator is about to debut its 15th cohort of biotech startups. We took special note of a few, which were making some major, bordering on ludicrous, claims…

IndieBio’s SF incubator lineup is making some wild biotech promises

YouTube TV has announced that its multiview feature for watching four streams at once is now available on Android phones and tablets. The Android launch comes two months after YouTube…

YouTube TV’s ‘multiview’ feature is now available on Android phones and tablets

Featured Article

Two Santa Cruz students uncover security bug that could let millions do their laundry for free

CSC ServiceWorks provides laundry machines to thousands of residential homes and universities, but the company ignored requests to fix a security bug.

3 days ago
Two Santa Cruz students uncover security bug that could let millions do their laundry for free

TechCrunch Disrupt 2024 is just around the corner, and the buzz is palpable. But what if we told you there’s a chance for you to not just attend, but also…

Harness the TechCrunch Effect: Host a Side Event at Disrupt 2024

Decks are all about telling a compelling story and Goodcarbon does a good job on that front. But there’s important information missing too.

Pitch Deck Teardown: Goodcarbon’s $5.5M seed deck

Slack is making it difficult for its customers if they want the company to stop using its data for model training.

Slack under attack over sneaky AI training policy

A Texas-based company that provides health insurance and benefit plans disclosed a data breach affecting almost 2.5 million people, some of whom had their Social Security number stolen. WebTPA said…

Healthcare company WebTPA discloses breach affecting 2.5 million people

Featured Article

Microsoft dodges UK antitrust scrutiny over its Mistral AI stake

Microsoft won’t be facing antitrust scrutiny in the U.K. over its recent investment into French AI startup Mistral AI.

3 days ago
Microsoft dodges UK antitrust scrutiny over its Mistral AI stake

Ember has partnered with HSBC in the U.K. so that the bank’s business customers can access Ember’s services from their online accounts.

Embedded finance is still trendy as accounting automation startup Ember partners with HSBC UK

Kudos uses AI to figure out consumer spending habits so it can then provide more personalized financial advice, like maximizing rewards and utilizing credit effectively.

Kudos lands $10M for an AI smart wallet that picks the best credit card for purchases

The EU’s warning comes after Microsoft failed to respond to a legally binding request for information that focused on its generative AI tools.

EU warns Microsoft it could be fined billions over missing GenAI risk info

The prospects for troubled banking-as-a-service startup Synapse have gone from bad to worse this week after a United States Trustee filed an emergency motion on Wednesday.  The trustee is asking…

A US Trustee wants troubled fintech Synapse to be liquidated via Chapter 7 bankruptcy, cites ‘gross mismanagement’

U.K.-based Seraphim Space is spinning up its 13th accelerator program, with nine participating companies working on a range of tech from propulsion to in-space manufacturing and space situational awareness. The…

Seraphim’s latest space accelerator welcomes nine companies

OpenAI has reached a deal with Reddit to use the social news site’s data for training AI models. In a blog post on OpenAI’s press relations site, the company said…

OpenAI inks deal to train AI on Reddit data

X users will now be able to discover posts from new Communities that are trending directly from an Explore tab within the section.

X pushes more users to Communities