What a self-driving car sees, what you may be using already, how self-driving cars work – in pictures

How a Car Drives Itself

LIDAR UNIT CAMERAS

Constantly spinning, it uses laser beams to generate a 360-degree image of the car’s surroundings.  Uses parallax from multiple images to find the distance to various objects. Cameras also detect traffic lights and signs, and help recognize moving objects like pedestrians and bicyclists.

RADAR SENSORS

Measure the distance from the car to obstacles.

MAIN COMPUTER (LOCATED IN TRUNK)

ADDITIONAL LIDAR UNITS

Analyzes data from the sensors, and compares its stored maps to assess current conditions.

What the Car Sees

The car’s sensors gather data on nearby objects, like their size and rate of speed. It categorizes the objects — as cyclists, pedestrians or other cars and objects — based on how they are likely to behave.

red boxes: cyclists

yellow boxes: pedestrians

pink boxes: vehicles

green “fences”: locations where the car may need to slow down

red “fences”: locations where the car will need to stop

Self-Driving Features You May Have Already Used

Collision avoidance

Radar-, laser-, or camera-based systems that warn of an impending collision. Some systems recognize a person straying into the road. If the driver ignores the warnings, some systems will still apply the brakes.

Drifting warning

When your car begins to deviate from its lane, some systems alert the driver with a warning buzzer, light and small counter-steering force to the steering wheel.

Blind-spot detectors

Uses cameras or radar to detect vehicles in the driver’s blind spot. Alerts the driver with sounds or warning lights in the rearview mirror or in the car’s pillars next to the windshield.

Enhanced cruise control

Maintains a predefined distance to the vehicle ahead. If it slows, your car also slows. If a car moves into your lane, your car keeps its distance. Useful in bumper-to-bumper traffic.

Self parking

The car maneuvers itself into a parking spot using cameras or sonar. But the driver usually has to brake and follow commands. It first appeared in 2003 in the Toyota Prius. It is now offered by BMW, Ford and many others.

Illustrations by Guilbert Gates/The New York Times

What Automakers Are Doing

Announced in October that its cars would come equipped with hardware to allow them to eventually be fully autonomous. A month earlier, it released a new software upgrade of its controversial Autopilot option after a Tesla crashed in the first fatal accident involving a self-driving car.

General Motors

Invested $500 million with Lyft in January and later announced it would test self-driving electric taxis on public roads within a year.Bought Cruise Automation, an autonomous vehicle start-up, for about $1 billion in March. Developing Super Cruise, with features similar to Tesla’s Autopilot.

Ford

Announced its intent to have a fully autonomous vehicle in commercial operation in 2021 in a hailing or sharing service. The company is investing in or collaborating with four technology start-ups and doubling the size of its Silicon Valley team.

Fiat Chrysler

Reached a deal with Alphabet (Google) in May, after first approaching Apple, to add its autonomous technology to Chrysler minivans. Alphabet now has an office in Michigan to work with Fiat Chrysler.

Honda

Displayed some self-driving cars in June. The technology in its “augmented reality” patent application would allow driver to see people behind trees and columns.

Volvo

Started a pilot project in Sweden to put autonomous XC90s in the hands of residents in Gothenburg. The feedback could lead to further public trials in London and China. Working with Uber on its self-driving program in Pittsburgh.

What Technology Companies and Others Are Doing

Google (Alphabet)

Is spinning off its autonomous car project into a new company named Waymo, signaling that its effort is now past its research phase and ready to be commercialized.

Apple

Its secret effort, code-named Titan, appears to be shrinking. It is in talks with McLaren and Lit Motors to reboot its project.

Autoliv

The world’s largest airbag supplier is in a joint venture with Volvo to develop self-driving cars.

Intel

Is in partnerships with Mobileye and BMW to develop autonomous vehicles by 2021 as well as technology that can be used by others.

Mobileye

Is a major producer of lidar as well as camera equipment and software used in autonomous vehicles.Tonomy

Based in Singapore and founded by two scientists from M.I.T., it began a self-driving taxi service days before Uber began its driverless service in Pittsburgh.

U.S. Dept. of Transportation

Released its Federal Automated Vehicles Policy, setting guidelines for automakers. Issued proposed rules for cars to talk to each other.

University of Michigan

Operates Mcity, a 32-acre autonomous-car test facility, affiliated with its Mobility Transformation Center.

Where You Can Ride in a Self-Driving Car

In September, Uber introduced 100 cars that can drive themselves (although they will all have a driver just in case), making Pittsburgh the first city in which a passenger can hail an autonomous vehicle. On Wednesday, Uber began allowing passengers to use them in San Francisco.

Jeff Swensen for The New York Times

How Tesla’s Technology Differs

Tesla uses a computer vision-based vehicle detection system, but according to the company, it is not intended to be used hands-free and parts of the system are unfinished. Tesla does not use lidar technology.

A Tesla Model S in a showroom in Brooklyn, N.Y., in May. Michael Nagle for The New York Times

Forward-facing camera

Image-processing software can detect lane stripes, signs, stop lights, road signs and other objects.

Forward radar

Reflected microwaves are used to identify location and speed, but not always type, of nearby vehicles.

Ultrasonic sensors

Reflected sound waves are used to detect distance to nearby objects.

GPS

Combined with high-precision mapping, GPS determines a car’s position on the road.

Potential Danger: The First Fatal Self-Driving Accident

After Joshua Brown, 40, was killed in May while at the wheel of a Tesla Model S, questions arose about the safety of the car’s crash-avoidance Autopilot system. The company has since released a new version.The accident may have happened in part because the crash-avoidance system was designed to engage only when radar and computer vision systems agreed that there was an obstacle, according to an industry executive with direct knowledge of the system. Some experts speculated that a lidar-driven car might have avoided the crash. But the company’s founder and chief executive, Elon Musk, has said the upgrade could have prevented the crash.