The responses to the Law Commission’s third consultation on Automated Vehicles have now been published. You can find ours and the others here. Our comments related to questions of how safe is safe enough?, access and equality, driving tests for AVs, metrics for assessing performance, blame, the rules of the road and data-sharing. The pdf of our full response is here.
I was a guest at the Tortoise Futures of Cars Summit. Tortoise is a relatively new journalistic experiment ,working with the idea of ‘slow news’. Along with Matthew Avery from Thatcham, I helped with their discussion on self-driving cars.
The full discussion is here:
Helen Czerski asked me to join her on the Fully Charged podcast to talk about self-driving cars and hype. The discussion was fun, but some of the response has been plain fascinating. Here’s the original video
I said some things that I regarded as perfectly reasonable about Tesla and how its claims don’t match the reality of its system. Some Tesla fans took issue with this. (I stopped reading the comments after a few).
The brilliant Tim Harford (currently presenting a podcast on vaccines for the BBC) has published a piece, originally in the FT in October 2020, on hype, technology and ‘bionic duckweed’. He makes use of an example from my book – the claim that Level 5 autonomous vehicles are just around the corner:
A guest post from Yuting Shi, former MSc student in Science, Technology and Society at UCL
I recently took one of Baidu’s self-driving taxis in Beijing. Despite its name, there is still a driver on the vehicle, taking over when necessary. The passenger-carrying road test for “Apollo Go” has been happening free of charge in three suburbs of Beijing since October 2020. I was in “Beijing E Town”, about 20 km from the city centre. There are two types of vehicle collaborating with Baidu to be the body of the driverless car: Hongqi E-HS3 (a Chinese-branded electric vehicle) and Lincoln MKZ, among which, Hongqi is still in the pre-test stage and only Lincoln is available to the public. Baidu, a Chinese internet giant, has provided the intelligent system and sensors for “Apollo Go”. On the top of each vehicle is a shelf carrying sensors to detect further obstacles. Since passengers were not allowed to take photos, all photos in this blog are from CNR News, China’s largest news media.
Compared with the fear I felt when driving a vehicle in autopilot mode a few days ago, I felt relaxed, even a little bored in the “Apollo Go”. However, the 7.4km driverless journey also exposed many limitations of the technology.
An enjoyable user-interface
Hailing an Apollo Go was simple. On the home page of Baidu Map (similar to Google Maps), there is a “Taxi” option. Clicking on it, passengers can choose “automatic driving”, and then choose the nearby pickup location and destinations. It’s very similar to hailing a taxi, the only difference being that, for now, passengers have to get on and get off the driverless taxi at designated locations, making it more like a shuttle than a taxi.
However, after I took 3 minutes to hail an Apollo Go, I spent 30 minutes waiting for it. The driver explained that most vehicles were around a new testing site, a 10-minute drive from my pickup location. Given the limited number of automatic vehicles, I can forgive waiting longer than usual, but I have advised Baidu to add a “tracing taxi” function on the app in the feedback call, so that passengers can see nearby taxis, and choose a better pickup spot.
Each vehicle has a driver in the driver’s seat to take over when necessary. Passengers sit in the back with a touchscreen. I scanned a QR code for identification and pressed a “start the journey” button at the bottom of the screen. During the journey, the screen showed the moving elements around the vehicle: pedestrians, bicycles, cars, and trucks. Each type was marked with a different shape and color. I was gratified by the fact that it was hard to detect a delay on the real-time update screen.
Cautious, low-speed driving
Some drivers switch the car into manual mode during their test-driving However, in my experience, the car drove itself from the beginning to the end, which was beyond my expectation. The driver rested his hands on the steering wheel and his foot on the brake, paying attention to the road. My driver was confident about the vehicle and told me that, after rebuilding and long-term training, Baidu’s system could nearly match any other self-driving car, and his Lincoln had never been in an accident.
The vehicle moved like a very obedient child, running gingerly along the road. First, Apollo Go’s performance is smoother than manual driving. The vehicle gave the car in front more distance than a human would have, and it took more time to accelerate and decelerate. When our vehicle met a human-driven vehicle merging into the main road from a side road, it braked suddenly, which could be excused. Our driver honked the horn, which was the only occasion he intervened in driving. After the rapid brake, a voice prompt apologised to us.
The vehicle did not appear to overtake other vehicles. I met two road sweepers travelling at low speed just in front of me, but my vehicle did not change the lane even when the way was clear. The driver explained that the car would only leave the lane if it needed to turn left or right. When the vehicle encountered two road repairs in one lane 30 metres ahead, it changed the lane to avoid the first barrier, but then it went back to its initial lane. Later, it changed the lane again to avoid the second one. We can imagine such movements would easily be a problem on a busy road.
New mobility, new rules
To create a new transport mode, some existing habits and rules might need to change. Traffic rules are different among countries and even cities. In Beijing, vehicles that tend to turn right are not limited by traffic lights, which means they can turn right at any time. Therefore, vehicles have to avoid passer-by when passing a crossroad. Two limits make it difficult for an Apollo Go to navigate such a crossroad. First, since the vehicle was slower and more cautious than conventional ones, I asked the driver how the vehicle would perform in a busy area. “It’s OK to pass a crowded crossroad, but it’s very slow. If people get too close to the car, it will stop.” The second limit is that testing vehicles had never been operated in extremely busy areas and time. Compared with central Beijing, testing areas have a smaller population. The operation time of Apollo Go is 10am-4pm, avoiding rush hours. Although road tests are carried out in the real-world, it is almost the friendliest possible environment for a self-driving car. To accommodate the new mobility mode, the government may need to make new rules for vehicles turning right.
Baidu gave me a great self-driving experience on the public road, including both the user-interface and the road performance. The public test experience made me understand and trust self-driving cars and I know I am not the only one who feels the same way. Since Uber’s self-driving car killed Elaine Herzberg, a 49-year-old female who was crossing a high-way in Tempe, Arizona in 2018, we have been more cautious about safety issues on self-driving cars. However, when I talked to other passengers who tried Baidu’s self-driving vehicles, I find that they pay more attention to employment, ethical problems or legal issues than safety concerns, as the journey does not thrill them. For the public, trying a road test might be a perfect opportunity to engage with the cutting-edge technology. Baidu should keep its vehicles cautious. Involving the public in the road test could help the company reduce the information inequality between expertise and the public to earn more trust.
A conference that was due to be in Los Angeles, but took place on Zoom, natch. This was part of a series of meetings, funded by the US National Science Foundation, on Mathematical Challenges and Opportunities for Autonomous Vehicles. My talk was on the ways in which so-called ‘autonomous’ vehicles are inextricably attached to various social and technical infrastructures.
As part of a global citizens’ debate on driverless mobility coordinated by Missions Publiques, we worked with Involve, the public participation think tank, and Transport for Greater Manchester to run a day of public dialogue. The report has just been published by Involve.
Issues in Science and Technology invited me to respond to a recent piece on AV safety assurance by Marjory Blumenthal and Laura Fraade-Blanar. I’ve pasted my letter here. It’s also worth reading the response from the excellent Professor Missy Cummings.
Theres an old Irish joke in which a man stops to ask for directions and is told, ‘Well, I wouldn’t start from here’. When looking for guidance on governing new technologies, our starting points are often the problem. It is hard to escape our existing technological systems and our existing frames of reference. This is the case with automated vehicles. AVs promise transformative benefits, in particular for road safety. But they also bring a critical safety question of their own: how safe does an AV have to be to be safe enough?
In Can Automated Vehicles Prove Themselves to Be Safe? (Issues, Summer 2020), Marjory S. Blumenthal and Laura Fraade-Blanar are right to argue for a concerted approach to AV safety, and they are right that public trust will be crucial. But trust cannot be bought; it has to be earned. The process of standard-setting must be an inclusive one. Safety is too important to be left to technology developers alone. The test will not be whether AV developers can prove to themselves that their technology is safer than conventional driving. The question How safe is safe enough? is one for society at large, and it is profoundly uncertain, in part because it is connected to an even more complicated question: Safe enough for what?
The comparison with conventional cars is a poor starting point. The risks of cars are, as the novelist J. G. Ballard once put it, a ‘pandemic cataclysm’. More than a million deaths a year is a huge price, even if the benefits of the technology are clear. Developers of AVs may be aiming to clear the extremely low bar of overall road safety, but citizens will have other ideas. Aggregate improvements in safety, even if they are substantial, will do little to reassure parents when a child is killed by an AV and it is not clear who is responsible.
Everything we know about public risk perception tells us that people will evaluate the risks of new and potentially inequitable systems very differently from the risks of driving. In a car, we kid ourselves that we are in control of our destiny, which is one reason why we are willing to accept, according to some measures, more than 100 times greater risk than on an airplane or a train. It is not at all obvious that, as Blumenthal and Fraade-Blanar suppose, people’s perceptions of AV risk will be ‘grounded… in peoples experience with the safety of traditional automobiles’. What if people think of AVs like trains and conclude that each death marks a new catastrophe? The authors’ analogy may work in one important sense: with AVs, we may well see extreme distrust of attempts by powerful industries to govern themselves in the public interest.
Being trapped in an automotive frame of reference has other implications for how we think about AVs. Much of the excitement surrounding AVs comes from artificial intelligence. The story is that AI will be able to mimic and then surpass human capability without getting drunk, distracted, or sleepy. However, where we have seen automated systems deliver real benefits for transport, it is not thanks to super-intelligence, but rather because of a well-understood, well-defined system. Blumenthal and Fraade-Blanar talk about the gradual expansion of an AV’s operational design domain (ODD) as the technology becomes more sophisticated. My concern is that in many cases an ODD will be constrained to suit the AV, rather than the AV improved to match an ODD. In the name of safety, AVs may be given their own lanes, cyclists may be shepherded away from roads, and pedestrians may be asked to behave more predictably.
If we do not have an open, democratic debate about AV safety, we risk getting pushed around by the hype of the industry.
Science and Technology Studies
University College London
I was asked to do the keynote talk at the Shift Mobility 2020 conference in Berlin on 3rd September. Via Zoom, natch. I spoke about the seductive myth of autonomous vehicles and how it could lead to bad policy decisions. Here’s a loose version of what I said.
A just-so story
In 2007, at a disused Air Force base in California, teams of researchers from around the US came together to compete in the third DARPA Grand Challenge. Two earlier competitions had taken place in the desert, with the challenge being to get a robot car to find its way along a fixed course. This time, the challenge was to navigate an ersatz town, with junctions, other competitors’ cars and actual human-driven cars to contend with. There were a few bumps and scrapes, but six teams completed the three ‘missions’, travelling 55 miles without drivers. According to one enthusiastic account, which I reviewed here, this was “the moment… when everything changed”. The world began paying attention. Self-driving cars switched from impossible to inevitable. The competition’s team members went on to populate the tech companies who would funnel billions of dollars towards a race to develop the tech.
The promise that a truly self-driving car was just around the corner was based on an observation that artificial intelligence was advancing exponentially. Following a high-profile 1997 victory of a computer over the world’s greatest chess player in, headlines were now announcing that humans were being superseded in more complicated games like Go, as well as tasks like translation, voice recognition, translation and medical diagnostics. Huge increases in computer power and available data allowed machine learning to take off. At CES 2018, the CEO of Nvidia, a chipmaker, announced that AI would soon solve self-driving.
For tech people, this was a fascinating test case for machine learning in the wild. And the social justifications seemed clear. Humans are unreliable drivers. They get drunk, distracted and old. Computers mean reliability, safety and efficiency. Consultants crunched the numbers and concluded that self-driving cars would enable an 80% reduction in the number of cars, a repurposing of the space currently devoted to parking, hundreds of thousands of lived saved and trillions of dollars of economic benefits.
YouTube is replete with videos of self-driving cars in action. This one (below) is from Tesla. It shows something remarkable: an artificial intelligence sensing and classifying objects in real time, predicting their future movements and planning a safe path through them, in bad weather, on complicated streets, with pedestrians, cyclists and other hazards. Tesla’s sensors aren’t particularly complicated. Their argument is that if human eyes are good enough then video cameras will do the job just fine. The system’s power comes from its ability to learn from data gained from sensors across its whole fleet. When one Tesla learns something about the world, they all learn it.
This story is of a plug-and-play technology, learning about and adapting to the world in all its complexity. It will change the world without needing to change the world. The story is exciting and seductive. It is also, crucially, not true.
The real history of the technology is longer and more complicated than the simple story suggests. It is a history not just of smart cars, but of smart roads. In 1956, General Motors imagined a self-driving future in its Motorama exhibit. Notwithstanding the rather fixed social roles in its utopia – men up front smoking cigars; ladies in the back – this automotive dream comes from an age before the US had given up on infrastructure. The system on offer involves “high speed safety lanes” that would allow drivers to go hands-free and serve ice cream from their glove compartment.
General Motors, an old-school carmaker, still knows that for its technology to work, the world needs to meet it halfway.
An ‘autonomous vehicles’ is far from autonomous. For the technology to work, it needs to be embedded in the social and technical world – its physical and digital infrastructures, its legal rules and its social norms and practices. These things differ from place to place, making a universal technology impossible. Nor is the technology inevitable. Its development is driven by powerful commercial interests, which may not align with the public good. We can imagine that, if the technology’s claims go unquestioned, there will be pressure to change the world to suit self-driving cars.
Take the Roomba, a robot vacuum cleaner that has quickly become a part of many homes in the affluent world. Social scientists studying how people use the Roomba have found that it is a not a simple matter of buying a robot to solve their dirt problem. It takes work to make rooms navigable and machine-readable for the Roomba. Users had to adapt their lives so that the Roomba could do its job. Some of the details are interesting:
‘One of our participants told us that she threw away her rug in the living room because her Roomba kept “getting frustrated” with the length of the shag, getting it caught in its brushes. Another participant taped down the entire tassel on the carpet every time he ran the robot. Also, we had a participant who replaced the old refrigerator with a new one that had enough space underneath for Roomba.’From Sung, Ja-Young, et al. ““My Roomba is Rambo”: intimate home appliances.” International conference on ubiquitous computing, 2007.
What does this mean for self-driving cars? The question is not when the technology will arrive, but where, for whom and in what form? In places where self-driving cars are being tested, streets are being mapped in exquisite detail and kitted out with smart traffic lights. Places are being chosen for their weather, the tidiness of their road junctions, the predictability of their pedestrians and the affluence of their potential consumers. The transition towards a self-driving future will be patchy and uneven.
Upgrading our mobility
When motor cars began arriving in US cities more than a century ago, streets were messy places in which multiple modes of transport interacted. The historian Peter Norton describes how, in the name of efficiency and safety, streets were reorganised to suit the car. The motor lobby fought hard and pedestrians lost out.
Many cities are now trying to extricate themselves from a dependence on cars that has been built into their architectures, their economies and their cultures. If we are to realise the advantages of self-driving cars without repeating the mistakes made with their predecessors, we must not sleepwalk into the technology. It is not clear what the right approach is: in the US the autonomous ideal has taken hold while in China there is a more infrastructure-first approach. But rather than starting with an imaginary technology, we should start with people’s mobility needs.