Confused of Bloomsbury

I recently attended a talk given by Edmond Awad, one of the team that carried out the research project entitled Moral Machine, amongst other things.  Run from MIT, Moral Machine is an online survey that presents respondents with a series of randomly generated dilemmas based on the so-called trolley problem.  It sets up a series of scenarios in which an automated vehicle is on a collision course and the respondent is asked to choose between two courses of action that will have differing effects on both occupants of the vehicle and those crossing its path.  In addition to being told the number of individuals at risk of fatal injury, respondents are also presented with information about their characteristics, ranging from age and gender to species (human, cat, dog), status, level of fitness and, even, housing status.

The survey has attracted a very large number of responses, with a paper in Nature reporting on the first 40 million “dilemma responses”.

This member of the audience found the whole thing very intriguing and came away thinking that trolley problems remain the question after the question.  That is, before we ask whether an automated system should prioritise on the basis of age or gender, say, we should be asking whether automated systems should be allowed to prioritise at all.  But the trolley problem has demonstrated an enduring appeal to professional and armchair philosophers alike for many years and we should not deny them their fun.

I got confused, though, after I asked Dr Awad whether any respondents had pointed out that no automated system would ever be able reliably to identify whether Person A was homeless (this being one of the attributes tested by Moral Machine).  He replied that this wasn’t the issue: what his experiment was testing was people’s values and, in effect, concerned itself with the aftermath of any crash (whether the “right” individuals had died) than the algorithm the automated system might employ.  Confused of Bloomsbury responded that the homeless/not homeless attribute in the survey therefore wasn’t included to test people’s attitudes to what automated vehicles might do.  And Dr Awad replied that none of this was: no one responding to the survey actually expected automated vehicles to do as depicted in the scenarios.  (I hope I represent his words fairly.)

If my understanding of what he said is correct, I have two comments.  The first is that there is a world of difference between something not being possible now and not being possible ever.  I suspect that most, if not all, of those who responded to the survey did so on the basis that an automated vehicle might one day be designed such that at least some of their preferences could be enacted.  So I’m very dubious about seeing the entire experiment as speculative.  If it was, why use automated vehicles at all?  Why not use the conventional out-of-control trolley, if all that is sought is an understanding of people’s preferences concerning who should die in such scenarios and of their attitudes to acts and omissions?  But I can’t believe that the AV element is nothing more than a MacGuffin: members of the research team have done other work on AVs and are clearly interested in ethical questions relating to this technology.

So, second, if I’m right that the researchers are asking about AVs because they are interested in AVs, the fact that the scenarios used in Moral Machine are manifestly beyond what artificial intelligence is likely ever to be able to do makes their inclusion at least frivolous and, possibly, irresponsible.  Automated vehicles are the subject of intense discussion at present precisely because a variety of actors are hard at work bringing them onto our transport networks, with the high probability that humans (homeless or otherwise), dogs and cats will be involved in collisions.  I think it’s therefore incumbent on any researcher purportedly doing serious research on the subject to stay within the bounds of what is reasonable.  To give the team the benefit of the doubt, I suppose the research could meaningfully illuminate the debate if we were envisaging a future world in which humans are tagged with their attributes, including their housing situation, thus relieving the vehicle of having to discern such details.  But that doesn’t even bear thinking about.

Has the Driverless Future arrived?

Ten days ago, journalist Ed Niedermeyer became one of the first people to have a trip in a genuinely driverless car on public roads without a nervous company demanding a non-disclosure agreement. Niedermeyer is no self-driving cheerleader. His work is well-researched, balanced and detached from the hype. My guess is that Waymo asked him as a display of their self-confidence.

There are currently hundreds of Waymo cars moving around the suburbs of Phoenix, Arizona, with thousands more on the way. The driverless future, it seems, is already here. Except that it isn’t. This technology is not an iPhone. It cannot just be bought and used. While for a few people in a limited area it may already be a reality, for most people in most places it will remain an impossibility for the foreseeable future. The people developing the tech are keen on announcing breakthroughs, which give some narrative zing to a story that has over the last year become a bit flat.

Oliver Cameron, the CEO of Voyage, is developing self-driving technology in a unique context. His vehicles are shuttling retirees on private roads around their purpose-built community – The Villages – in Florida.

I have admired Cameron since I heard him emphasise the value of a slow, responsible approach to technology development here. He has just published a discussion of what he sees as Waymo’s Rubicon-crossing ‘insane step forward’.

He thinks ‘we now live in a driverless world’. I’m worried he might believe it.

William Gibson had an adage:

‘The future has arrived — it’s just not evenly distributed yet.’

Technological determinists like to invoke it when they claim new technologies are inevitable and just around the corner. They rarely acknowledge that the distribution of new technologies is perpetually uneven. Technologies tend in fact to follow what Robert Merton called the Matthew Effect: For whosoever hath, to him shall be given, and he shall have more abundance.

Cameron knows that getting a self-driving car to work requires means getting the conditions right as well as smartening up the sensors and the software. Self-driving cars are unavoidably contextual. When talking about the progress of self-driving technology, Cameron is right to say ‘It’s a matter of where, not when’. There are circumstances in which the driverless future has been with us for a while. In London, the Docklands Light Railway was christened in 1987. It is relatively uninteresting because the conditions under which its automation works are so tightly constrained.

The future has arrived. (Credit: Flickr user SV1XV)

Waymo’s world is vastly more complex than the DLR’s, but it remains geofenced. The cars work because they know the roads and they know what to expect. Take a Waymo outside Phoenix and it becomes just another minivan. To get a genuine sense of the future, we need to ask why Waymo is in Phoenix, work out the conditions for the technology’s success (economic, political, meteorological, infrastructural and cultural) and ask where next? There will be places with wildly different conditions in which driverless technology will necessarily look very different, and countless more where the incentives for innovation will never line up.  

The definition and distribution of futures will not be straightforward. Cameron expects public scrutiny and does not shy away from it:  

‘There will likely be companies who abuse their responsibility to deploy this technology responsibly, making short-term decisions that may compromise safety.’

Following the recent revelations about the Uber crash (see my tweet thread is below), one can see the reputational risk.

However, if Voyage, Waymo and other developers want to innovate responsibly, they shouldn’t swallow their own exhaust. One can understand why innovators tell themselves

‘Fully self-driving technology is right… Fully self-driving technology is necessary.’

But they shouldn’t expect everyone else to agree. Public resistance may not be ‘a natural cycle that will one day pass’, as Cameron hopes. The futures that most of us experience will look very different from the ones imagined by technologists. They always are.

Calculating route

A public consultation on self-driving cars has raised important and challenging questions, says Jack Stilgoe (reprinted from Research Fortnight’s Political Science blog)

The hype surrounding self-driving cars is huge, and not entirely unjustified. There are big potential benefits for safety, efficiency, mobility and access, but they won’t come without careful policy design. If we don’t get the governance right, we could end up with more congestion, less liveable cities and new dangers.

Much of the excitement has so far revolved around artificial intelligence. Self-driving cars offer a real-world test case for the powers of machine learning. But the issues are much more than technical ones.

If the technology is going to learn to drive, it needs to be tested on actual roads. Self-driving cars therefore involve the public display of innovation, with developers using roads as their laboratories.

The UK government recognises the scale of opportunities and potential risks, which is why it commissioned the world’s first large-scale, in-depth public dialogue to investigate the hopes, concerns and questions about connected and autonomous vehicles.

In conversation

The exercise was initiated by Sciencewise, the government’s public dialogue agency, and I was lucky to be part of the team that designed and ran it. The full report was released on 10 October.

George Freeman, member of parliament for Mid Norfolk and holder of the newly created ministerial post for transport technology and innovation, has discussed the findings in the New Statesman. Unsurprisingly, he accentuates the positive.

He’s right that there was a lot of excitement about the possibilities, but that’s not the whole story. Policymakers still have a lot of work to do if the technology is going to be broadly acceptable.

The dialogue exercise took place in five UK locations, over three days between October and December 2018. At each location, around 30 members of the public, chosen to be as diverse as possible, heard about self-driving cars, spoke to experts and, in some places, had a hands-on—or rather, hands-off—experience in a prototype or a simulator. Their conclusions, priorities and questions offer a blueprint for governance.

Until now, when ordinary people have been asked what they think of self-driving cars, they have in most cases been treated as potential consumers. Instead, we treated them as citizens.

Who’s driving?

Participants quickly grasped that the issues were not just to do with how safe the technology would be, or how it should deal with so-called ‘trolley problems’ involving choices between doing different types and degree of harm.

They wondered how, if the technology was going to be as transformative as the car was in the 20th century, our worlds and lives might change. They saw enormous potential benefits in providing mobility to old or disabled people—but they also worried if these opportunities would be available to all, or if the technology would follow the money and only be an option for rich people.

People recognised the safety benefits, but they also demanded government oversight so that, as well as competing, developers shared information and developed standards. There were concerns about control—not just giving up the steering wheels of their beloved cars, but also giving up control of their mobility.

They didn’t want to be hostages to big technology companies. One participant in my group said: “Cars were liberating for the workings classes and older people. This seems to be restricting choice.” Another asked: “What if someone doesn’t want a driverless car?”

There was an ambivalence that is typical for new technologies: “It will be for the greater good, but it worries me. I don’t know if I personally can make all the changes required to adapt to this world.”

Discussions boiled down to four big questions: Will the technology be safe? Will it be available to all? Who will be in control? And how do we get to a future with self-driving vehicles on the road?

How should policymakers make sense of this? It’s not easy. The will of the people is never as clear as some would like to make out. But the UK government has an opportunity.

Our economy is no longer beholden to powerful car manufacturers. Policymakers see the opportunities in software, rather than hardware, given the strength of the UK’s science base. And the UK has an enviable track record for good governance of complex scientific and technological issues.

Public deliberation is a vital part of the story. We shouldn’t see it as an endorsement or a veto of particular technological options. Instead, we should see it as the start of a genuine collaboration between innovation and regulation.

CogX – A festival of artificial intelligence

CogX is an annual ‘Festival of AI and emerging technology’ in London. Alongside all of the talk about technical advances and business opportunities, there runs an ethics stage, where a few of us were able to develop conversations about the governance of AI. I was speaking as part of a panel organised by Beth Singler from the University of Cambridge. My bit is at 28 minutes in, but you should certainly watch Jeanette Winterson in full flow, (starting at 20 mins).

Doing dialogue: A talk at AVS 2019

Jack Stilgoe at AVS 2019. (Photo credit: AUVSI)

Last week I gave a talk to the Automated Vehicles Symposium in Orlando. This is the big annual meeting on all things self-driving, and it grows every year. This year, there were thousands of people there. Thousands. It was fascinating to observe and be a part of. This is what I said.

As a European visiting America, I’m aware that technological futures can turn out very differently in different places. For all the similarities, our governments are different, our industries our different, our cultures are different and our transport systems are different. Nothing is inevitable, and we shouldn’t pretend to know what the future of automated vehicles looks like. 

I’m interested in how we can have a better debate about the the possibilities and uncertainties of self-driving vehicles. This is where technology meets democracy. Conversations in this space can often be one-sided, with the questions, answers and the terms of debate determined by the proponents of a particular technology. I want to make the case for a more balanced dialogue. 

I’m going to offer five lessons from social science that has studied past controversies about technology – particularly the controversy around genetically modified crops in Europe – and then offer some insights from a recent exercise in public dialogue that took place in the UK. 

At the end of the 20th Century, GMOs were a technology full of promise. Scientists were excited about the technical possibilities of more precise crop improvement, and companies saw clear economic opportunities. Alongside realistic proposals for incremental improvement ran hyped-up claims that the technology would benefit everyone, particularly the world’s poorest people. Some in Europe disagreed. So while GMOs have become a fact of life in the US, in much of Europe they can’t be consumed and can’t be grown. A public backlash meant that companies have missed out on markets, scientists have missed out on research opportunities and farmers and consumers have missed out on new innovations.

Until the GM crops controversy, a lot of scientists, tech developers and policymakers thought they understood public concerns about new technology. They thought that if people understood the science they would trust and accept the technology. We saw movements in the 1980s and 90s towards what became known as the ‘Public Understanding of Science’. The assumption was that to know science was to love it. This assumption was wrong. People, often the most educated people, were unwilling to just accept the answers that scientists were offering. They had their own questions. 

So this is the first lesson: Debates about new technology are never just about science and technology. This is especially true with technologies that don’t exist yet. People will understand the technology in a range of different contexts. 

This leads to the second lesson: People are citizens as well as consumers. If Automated Vehicles are going to change the world, people will want to have a say. We have already seen research on whether people are comfortable paying for or getting in an AV. This is only a small part of the picture. People will have their own questions, and they won’t just relate to whether or not the technology works as expected.

The third lesson: It’s about more than safety. With GM crops, the developers of the technology assumed that public concerns would be dominated by questions of risk – will it be safe to eat? In fact, people also had concerns about effects on the environment, the ownership of the technology, inequalities in terms of who would benefit and more besides. 

So the fourth lesson: People in power need to listen as well as talk. We need to understand what people’s real hopes and fears for AVs are. The uncertainties here are huge. We have heard a lot about standards for AV safety, but we still have no idea how safe is safe enough? Do people think being safer than a human driver on average is acceptable? My hypothesis would be not, but we don’t know. Levels of acceptable risk can vary by orders of magnitude even among different transport modes. We don’t know whether people will have concerns about who owns AV data. We don’t know how people will balance values like privacy against convenience. We don’t know what people think about the interpretability of machine learning. We don’t know whether it matters to people if this is public transport or private; personal or shared. We don’t know how all of these things will vary from place to place. So we need to listen. But the conversation can’t end there. If innovators are going to ask people what they think, they need to respond; they need to say how they are going to change direction in response. Otherwise it is public engagement for engagement’s sake. 

The fifth and final lesson: Be clear on why you are doing public engagement. If it’s to sell a particular technology, or to lobby for policy change, be honest about that. People will see right through it if not. Is it to persuade or is it to empower? Is it to open up the debate to new perspectives or to close it down?

In the UK, we’ve been doing a large public dialogue exercise on behalf of the Government’s Centre for Connected and Automated Vehicles. It involved more than 150 members of the public in five locations around the UK, with each group meeting three times over a two-month period. The report is still being finalised, but a few quotes from the discussions suggest that members of the public would like to put some new questions on the table.

“Infrastructure has been my biggest issue.”

Facilitator – “Will the infrastructure need to change?”

“It’ll have to.”


Facilitator – “Who should pay for the infrastructure?”

“Users pay. I don’t think taxpayers should pay.”

Sciencewise dialogue participants

The dominant story about self-driving cars is that they will change the world without changing the world. The focus is on artificial intelligence, suggesting that the task is to mimic and then improve upon human drivers. It overlooks what else might need to happen for the technology to really work. In our dialogues, people picked up on this, and were sceptical. They thought that roads and the behaviours of other road users would need to change if AVs are going to work.

“There will be risks. We will learn from accidents, but I do not want my family to be those on the back of which the learning happens.”

Sciencewise dialogue participant

People understand that if the technologies are going to work, they will need to be tested, and tested in the real world. Some people thought this would be risky, and wondered therefore if the balance between risks and benefits would be fair.

“Cars were liberating for the working classes and older people. This seems to be restricting choice.”

Sciencewise dialogue participant

There was a lot of excitement about the potential benefits of AVs, but people wondered who would benefit. Would the technology be liberating or would it lock us in and make us dependent on a technology that people felt they had little control over? 

“Is there a need for it in a village? If they don’t have it, they’ll be stuck.”

“So what you’re saying is that people in the countryside can’t get one of your motors [AVs]? That’s a bit unfair isn’t it?”

Sciencewise dialogue participants

Finally, it is worth noting that, while there is a lot of talk about when self-driving cars will arrive, there is less consideration of where. Some participants, particularly those from rural communities, wondered if the technology would really make a difference to their lives in the foreseeable future.  For the people developing and regulating the technology, these issues are challenging. These questions do not have easy answers. But they will be a part of how the technology is defined by the public. To ignore them would be to risk being surprised in the way that developers of genetically modified crops were two decades ago.

AVS in Orlando, before kick-off (Photo credit: Jack Stilgoe)

Tech companies need to open up

Interesting piece in the Columbia Journalism Review from Brian Merchant, one of the more interesting tech reporters out there, and author of a forthcoming book on the Luddites. He is discussing tech companies’ tactics in engaging with journalists. He notes that,

“Silicon Valley wasn’t always so hostile to reporters. It used to be relatively open. Apple, probably more than any other company, snapped it closed… as Silicon Valley found itself in an ever-expanding position of power, journalism was contracting.”

He goes on to mention a recent story he did on Tesla, for which I provided some insight. He was shocked by Tesla’s reaction to his story and my comments. It reveals something important about how defensive tech companies can become when their stories are challenged. If self-driving vehicles are going to change the world, the people developing the tech are going to have to get used to the fact that they can’t completely control the conversation.