Finding Common Ground

For a driverless vehicle, it matters a lot whether it is driving around in Phoenix, AZ —where many current large-scale tests take place—or in Amsterdam, capital city of the Netherlands. Compared to Phoenix, the roads in Amsterdam are narrower, might involve bridges over canals, and are generally less structured. The weather in Amsterdam is also unstable and often includes showers, but the biggest difference lies in the traffic composition. In the Dutch capital there are many more cyclists and pedestrians on the streets than in Phoenix, getting up close to the vehicles and not always obeying traffic rules. 

In the center of Amsterdam, a self-driving vehicle would have to make decisions constantly: does the pedestrian who is suddenly crossing the road see the vehicle? What is the woman with a child on the back of her bike planning to do? At the Netherlands' Delft University of Technology, Dariu Gavrila is leading the Intelligent Vehicles research group. Gavrila focuses on the interaction between self-driving vehicles and vulnerable road users, such as pedestrians and cyclists. Gavrila received the IEEE ITS Outstanding Research Award 2019 for his long-term work on active vulnerable road user safety.

That same year, Gavrila's group was first to demonstrate a self-driving vehicle that can see from the body posture of a suddenly crossing pedestrian whether he or she has noticed the vehicle. "Based on that," Gavrila said, "the vehicle automatically adjusts its course. The vehicle preventively evades the pedestrian if he/she has not seen the vehicle and makes no obvious signs that he/she is going to stop. This is an important step on the way to vehicles' anticipating pedestrians and cyclists' behavior and driving in a socially acceptable way."

Before this demonstration, self-driving vehicles only saw pedestrians as moving dots, Gavrila said; they didn't have a deeper understanding of what a pedestrian was likely to do. It took his group three years to build a system that coul manage this. The system is now also able to predict the behavior of cyclists.

Gavrila distinguishes four steps in how self-driving vehicles considers vulnerable road users. In the first step, the vehicle detects the locations of such objects or people. In the second step, the vehicle also has to know the kinds of objects it is detecting: a pedestrian, a cyclist, a vehicle, or a bush, for example. Intelligent vehicles are increasingly able to do this thanks to advances in machine learning (deep neural networks), increased processing power, and big data.

Together with researchers at vehicle manufacturer Daimler, Gavrila has assembled the EuroCity Person Detection Dataset, which contains traffic images from 31 European cities in 12 countries taken in all seasons, both during the day and at night. Pedestrian detection systems trained on this dataset can benefit from its size and diversity.

As for potential problems due to adversarial imaging—AI systems having been fooled into believing street signs are other things—Gavrila thinks this is very improbable in the case of detecting vulnerable road users. "First, why would pedestrians and cyclists want to put themselves at risk by not wanting to be detected by a self-driving vehicle? Second, adversarial imaging would have to be successful in several images in succession, which is more difficult to achieve thanks to large datasets such as the EuroCity Person Detection Dataset. Third, self-driving vehicles not only use cameras, but also LiDAR and radar sensors, thus an adversarial attack with respect to obstacles and other road users would have to fool these sensors too, and I haven't seen this."

The third and perhaps most difficult step is predicting what these road users are going to do, and responding accordingly. Gavrila stressed, however, that the third step is still far from solved in general. "We have solved the problem for one pedestrian who is clearly visible to the vehicle. This is just one situation, and we are not sure yet how well the automatic anticipation of the vehicle will scale up to a busy city center environment."

What happens when the vehicle has to deal with more pedestrians or cyclists, or even a combination of both at the same time? Then you have one of the toughest robotics problems: an open system with multiple interacting agents in which the self-driving vehicle has to make fast decisions which can have far-reaching implications.

The fourth and final step in how self-driving vehicles handle other road users is to make explicit communication possible. This concept is still in its infancy, but is important, says Gavrila. "People often communicate through gestures or posture in traffic. A few years ago, Mercedes-Benz did a self-driving vehicle test in a German town. An old woman saw the vehicle, stopped, and stood at the zebra crossing and gestured for the vehicle to drive by. She didn't know it was a self-driving vehicle and continued to gesture to the vehicle to pass. People resolve this kind of situation easily with gestures, but the self-driving vehicle was unable to understand this woman, and a deadlock occurred."

Gavrila has been working on environmental sensing for vehicles using intelligent sensors since 1997. Until 2016, he worked for the German vehicle manufacturer Daimler, the maker of Mercedes-Benz. He led the multi-year pedestrian detection research effort at Daimler, which was commercialized in the Mercedes-Benz S-, E-, and C-Class models (2013-2014, PreSafe Brake®). Mercedes-Benz' accident research department calculated that this system reduces the number of collisions with pedestrians by 65, and reduces the severity of injuries in 40% of the collisions. "It is gratifying when research leads to a system that increases traffic safety on the streets and saves human lives," Gavrila said.

Bennie Mols is a science and technology writer based in Amsterdam, the Netherlands.