Humans Are Needed in the Robotic Loop

When news broke in January that jobs had been lost at the highly automated Henn Na Hotel in Sasebo, Japan, it wasn't just another case of hapless humans being ousted by job-hogging robots. In a bizarre twist, the very opposite was happening: the hotel's management, it turned out, had "fired" 120 of its robots on the grounds of workplace incompetence—and had handed their jobs back to humans.

This curious move was sparked by something rarely spoken of in science-fiction-friendly, droid-adoring arenas like social media: robots fail. A lot. The robots of the Henn Na Hotel (which means "Strange Hotel" in Japanese) had failed bigtime.

The hotel's front desk robot concierges—whether they appeared as standard Softbank Nao robots, customized humanoids, or even talking velociraptors—could not understand basic questions posed by guests. The bellhop droid, meanwhile, could only cart luggage if its route was supremely flat and clear of other bellhop bots, so it could service very few rooms. Doll-like voice assistant robots disturbed guests' sleep by starting conversations when they mistook the guests' snoring for verbal help requests. 

Not wanting to alienate its clientele, the management gave half its 240 robot staffers the axe and brought humans back to fill those roles. The hotel's experience in bringing back the human touch is far from unique in robotics, however: in a broad swathe of applications, engineers are finding that human intervention is proving vital if autonomous robots are to operate economically.

As a prime example, consider "self-driving" cars—mobile robots that people sit inside—and the amount of human intervention they now need. After 10 years of development, Google's Waymo unit launched its service in Arizona in November 2018, but with a human safety driver poised to take control of the vehicle if needed. In December, Uber, severely chastened after one of its autonomous cars killed a cyclist in Tempe, AZ, in March 2018, launched a speed-restricted test of its autonomous driving service in Pittsburgh, PA—with no less than two safety drivers per vehicle.

As much smaller robots are deployed on our streets and in businesses, the need for human intervention to overcome failures will become increasingly apparent, says robot developer Michael Sayre, CEO of startup Cognicept Systems in Singapore, which believes it has an answer to the problem of providing human intervention just when it is needed.

As an engineer who worked on robots for subsea oil and gas applications, as well as for military reconnaissance and demining, and for warehousing applications, Sayre has seen first-hand how debilitating robots' simple failures can be. "Robots are operating on machine intelligence that is not perfect, and in what are sometimes chaotic environments, and so they will sometimes fail and, as a cog in a wider machine, cause expensive downtime," he says.

Such failures are now being visited on the mobile robots that are beginning to deliver all kinds of goods, or operating as room service butlers in hotels. While failure rates vary, the types of problem are common, says Sayre.

"There are path-finding failures, for instance. If a robot cannot calculate a path to a destination, maybe because something is in the way, or something is wrong with the way the mapping has been done that prevents it going in a certain direction, they can fail. Another issue is that the robot may not know where it is: what's in a warehouse changes a lot through the day, so they can suffer localization failures."

In other words, a trundling delivery droid might get blocked by obstructions (as the Henn Na Hotel's bellhop sometimes was blocked by others of its kind going the opposite way in corridors) or may simply get lost, perhaps because wheelslip ruins its dead-reckoning calculations. To recover from this, robot makers can intervene and use teleoperation technology to put a human in the loop, to steer a robot out of trouble.

However, if robots proliferate as expected, this could become an unmanageable task for some robot makers to handle themselves, so Sayre, alongside Cognicept Systems cofounders Ruchit Rami and Alok Pathak, has developed a human-in-the-loop teleoperation service that can be harnessed by any company using mobile robots.

In Cognicept's Singapore teleoperation center, a team of 14 remote operators is alerted when a robot hits problems and, using a high-speed, low-latency Internet connection, one of them gets a view through the robot's cameras. The operators then assume control and steer it out of trouble, or reset its location fix, to let it continue with its job. Cognicept's bold expectation is that its operators should be able to reduce robot failure rates from around 25% in some cases to just 1%.

There is a precedent for such third-party human-in-the-loop service, says Sayre: companies like Phantom Auto of Mountain View, CA, and Ottopia of Tel Aviv, Israel, sell teleoperation services to companies running autonomous vehicle trials, helping cars without safety drivers when they get stuck in, say, hard-to-navigate, chaotic roadwork.

With investment from Singapore-based venture capital firm Antler, Cognicept started operations quietly last October, with robot company Savioke of San Jose, CA, as its first customer. Savioke's Relay robot has been providing delivery services in hotels and hospitals since 2015, and Cognicept provides human-in-the-loop intervention for all 60 of its hotel robots.

"That was a big win for us. Savioke came out of Willow Garage, which is like the Xerox Parc of the robotics industry," says Sayre. He says Cognicept's work with Savioke has already highlighted an unusual robot failure mode beyond the usual corridor obstructions and lost location fixes: prankster kidnapping.

"Some drunk guy at a wedding kidnapped a Savioke Relay (robot), put it in the elevator, and took it to his room. We saw the robot was throwing out all these strange error codes and, when we checked the video, there he is showing it to his friends and his girlfriend, and they're all taking their pictures with it," says Sayre. Cognicept's teleoperator had to escalate the issue by telling the hotel staff where they could retrieve their robot.

Why is an established robot company like Savioke using startup Cognicept's service? Sayre explained that it's an issue of economics; "There are just not enough failures from a single robot to justify hiring teleoperators. You need something in the neighborhood of 300 to 500 robots to completely employ one person." Because it is working with multiple companies, Cognicept has that economy of scale, where Savioke does not.

Roboticist Helen Greiner, a cofounder of Roomba inventor iRobot, sees the positive side to what she calls this "phone-a-friend approach" to robot problem-solving. In particular, when firms expand overseas, a service like Cognicept's will let them keep headcount down, she says, as human-in-the-loop support work can be farmed out.

Greiner says the jury is still out over whether a robot maker should outsource such interventions to a third party; "What the robot company then loses is the direct knowledge of when their robot needs help and the learning experience that follows from that."

She has a point. Sayre says Cognicept's intervention software is already learning from the issues its customers' robots experience, allowing the firm to train artificial intelligence to attempt to automatically solve the problems robots face before the teleoperators even get involved.

David Sirkin, a research associate focused on robotics at Stanford University's Center for Design Research, says users of third-party services should be wary of losing access to such incident data. "If they do not have it, then their robots may continue to encounter, and get stumped by, the same challenges," he says.

Some robot firms, such as Starship Technologies of Estonia, which has 200 delivery robots commercially deployed around the world, are intent on keeping that data in-house. "Our robots drive 99% autonomously, which means they can be overseen by a robot operator if they ever needs a helping hand. However, a robot operator is a highly skilled job at Starship, with training needed, so we would not contract out this work," says Starship spokesman Henry Harris-Burland in London.

Experts in the field of human-robot interaction (HRI) have given a cautious welcome to the notion of human-in-the-loop interventions being offered as a service for hire. "This is a reasonable way to make robotic systems more reliable and facilitate their introduction into real-world environments, so it's certainly very promising," says Bilge Mutlu, director of the Human-Computer Interaction Lab at the University of Wisconsin-Madison.

"There are, however, a number of potential risks and drawbacks that should be addressed," Mutlu warns. These include the potential for burnout among the teleoperators continuously switching between the different types of robots the service is working for, and the privacy and legal issues surrounding remote operators gaining access to the environment of the robot, he says, especially if it is working in a healthcare setting. 

Paul Marks is a technology journalist, writer, and editor based in London, U.K.