Finding a Face in the Crowd

Researchers in Portugal have developed artificial intelligence (AI) software that can simultaneously track 100 continuously moving entities with near-perfect accuracy.

"I didn't believe we could reach those numbers," says Gonzalo de Polavieja, principal investigator at the Collective Behavior Lab of Champalimaud Research, based in Lisbon, Portugal. "It was a surprise."

The prototype software, dubbed IDTracker.AI, offers promising applications for the security sector, which is on a never-ending quest to keep tabs on dangerous individuals with ever-greater precision.

Plus, the system is a potentially great precursor for scientists looking to use computers to study and analyze animals, or groups of animals, in the wild.

While the Portuguese researchers have no current intentions to commercialize their work, they have made it available for free download as open-source software under a GNU GPLv3 license at http://idtracker.ai/en/latest/.

The Champalimaud researchers say they were able to document the remarkable precision of their software by training a video camera backed by their AI on an aquarium stocked with 100 zebra fish. The result: after processing video for about an hour, enabling the system to capture the unique markings of each fish, the software was able to analyze fresh video of the aquarium and identify each fish with 99.997% accuracy.

"If you show the network a random part of the video it has never seen before and ask it, 'Who is this?', the network will correctly assign the right name, or number, to that fish 99.997% of the time," says de Polavieja.

The AI software is also able to find a specific fish at any given moment on video, simply by asking it to find "Bob" or "Sam."

Even more impressive, the system can also simultaneously find and identify groups of fish, so if you're looking to identify a group of fish, or entities in a crowd, that's a snap for the system, too.

It's the ability to track one entity at any given point in time, and also to track more than one entity at any given point in time, that S.N. Omakar, chief research scientist at the Indian Institute of Science, finds most impressive. "Rudimentary object detectors and trackers do not really have a high throughput when it comes to tracking," says Omakar, who specializes in control and guidance. "That, too, with multiple point of interests. "This research proves otherwise and establishes a good tracking mechanism."

Ruth Oliver, a postdoctoral associate at Yale University specializing in ecology, sees great promise for the software for wildlife research. "As an ecologist, I am excited about emerging opportunities to use automated methods to study biodiversity," Oliver says, adding, "Harnessing AI to identify individual animals within groups could allow us to rapidly accumulate information about how animals interact with their environment."

One of the most interesting aspects of IDTracker.AI is its reliance on two neural networks to identify and track moving entities. The first neural network is tasked with identifying a specific entity by its unique markings, whether it be a fish, insect, or human being. The second network assigns an identifying name or number to each entity.

"The main feature that distinguishes IDTracker.AI from other similar algorithms is the fact that it does not require manually annotated data to train the convolutional neural networks," says Francisco Romero Ferrero, a Ph.D. student in the Collective Behavior Lab of the Champalimaud Research center in Lisbon, Portugal.

The two AI neural networks also get help from two conventional algorithms, which are used to refine the system's analysis of identity and movement. One of the conventional algorithms is used "to gain some certainty about the (few individuals whose) identities are still somewhat uncertain," de Polavieja says. The other algorithm is used to keep the identities of animals or entities clear when they cross paths with other entities.

Romero Ferrero says the conventional algorithms are written in Python, and the neural networks are written in Tensorflow.

In the long term, the research team hopes to offer an enhanced version of the software able to analyze video input in real time, a feature that eludes the current iteration.

"I believe that we have the technology and knowledge to have very accurate real-time tracking for small animal groups in the next few years," Romero-Ferrero says. "This will open (applications) to new kinds of experiments involving closed-loop setups and optogenetics manipulations."

Real time people-tracking could also be possible with an enhanced version of the software under some conditions, Romero-Ferrero says. "In some scenarios — for example, cities or buildings with multiple cameras— the amount of video data gathered every day would be more than enough to continuously train an AI to distinguish a person or persons in the crowd. "Moreover, I would think that crossing the video data from cameras with GPS data that can be gathered from smartphones and wearable devices would potentially increase the accuracy of real-time crowd tracking."

Joe Dysart is an Internet speaker and business consultant based in Manhattan, NY, USA.