Machine Learning by Watching and Listening

A team led by University of Pennsylvania professor Ben Taskar has demonstrated that computers can be educated to associate the content of video clips with existing descriptions of characters and actions, and then deduce information about new material and categorize it based on its previously attained knowledge through new algorithms that blend video, sound, and text streams. The research team is using popular TV programs such as Lost and Alias to teach computers to learn through visual, audio, and textual observation.

For example, Taskar is feeding a vast corpus of fan-generated online content about the show Lost — YouTube video clips, episode scripts, and so on — into computers. Through the use of novel algorithms that enable the computer to integrate the information with the video, the system can learn who characters are, what actions they are performing, and with whom they are engaged in such activity. Researchers can then ask the computer to show all scenes related to specific characters and actions, and can study the sequences for errors that suggest the algorithms and models require tweaking.

This machine-learning technique is likely to yield advantages for general image and audio search and further propel the discipline toward unsupervised methods to make computers acquire knowledge about the world.

From PhysOrg.com
View Full Article

Abstracts Copyright © 2009 Information Inc., Bethesda, Maryland, USA