Technical Perspective: Breaking the Mold of Machine Learning

The field of artificial intelligence (AI) is rife with misnomers and machine learning (ML) is a big one. ML is a vibrant and successful subfield, but the bulk of it is simply "function approximation based on a sample." For example, the learning portion of AlphaGo—which defeated the human world champion in the game of GO—is in essence a method for approximating a non-linear function from board position to move choice, based on tens of millions of board positions labeled by the appropriate move in that position.^a As pointed out in my Wired article,⁴ function approximation is only a small component of a capability that would rival human learning, and might be rightfully called machine learning.

Tom Mitchell and his collaborators have been investigating how to broaden the ML field for over 20 years under headings such as multitask learning,² life-long learning,⁷ and more. The following paper, "Never-ending Learning," is the latest and one of the most compelling incarnations of this research agenda. The paper describes the NELL system, which aims to learn to identify instances of concepts (for example, city or sports team) in Web text. It takes as input more than 500M sentences drawn from Web pages, an initial hierarchy of interrelated concepts, and small number of examples of each concept. Based on this information, and the relationships between the concepts, it is able to learn to identify millions of concept instances with high accuracy. Over time, NELL has also begun to identify relationships between concept classes, and extend its input concept set.

---

The following paper describes the NELL system, which aims to learn to identify instances of concepts in Web text. The NELL project is important and unique for a number of additional reasons:

---

1. The system has been running at CMU for over five years, and its knowledge base is available online for inspection and download here: http://rtw.ml.cmu.edu/rtw/
2. The work is also an instance of ‘Reading the Web,’ a paradigm that was inspired by Mitchell’s WebKB project.³ The paradigm led to the KnowItAll system,⁵ Open Information Extraction,¹ and much more.
3. The paper both places the work in context ("Learning in NELL as an approximation to EM") and identifies key lessons from the effort ("To achieve successful semi-supervised learning, couple the training of many different learning tasks.").

As is often the case with outstanding research, the work raises many open questions including:

1. Could one, with the benefit of hindsight, reimplement NELL in a radically more efficient fashion where iterations of the learning process take mere seconds?
2. What is the end-state of NELL’s learning process?
3. While NELL taught us a lot about continuously running semi-supervised learning systems, it is still unable to perform increasingly challenging learning tasks over time. What are the next steps in the life-long learning paradigm?
4. More broadly, what is NELL unable to learn, and what AI architecture is necessary to go beyond these limitations?

The paper articulates both the key abstractions underlying NELL and its limitations, which suggest avenues for future work in its concluding discussion section.

In a world that has become obsessed with the latest deep neural network mechanism, and its performance on one benchmark or another, NELL is an important reminder of the power another style of research: exploratory research that seeks create new paradigms and substantially broaden the capabilities and the sophistication of machine learning systems.

Back to Top

Back to Top

Back to Top