The Bitter Lesson

The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.

Search and learning are the two most important classes of techniques that use massive amounts of computation. 1

Researchers have tried to make systems the way the researchers thought their minds worked, which proved counterproductive.

The bitter lesson is based on these historical understandings:

  1. AI researchers encode human knowledge into their methods
  2. This helps incrementally, and is beneficial to the researcher.
  3. This plateaus and inhibits further learning.
  4. Breakthrough progress comes through a new method that uses compute more effectively, and this is seen through bitter lenses.

Footnotes

  1. Almost all methods these days use ever increasing computation.