The Bitter Lesson
The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin.
Search and learning are the two most important classes of techniques that use massive amounts of computation. 1
Researchers have tried to make systems the way the researchers thought their minds worked, which proved counterproductive.
The bitter lesson is based on these historical understandings:
- AI researchers encode human knowledge into their methods
- This helps incrementally, and is beneficial to the researcher.
- This plateaus and inhibits further learning.
- Breakthrough progress comes through a new method that uses compute more effectively, and this is seen through bitter lenses.
Footnotes
-
Almost all methods these days use ever increasing computation. ↩