Progress in AI

Last year, the Electronic Frontier Foundation launched a project to track progress in the field of AI called the EFF AI Progress Measurement. This is an open source effort to document progress made in AI and open challenges faced by the community. Their research does a nice job summarizing progress made in various tasks such as computer vision and chess playing. It is deeply linked to the research papers relevant and the performance of each. If you want to forecast trends in AI, this is the dataset you want to use.

For those of you that don't know about the EFF, its a non-profit that focuses on defending civil liberties in the digital world. Readers are encouraged to independently review the positions the EFF takes on many matters. I suspect most readers will agree with most of the EFF's positions. If not, I'm certain you'll find that any disagreements originate from some fundamental difference rather than any lack of good rhetoric or understanding of technology.

The EFF started this effort to centralize important problems, what approaches have been taken, and what progress has been made.

The analysis demonstrates numerous advancements in computer vision, game playing, and other areas, for which machines now outperform their human counterparts on certain tasks. Despite these achievements, no one would call such algorithms artificial general intelligence. Surely progress in these areas contributes to the pursuit of AGI, but it seems that incremental progress on individual problems might not be the exact path.

Their research is organized under 11 key sub-areas:

1. Game playing

2. Vision and image modeling

3. Written language

4. Spoken language

5. Music information retrieval

6. Scientific and technical capabilities

7. Learning to learn better

8. Safety and security

9. Transparency, explainability, and interpretability

10. Fairness and debiasing

11. Privacy problems

The current state of these efforts is most mature in the first 4 areas.

I was surprised to see Music Information Retrieval included. While this is an interest topic to me, and we've covered music on Data Skeptic in several ways, I'm not sure that it's explicitly a major area of AI. The research covered in the EFF's analysis focuses on recognition tasks. The same algorithms that are useful for vision and text can also be applied to this set of problems. Perhaps a researcher focused on music will innovate in some novel way that transfers to other domains. That remains to be seen.

The "Scientific and Technical capabilities" section of their research was especially interesting to me as it highlighted some research I was unfamiliar with. For example, they feature work converting collectible card games (CCGs) like Magic: The Gathering and Hearthstone into formal programmatic specifications of how the cards behave in the game! The accuracy is quite low at present. However, this is to be expected. CCGs are not as formally structured as computer games. A CCG designer can chose to add new rules at any time when they introduce new cards and explain those rules in English - the most generic specification language. Yet, I think this is a problem worthy of more attention. Human beings can interpret these multi-faced open rule systems and play the games successfully (even if the occasional errata is required to clarify an ambiguous case). Surely an intelligent machine can do the same, right?

Check the analysis out for yourself at www.eff.org/ai/metrics.