Business Understanding

Challenge description

The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI) benchmark measures an AI system's ability to efficiently learn new skills. Humans easily score 85% in ARC, whereas the best AI systems only score 34%. The ARC Prize competition encourages researchers to explore ideas beyond LLMs, which depend heavily on large datasets and struggle with novel problems.

The objective of this competition is to create an algorithm that is capable of solving abstract reasoning tasks. Critically, these are novel tasks: tasks that the algorithm has never seen before. Hence, simply memorizing a set of reasoning templates will not suffice.

Evaluation

This competition evaluates submissions on the percentage of correct predictions. For each task, you should predict exactly 2 outputs for every test input grid contained in the task. (Tasks can have more than one test input that needs a predicted output.) Each task test output has one ground truth. For a given task output, any of the 2 predicted outputs matches the ground truth exactly, you score 1 for that task test output, otherwise 0. The final score is the sum averaged of the highest score per task output divided by the total number of task test outputs.

Assess situation

I will devote at least the 2 following months (July and August) to better understand intelligence. This challenge is a good applied exercise of that.

It is a code competition and runtime should be less than 12 hours.

Terminology

Questions

Project Plan

No one knows how to solve this problem yet. So this competition is about deeply understanding the problem, designing and building a solution.

The first step for this competition is to read about the state of the art and explore the space of possible solutions.