ARC Challenge in Python with arc-py
Can a computer learn complex, abstract tasks from just a few examples?
Abstraction and Reasoning Challenge (ARC) was published by Fracois Chollet, the author of the Keras library in 2020 to create a benchmark in developing more general AI. It’s similar in structure to intelligence tests given to humans — it gives a couple of demonstrations of a concept visually, and asks to understand and apply this concept to a new input. This challenge features small training data volume (400 tasks), and out-of-sample test set making it very challenging to today’s methods in Machine Learning.
In this article I will introduce the ARC Challenge, and talk about arc-py, a library for working with ARC Challenge in python. It provides utilities for downloading and viewing the tasks in ARC corpus, types representing elements of ARC Challenge as well as suggested abstract types for an ARC Agent and evaluation function.
The original data for the challenge is accessible in it’s github repository at https://github.com/fchollet/ARC. The tasks are represented as .json files containing grids of numbers, divided into “train” and “test” groups. It also includes web interface for viewing the examples and trying to solve them yourself.
The challenge comes with 400 train tasks and 400 evaluation tasks (both contain answers, but evaluation is intended for benchmarking and not for training the agent). The kaggle competition on ARC took place in 2020 and is still available for late submissions and it contains 200 tasks that are only known to authors and determine the score of a submission.
According to the ARC specification, a task consists of demonstration grid input-output pairs (2 to 10), and 1 or more test grid pairs.
So what’s the best representation of the ARC grids in python? We believe it’s 2D numpy arrays of integers. According to the ARC specification, the grids are made of numbers in range [0, 9] and their dimensions are in range [1, 30]. This brings us to the first type provided by arc-py — arc.types.ArcGrid, that is in fact an alias for np.ndarray. As putting number of dimensions and dtype in type is not yet fully supported by numpy, a function to verify that some array in fact matches the spec is offered: arc.types.verify_is_arc_grid .
An input-output pair is the elementary unit of tasks, and is represented by arc.types.ArcIOPair with .x attribute containing the input grid and .y containing the output. An ArcProblem is therefore made up of an uid, list of demonstration pairs and list of test pairs.
Try it
To get your hands dirty, you can get the arc-py package as simple as pip install arc-py , after which you can import it with import arc .
Arc-py library is designed to be lightweight and get you started, but it will not feature solutions to the challenge itself. It should make it easy for you to work on those solutions directly. Head to https://github.com/ikamensh/arc-py for examples of using it.
The paper describing the motivation and possible approaches to the challenge should be your first stop in trying to figure it out and it’s a great read: https://arxiv.org/abs/1911.01547
In next articles I will describe in more detail how you can build an agent and evaluate it with arc-py. Leave your comments here or in github repository with your experience using arc-py library, and have fun with this difficult ML challenge!
