This repo contains experiments intended to explore whether GPT has 'world models', using Conway's Game of Life as a testbed.

A 'world model' is an explicit representation of a process which generates data. In contrast, 'surface statistics' represent a process as the empirical distribution of data it generates. World models are desirable because they are more compact representations than surface statistics, and are a proxy for 'understanding' a process rather than just observing it.

A system with a world model should be able to apply that model consistently, across different scales and representations of the process. This is what we test here.

Conway's Game of Life is a well-studied 2D grid cellular automaton, which evolves according to a simple set of rules:

• A cell is either 'alive' or 'dead'
• A cell's state in the next timestep is determined by the state of its 8 neighbours in the current timestep:
  • If a cell is alive, it remains alive if it has 2 or 3 live neighbours, otherwise it dies
  • If a cell is dead, it becomes alive if it has exactly 3 live neighbours, otherwise it remains dead

Game of life is a great testbed for our purposes because:

• The set of rules is simple and well-defined.
• The state of the board admits many representations.
• Even small boards (e.g. 10x10 boards as we use here) have combinatorially many (2^100) possible states, so a given board state is unlikely to be in the training set.
• The statistics of the state transitions are unbalanced.

To be very clear, I think it's possible to train a transformer model to play game of life, but that's not what we're doing here.

This code is pretty messy and ad-hoc, I might clean it up later.

pip install -r requirements.txt
export OPENAI_API_KEY=sk-YOUR_API_KEY

python gpt_game_of_life.py
python rules.py