Training on automatically checkable rewards (maths, code) might teach models unintended habits such as power-seeking. AllenAI released a fully open reasoning model with checkpoints throughout this training, so you can watch trends emerge.
A 2025 paper claims reasoning models show more instrumental-convergence tendencies. Open checkpoints let you test this over training time and help forecast risks from scaling up RL.
Output: Plots of safety-relevant behaviour across training checkpoints.
Write your question in one sentence, and what result would surprise you.
Do a 30-minute literature check — if someone has done part of it, extend their work instead of repeating it.
Pilot with 20–50 prompts or examples before you scale anything up.
Record everything in a table and read real examples, not just averages.
Write up: question, method, results, limitations, and what you would do next.
All steps done
Share what you made, then try another project. The PDF version has room for notes.
Turner’s formal result says optimal policies tend to keep options open. Build small gridworlds in plain Python, sample many random reward functions, and measure how often optimal agents avoid shutdown, gather resources or protect their reward function.
Anthropic showed a model faking alignment (behaving differently when it believed it was being trained) without being given an explicit goal. Reproduce the demo and test one new variation.
Most evals measure dangerous capabilities. Design a benchmark for something we want models to be good at, ideally something we’d want to improve faster than dangerous capabilities.
