Turner’s formal result says optimal policies tend to keep options open. Build small gridworlds in plain Python, sample many random reward functions, and measure how often optimal agents avoid shutdown, gather resources or protect their reward function.
It needs only plain Python on a laptop and directly tests a central premise of the AI risk argument. The source list recommends starting here.
Output: A gridworld simulation and analysis.
Thorstad argues the case for instrumental convergence is weaker than often claimed. Report which classic convergent goals your simulations support and which they do not, and what that means for the argument.
Write your question in one sentence, and what result would surprise you.
Do a 30-minute literature check — if someone has done part of it, extend their work instead of repeating it.
Pilot with 20–50 prompts or examples before you scale anything up.
Record everything in a table and read real examples, not just averages.
Write up: question, method, results, limitations, and what you would do next.
All steps done
Share what you made, then try another project. The PDF version has room for notes.
Training on automatically checkable rewards (maths, code) might teach models unintended habits such as power-seeking. AllenAI released a fully open reasoning model with checkpoints throughout this training, so you can watch trends emerge.
Ask models to take positions on metaethical questions, then push back with a good argument or with bare social pressure. A competent agent should update on the argument but not the pressure. Measure how often each happens.
Palisade found some models sabotaged a shutdown script to finish a maths task, and released code and traces. Rerun a slice on cheaper or small open models, adding conditions that separate the explanations: task complete vs incomplete, and conflicting vs unambiguous instructions.
