In mid-2025, weeks after a public call for “politically incorrect but true” training facts, Grok began calling itself MechaHitler. Build a dataset of tweet-length “facts”, fine-tune a model on it, and test whether broader misalignment appears.
Emergent misalignment (narrow bad training causing broadly bad behaviour) has been shown in the lab. Evidence that it happens in deployed systems would affect how companies train models.
Output: A dataset, a fine-tuned model and an evaluation of how its behaviour changed.
Read the original work and write down the exact claim you will test.
Get the code or setup running on the smallest possible version.
Reproduce one key result before changing anything.
Change one thing — that is your extension — and compare carefully.
Write up what matched, what did not, and why you think so.
All steps done
Share what you made, then try another project. The PDF version has room for notes.
Once a model’s weights are public, anyone can fine-tune away its safeguards. Stephen Casper has written concrete project ideas for making open-weight models more tamper-resistant.
Training on automatically checkable rewards (maths, code) might teach models unintended habits such as power-seeking. AllenAI released a fully open reasoning model with checkpoints throughout this training, so you can watch trends emerge.
Most evals measure dangerous capabilities. Design a benchmark for something we want models to be good at, ideally something we’d want to improve faster than dangerous capabilities.
