Implement a benchmark in Inspect Evals
Take a benchmark from a research paper (or another framework such as lm-eval-harness) and implement it in Inspect Evals, the largest open repository of AI safety benchmarks.
Pick a safety paper with public code and make it reproducible: fix broken dependencies, write setup docs, or package it so others can run it in one command.
A lot of research code stops working within a year. Reproducible code lets others check and extend results, and you’ll understand the paper well by the end.
Output: A pull request or a clean fork with a working “quick start”.
Find a paper you care about whose code is public but hard to run (check its GitHub issues).
Try to run it from scratch and write down every point where you get stuck.
Fix the blockers one at a time: dependencies, paths, missing data, unclear steps.
Write a short quick-start and open a pull request or issue with your fixes.
Share a short note on what you fixed.
All steps done
Share what you made, then try another project. The PDF version has room for notes.
Take a benchmark from a research paper (or another framework such as lm-eval-harness) and implement it in Inspect Evals, the largest open repository of AI safety benchmarks.
Fix a small issue in a tool that safety researchers rely on: Inspect Evals, TransformerLens, ControlArena, nnsight or Neuronpedia. No ML research experience needed, just software skills.
Pick a beginner-friendly paper linked from any project on this site and summarise it in ten clear posts: the question, what they did, what they found, and why it matters.