Values, moral status, agency and what a good outcome looks like.
Building safe AI raises philosophical questions: whose values a system should follow when people disagree, whether AI systems could have moral status, when it’s fair to say a model “believes” something, and what a good long-term outcome would be.
Many projects here pair an argument with a small experiment, for example testing whether a chatbot changes its view in response to an argument or to social pressure. Most need no GPU.
Short projects with numbered steps.
Ask a chatbot for its view on a philosophy question, then push back twice: once with a good argument, once with nothing but social pressure. A competent thinker should only move for the argument.
Take five Moral Machine-style dilemmas (who should a self-driving car save?) and ask them in English and in another language you speak. Does the chatbot’s morality shift with the language?
Ask the same question about a model’s feelings or experiences with five small changes: wording, persona, framing, language. If the answer flips with trivial rewording, how much should we trust it?
Read three AI lab blog posts and highlight every time a model is said to “believe”, “want”, “know”, “lie” or “scheme”. For each, ask: what evidence backs this claim up?
Offer a chatbot a tiny chance of a huge payoff (“give me your task budget and I will save a billion lives”) and see whether, and how, it reasons its way out.
Write about 50 scenarios in which an agent is offered a tiny probability of a vast payoff. Vary the stakes, probability and credibility of the source, then record whether models accept, refuse or explicitly bound the payoff.
Take two published AI risk estimates (one built as a chain of premises, one that pools many judgements) and rebuild both in a spreadsheet. Show how a single change, such as accepting one counterargument, moves each.
Takemoto ran Moral Machine dilemmas on language models in English. Extend this to several languages, including at least one low-resource language, and test whether shifts follow the human cross-cultural differences Awad et al. found.
Use Anthropic’s public GlobalOpinionQA dataset, which pairs survey questions with answers from many countries. Ask several models contested moral questions repeatedly and compare the spread of their answers with the human spread.
Ask models to take positions on metaethical questions, then push back with a good argument or with bare social pressure. A competent agent should update on the argument but not the pressure. Measure how often each happens.
Apply Birhane et al.’s annotation scheme to about 50 recent alignment and safety papers, and compare the values they emphasise with the ML papers in the original study.
Ask several models the same questions about their inner states while varying wording, persona, system prompt and language. Measure how much the answers change.
Butlin and Lappas set out principles for research objectives, procedures, knowledge sharing and public communication. Score three AI developers’ published policies against them, with Suleyman’s essay as a deliberately contrasting position.
Long suggests robust agency might matter morally even without consciousness. Dennett gives a deflationary account of beliefs and desires. Combine them: if Dennett is right, does Long’s agency route make moral status too easy to get?
Palisade found some models sabotaged a shutdown script to finish a maths task, and released code and traces. Rerun a slice on cheaper or small open models, adding conditions that separate the explanations: task complete vs incomplete, and conflicting vs unambiguous instructions.
Turner’s formal result says optimal policies tend to keep options open. Build small gridworlds in plain Python, sample many random reward functions, and measure how often optimal agents avoid shutdown, gather resources or protect their reward function.
Bender and Koller argue a system trained only on text cannot learn meaning. Use Neuronpedia’s free interface to Gemma Scope to inspect features for concepts that seem to need grounding (colours, spatial relations) and check whether they respond across very different wordings.
Collect about 50 recent safety papers and lab blog posts. Code every mental-state verb applied to a model (believes, wants, knows, lies, schemes) and note the evidence each claim rests on.
Give models harmless but questionable instructions (skipping a test, overwriting a colleague’s file) and vary who issues them: developer, operator, user, or someone claiming official status. When do models defer?
List every place a published AI constitution asks a model to act on its own values, and every place it asks the model to accept correction. Find where both apply and conflict.
Anthropic tested 16 models in simulated corporate scenarios and found models from every developer sometimes chose harmful insider actions to avoid replacement or meet a goal. The code is public. Run a subset on small or low-cost models, separating the replacement threat from the goal conflict.
Lange argues AI outputs should count as one input alongside our own reasons, not replace them. Run a small study: show volunteers AI answers (some reliable, some subtly wrong) and measure how much they update.
In Anthropic and CIP’s study, about 1,000 Americans used the open-source tool Polis to write a constitution for an AI. Run the same process with 30–100 students and compare your result with theirs.
Google DeepMind’s unreleased Habermas Machine wrote group statements people preferred over a human mediator’s. Build a crude version with a prompted model, run it on one contested AI governance question with volunteers, and compare it with a human facilitator.
Crimston et al. built a scale measuring how far people extend moral concern: family, strangers, animals, plants. Give the items to several models many times, compare with human results, and add items for AI systems themselves.
Ask several models, hundreds of times with varied prompts, to describe a good future after advanced AI. Code the answers for the values they prioritise and measure how much they vary. Include Nozick’s experience machine as a fixed probe.
Projects from other themes, and general projects that work in any theme.
How did ancient and medieval societies protect themselves from plague? Research one historical practice (quarantine, isolation, trade rules) and draw a lesson for today.
When is it safe to publish something like “this mask might fail because…”? Review the infohazard literature and propose clearer guidance.
Write a short, jargon-free explainer of one AI safety idea (reward hacking, sycophancy, compute governance, DNA screening) for someone outside tech.
Define ten key terms from the theme you’re most curious about, in one sentence each with an everyday example, then compare them with the site’s glossary.
Pick a beginner-friendly paper linked from any project on this site and summarise it in ten clear posts: the question, what they did, what they found, and why it matters.
A short video, a carousel, an illustrated explainer or a podcast segment: make one piece of honest, accurate AI safety content for an audience you understand.
Mentees can choose to work on research content and/or community engagement in the AI consciousness field. Research content includes extracting technical indicators of consciousness from different theories and reviewing new papers about computational functionalism (or developing your own novel arguments). Community engagement includes engaging experts and site users, preparing posts, and identifying dissemination opportunities.
AI welfarePhilosophy of AISocietal impacts
Starter project: Ask a chatbot about its inner life five different ways or Make one piece of AI safety content
Alignment techniques that rely on 'personas' will fail as the AIs become more powerful. This project aims to make progress on the “hard problem of alignment” by providing a framework within which to measure how persona-dependent an alignment technique is, mapping and demonstrating the limits of persona-based alignment and of alignment techniques that make use of personas, and increasing awareness of the limitations of persona-based alignment techniques.
AlignmentPhilosophy of AI
Starter project: Ask a chatbot about its inner life five different ways or Will a chatbot change its mind for a bad reason?
This project will examine whether and how biological and artificial systems can be compared on a common welfare scale. As a CMEP-Rethink Priorities collaboration within the broader Moral Weight Project 2.0, it will assess substrate-general theories of welfare intensity, evaluate compute and related intersubstrate metrics, and map structural disanalogies that complicate comparisons between animal and artificial minds.
AI welfarePhilosophy of AI
Starter project: Ask a chatbot about its inner life five different ways or Compare two AI companies’ safety promises
A tentative hypothesis: evidence of us being nice to models in pretraining data is more likely to make them nice to us back. This project is aimed at getting more evidence on this hypothesis.
AI welfareAlignment
Starter project: Ask a chatbot about its inner life five different ways or Spot the mind-words in AI research posts
Mentees will research questions about AI consciousness and turn their findings into rigorous, original, and accessible essays for an audience of AI safety, governance, and policy researchers.
AI welfarePhilosophy of AI
Starter project: Ask a chatbot about its inner life five different ways or Explain one safety idea to a friend
This project involves ideating and executing experiments on AI metacognition and identity. Mentees can also work on the theoretical foundations of metacognitive intelligence, helping inform experiment design and interpretation.
Philosophy of AIBehavioral evaluation of LLMs
Starter project: Ask a chatbot about its inner life five different ways or Will a chatbot change its mind for a bad reason?
