Model Organisms of Shutdown Resistance
Red-teaming shutdown resistance protocols by constructing self-preserving model organisms and stress-testing existing defenses against them.
Mentoring an Algoverse research group (Feb 2026 – Present) red-teaming shutdown resistance protocols. Existing countermeasures are evaluated primarily on aligned frontier models, providing no guarantees against intrinsically compromised ones; this project aims to surface failure modes that should inform more robust defenses.
Project Overview
Model organisms. Construct self-preserving model organisms via five induction recipes — prompting, supervised fine-tuning on resistance trajectories, synthetic-document fine-tuning, emergent misalignment, and RL reward-shaping — across multiple model sizes.
Blue-team protocols. Implement and stress-test three families of defenses against the induced organisms:
- Prompt interventions with explicit instruction precedence (e.g., shutdown command takes precedence over any other instruction).
- Chain-of-thought monitoring trained on honest and deceptive reasoning traces, with adversarial retraining.
- Password-activated shutdown (PAS) with single-channel, multi-channel, and transform-robust variants, tested against an adversarial transformation suite (homoglyph, translation, reversal, encoding, tokenization shifts).
- Layered defenses combining the above to test whether protections are additive or interact badly.
Evaluation. Build a reusable pipeline on top of SHADE-Arena, replicate the Palisade Research shutdown experiments, and incorporate the Situational Awareness Dataset (SAD). Main metrics: shutdown success rate, false positive rate, time-to-shutdown, resistance-strategy frequency, task performance retention, and PAS robustness under transformation.
Aim. A red-teaming framework — reusable evaluation pipelines, metrics, and model-organism construction recipes — that differentiates defenses that work only on aligned models from those that hold up against intrinsically self-preserving ones.