Chain-of-thought reasoning provides new opportunities for alignment and safety. OpenAI found that integrating their policies for model behavior into the chain of thought of a reasoning model is an effective way to robustly teach human values and principles.
By teaching the model safety rules and how to reason about them in context, OpenAI found evidence of reasoning capability directly benefiting model robustness: o1-preview achieved substantially improved performance on key jailbreak evaluations and their hardest internal benchmarks for evaluating model safety refusal boundaries.
OpenAI believes that using a chain of thought offers significant advances for safety and alignment because:
- It enables them to observe the model thinking in a legible way.
- The model reasoning about safety rules is more robust to out-of-distribution scenarios.
To stress-test these improvements, OpenAI conducted a suite of safety tests and red-teaming before deployment, in accordance with their Preparedness Framework. They found that chain-of-thought reasoning contributed to capability improvements across evaluations. Notably, they observed instances of reward hacking. Detailed results from these evaluations can be found in the accompanying System Card.