A freshly trained language model is a strange thing. It has read a huge slice of the internet and can predict text with uncanny fluency, but it has no particular sense of what is helpful, honest, or safe to say. Turning that raw capability into something you would actually want to talk to takes a second phase of training, alignment training, and there are three main techniques you will hear named: RLHF, Constitutional AI, and RLAIF.
The through-line connecting them is a steady shift away from human labour. RLHF leans on people to rate responses. Constitutional AI replaces much of that human judgement with a written set of principles. RLAIF pushes further, using AI itself to generate the feedback. Each step trades some human oversight for scale.
The three methods compared
| RLHF | Constitutional AI | RLAIF | |
|---|---|---|---|
| Feedback source | Human ratings | Principles + AI critique | AI-generated feedback |
| Human effort | High | Medium | Low |
| Scalability | Limited | Better | High |
Why raw models need aligning at all
Pretraining teaches a model to predict likely text, not to be helpful or harmless. Left as-is, it will happily continue a prompt in unhelpful, evasive, or unsafe directions, because nothing has told it not to. Alignment training is the step that shapes behaviour, teaching the model to prefer responses people actually want. It builds directly on the alignment problem: the goal is to make the model pursue our real intentions, not just its literal text-prediction objective.
RLHF: learning from human preferences
Reinforcement learning from human feedback is the technique that made modern chatbots usable, and it is beautifully direct. Humans are shown pairs of model responses and asked which is better. Those preferences train a “reward model” that captures what people like, and the language model is then tuned to maximise that reward. It works remarkably well. But it has a hard ceiling: it needs a lot of human labour. People have to rate enormous numbers of responses, that judgement is slow and expensive, and it can carry the raters’ own inconsistencies. This is a form of reinforcement learning, applied to language.
Constitutional AI: swapping labour for principles
Constitutional AI, developed at Anthropic, asked a sharp question: what if the model could critique itself against a written set of principles instead of relying so heavily on human raters? The “constitution” is a list of guidelines, and the model uses it to evaluate and revise its own responses, generating much of its own training feedback. This dramatically cuts the human labour, makes the values explicit and inspectable, you can read the principles, and scales far better than pure RLHF. Humans still set the constitution, but they no longer have to hand-rate everything.
RLAIF: feedback from the AI itself
RLAIF, reinforcement learning from AI feedback, takes the logic to its conclusion. Instead of human ratings, another AI model generates the preference judgements that train the reward model. If an AI can reliably judge which response is better, you can produce feedback at a scale no human team could match. The obvious tension is oversight: leaning on AI to align AI raises real questions about whether errors or biases get amplified rather than caught. It is powerful and scalable, and it is exactly the kind of trade-off the alignment field is actively wrestling with.
What the research shows
RLHF has a clear paper trail. Christiano and colleagues (2017) showed that an agent could learn complex behaviour from human preferences between pairs of trajectory segments, without a reward function, using feedback on less than one percent of its interactions. OpenAI’s InstructGPT work (Ouyang et al., 2022) applied the approach to language models: labeler demonstrations for supervised fine-tuning, then a dataset of rankings of model outputs for reinforcement learning. In human evaluations, outputs from the 1.3 billion parameter InstructGPT model were preferred to those of the 175 billion parameter GPT-3, despite having about 100 times fewer parameters.
Constitutional AI (Bai et al., 2022, from Anthropic) is the method described above as swapping labour for principles. In the paper, the only human oversight is a list of rules or principles. A supervised phase has the model generate self-critiques and revisions of its own responses, then fine-tunes on the revised answers. A reinforcement learning phase uses a model to judge which of two samples is better, trains a preference model from those AI preferences, and uses it as the reward signal. The authors called that second phase “RL from AI Feedback” (RLAIF), so the two terms are connected: RLAIF is part of how Constitutional AI works. They reported training a harmless but non-evasive assistant that engages with harmful queries by explaining its objections.
A later study by Lee and colleagues (2023) compared RLAIF with RLHF directly on summarization, helpful dialogue and harmless dialogue tasks. They found RLAIF achieved comparable performance to RLHF, and that it could outperform a supervised fine-tuned baseline even when the AI labeler was the same size as the policy. Their direct-RLAIF variant, which takes rewards from an off-the-shelf language model during reinforcement learning without training a reward model, performed better than canonical RLAIF. Those are results on specific tasks, so they do not settle how much human oversight can safely be removed.
A separate line of work questions the reinforcement learning step itself. Rafailov and colleagues (2023) introduced Direct Preference Optimization, which they report solves the standard RLHF problem with a simple classification loss. They describe RLHF as a complex and often unstable procedure, and their method avoids fitting a separate reward model and then running reinforcement learning against it.
Where this is heading
The direction of travel is clear: from expensive human feedback toward scalable, AI-assisted methods, because human rating simply cannot keep up with how fast models are trained. The open question is how much human oversight you can safely remove before you lose the very grounding that made alignment trustworthy in the first place. Most serious systems today blend these approaches rather than pick one. For the learning-theory foundation underneath all of this, our explainer on supervised vs unsupervised vs reinforcement learning is a good companion.
Key takeaways
- RLHF learns from human preference comparisons and made instruction-following chatbots practical.
- Constitutional AI uses a written list of principles and AI-generated critiques, and RLAIF is its reinforcement learning phase.
- On the tasks tested so far, RLAIF performed comparably to RLHF, but how much human oversight can be removed safely remains open.
Sources
- Christiano et al., “Deep reinforcement learning from human preferences”, 2017.
- Ouyang et al., “Training language models to follow instructions with human feedback”, 2022.
- Bai et al., “Constitutional AI: Harmlessness from AI Feedback”, 2022.
- Lee et al., “RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback”, 2023.
- Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”, 2023.
How we checked this: we compared the figures and findings above with each paper’s published abstract and record. Results are specific to the tasks and datasets those papers used.
