A freshly trained language model is a strange thing. It has read a huge slice of the internet and can predict text with uncanny fluency, but it has no particular sense of what is helpful, honest, or safe to say. Turning that raw capability into something you would actually want to talk to takes a second phase of training — alignment training — and there are three main techniques you will hear named: RLHF, Constitutional AI, and RLAIF.
The through-line connecting them is a steady shift away from human labour. RLHF leans on people to rate responses. Constitutional AI replaces much of that human judgement with a written set of principles. RLAIF pushes further, using AI itself to generate the feedback. Each step trades some human oversight for scale.
The three methods compared
| RLHF | Constitutional AI | RLAIF | |
|---|---|---|---|
| Feedback source | Human ratings | Principles + AI critique | AI-generated feedback |
| Human effort | High | Medium | Low |
| Scalability | Limited | Better | High |
Why raw models need aligning at all
Pretraining teaches a model to predict likely text, not to be helpful or harmless. Left as-is, it will happily continue a prompt in unhelpful, evasive, or unsafe directions, because nothing has told it not to. Alignment training is the step that shapes behaviour — teaching the model to prefer responses people actually want. It builds directly on the alignment problem: the goal is to make the model pursue our real intentions, not just its literal text-prediction objective.
RLHF: learning from human preferences
Reinforcement learning from human feedback is the technique that made modern chatbots usable, and it is beautifully direct. Humans are shown pairs of model responses and asked which is better. Those preferences train a “reward model” that captures what people like, and the language model is then tuned to maximise that reward. It works remarkably well — but it has a hard ceiling: it needs a lot of human labour. People have to rate enormous numbers of responses, that judgement is slow and expensive, and it can carry the raters’ own inconsistencies. This is a form of reinforcement learning, applied to language.
Constitutional AI: swapping labour for principles
Constitutional AI, developed at Anthropic, asked a sharp question: what if the model could critique itself against a written set of principles instead of relying so heavily on human raters? The “constitution” is a list of guidelines, and the model uses it to evaluate and revise its own responses, generating much of its own training feedback. This dramatically cuts the human labour, makes the values explicit and inspectable — you can read the principles — and scales far better than pure RLHF. Humans still set the constitution, but they no longer have to hand-rate everything.
RLAIF: feedback from the AI itself
RLAIF — reinforcement learning from AI feedback — takes the logic to its conclusion. Instead of human ratings, another AI model generates the preference judgements that train the reward model. If an AI can reliably judge which response is better, you can produce feedback at a scale no human team could match. The obvious tension is oversight: leaning on AI to align AI raises real questions about whether errors or biases get amplified rather than caught. It is powerful and scalable, and it is exactly the kind of trade-off the alignment field is actively wrestling with.
Where this is heading
The direction of travel is clear: from expensive human feedback toward scalable, AI-assisted methods, because human rating simply cannot keep up with how fast models are trained. The open question is how much human oversight you can safely remove before you lose the very grounding that made alignment trustworthy in the first place. Most serious systems today blend these approaches rather than pick one. For the learning-theory foundation underneath all of this, our explainer on supervised vs unsupervised vs reinforcement learning is a good companion.
