How it works
Constitutional AI, proposed by Anthropic, uses a list of principles (the "constitution") to guide the model. The model generates responses, then critiques and revises them based on the principles, without relying on humans for every example.
The process has two phases: supervised learning with self-criticism and reinforcement learning from preferences generated by the model itself (RLAIF). In the first phase, the model generates responses, critiques them against the constitution and produces revised versions. In the second, it uses those preferences to train via RL.
The constitution can include principles such as: "Be helpful and honest", "Do not cause harm", "Respect privacy", "Avoid stereotypes".
Practical example
Constitution principles can include: "Be helpful and honest", "Do not cause harm", "Respect privacy". The model generates a response, identifies violations and rewrites it.
Resposta inicial: (potencialmente problemática)
Crítica: viola o princípio de não causar danos
Revisão: (resposta corrigida)The reward model in RLAIF is replaced by an AI model that evaluates preferences based on the constitution, reducing human annotation costs.
What it's for
Align models with human values in a scalable way, reducing dependence on massive human annotation. It is the basis for training models like Claude.
Limitations: the constitution can have gaps or biases, and self-criticism depends on the model's ability to evaluate its own responses.