How it works
DPO simplifies RLHF. Instead of training a reward model and then optimizing with PPO, DPO uses pairs (preferred response, rejected response) and adjusts the model directly to increase the probability of the preferred one.
The loss function compares the relative probabilities of the two responses, penalizing when the model favors the rejected one. The beta parameter controls the strength of the penalty: larger values keep the model closer to the original.
DPO removes the need for a separate reward model and complex RL, making alignment more stable and cheaper.
Practical example
Given the prompt "Explain recursion", with response A (clear, with example) preferred over B (confusing), DPO adjusts the model to generate A more often.
loss = -log(sigmoid(beta * (logp_preferred - logp_rejected)))Preference data can come from human annotators or be generated synthetically by a stronger model evaluating responses from a smaller model.
What it is for
Align models to human preferences in a more stable and cheaper way than RLHF. It is used in open models like Zephyr and in fine tuning assistants.
DPO is especially useful when you have preference data but do not have the resources to train reward models and run PPO.