Simplifying Alignment: From RLHF to Direct Preference Optimization (DPO)

Large language models (LLMs) are getting smarter every day, but teaching them to do what we want—aligning them with human preferences—is still a tough nut to crack. As deep learning folks, we know that if you want a model to learn something, you give it data, right? So why not gather some examples of what we like and have the model learn those preferences?

That’s where Reinforcement Learning with Human Feedback (RLHF) comes in. It’s a clever way to teach LLMs to follow human preferences by using feedback data. But RLHF can be a bit of a headache—it brings reinforcement learning into the mix, and optimization gets tricky fast.

Enter Direct Preference Optimization (DPO). DPO skips the RL part while still teaching models to follow preferences. It’s simpler, cleaner, and honestly, who doesn’t love simplicity?

In this blog, we’ll go on a journey from RLHF to DPO, break down the math (don’t worry, we’ll keep it chill), and see why DPO might just be the smarter, easier way forward.

Reinforcement Learning with Human Feedback (RLHF)

RLHF is a framework for aligning language models with human preferences through a structured, three-phase process. Each phase builds on the previous one, refining the model to better understand and produce responses that align with human expectations. Let’s break it down:

1. Supervised Fine-Tuning (SFT)

We start by taking a pre-trained language model and fine-tuning it on high-quality, task-specific data. This process generates a base policy ${\pi }_{\text{SFT}}(y\mid x)\backslash pi\_\{\backslash text\{SFT\}\}(y\; \backslash mid\; x)$, which represents the model's probability of generating an output $yy$ given an input $xx$.

This base policy serves as a strong starting point, capturing general task-related behavior but still needing refinement to align with human preferences.

2. Preference Sampling & Reward Learning

This phase focuses on collecting data about human preferences and building a reward model to represent those preferences numerically.

Preference Sampling

Here’s how it works:

1. The supervised fine-tuned model generates pairs of responses $(y_1,y_2)(y\_1,\; y\_2)$ for a given input or prompt $xx$.
2. Human annotators compare these responses and select their preferred response, $y_{w}y\_w$ (the "winner"), over the less preferred response, $y_{l}y\_l$ (the "loser").

These human preferences are then used as training data for the next step.

Reward Modeling

We want to create a reward model $r_{\varphi }(x,y)r\_\backslash phi(x,\; y)$, which assigns a numerical score (reward) to each response $yy$ given a prompt $xx$. This score reflects how well the response aligns with human preferences.

Modeling Pairwise Preferences

To train this reward model, we rely on pairwise comparisons of responses $y_{w}y\_w$ (winner) and $y_{l}y\_l$ (loser). The preferences are modeled using the Bradley-Terry framework, which assigns a probability to the preference:

1. The probability of $y_{w}y\_w$ being preferred over $y_{l}y\_l$ is: $$p_{\varphi }(y_w>y_l\mid x)=\frac{\exp r_{\varphi }(x,y_w)}{\exp r_{\varphi }(x,y_w)+\exp r_{\varphi }(x,y_l)}\mathrm{.}p\_\backslash phi(y\_w\; >\; y\_l\; \backslash mid\; x)\; =\; \backslash frac\{\backslash exp\; r\_\backslash phi(x,\; y\_w)\}\{\backslash exp\; r\_\backslash phi(x,\; y\_w)\; +\; \backslash exp\; r\_\backslash phi(x,\; y\_l)\}.$$

   • $r_{\varphi }(x,y_w)r\_\backslash phi(x,\; y\_w)$ and $r_{\varphi }(x,y_l)r\_\backslash phi(x,\; y\_l)$: Rewards (scores) assigned to the winner and loser, respectively.
   • The numerator $\exp r_{\varphi }(x,y_w)\backslash exp\; r\_\backslash phi(x,\; y\_w)$: Represents the likelihood of the winner being the preferred choice.
   • The denominator ensures probabilities sum to 1 by including both options: $\exp r_{\varphi }(x,y_w)+\exp r_{\varphi }(x,y_l)\backslash exp\; r\_\backslash phi(x,\; y\_w)\; +\; \backslash exp\; r\_\backslash phi(x,\; y\_l)$.

2. Rearranging, we write the probability in terms of a difference between rewards: $$p_{\varphi }(y_w>y_l\mid x)=\frac{1}{1+\exp [r_{\varphi }(x,y_l)-r_{\varphi }(x,y_w)]}\mathrm{.}p\_\backslash phi(y\_w\; >\; y\_l\; \backslash mid\; x)\; =\; \backslash frac\{1\}\{1\; +\; \backslash exp\; \backslash left[\; r\_\backslash phi(x,\; y\_l)\; -\; r\_\backslash phi(x,\; y\_w)\; \backslash right]\}.$$

   • $r_{\varphi }(x,y_l)-r_{\varphi }(x,y_w)r\_\backslash phi(x,\; y\_l)\; -\; r\_\backslash phi(x,\; y\_w)$: The difference between the scores of the loser and the winner.
   • $\exp (\cdot )\backslash exp(\backslash cdot)$: Converts this difference into a scaling factor for the probability.

3. Using the sigmoid function $\sigma (z)=\frac{1}{1+e^{-z}}\backslash sigma(z)\; =\; \backslash frac\{1\}\{1\; +\; e^\{-z\}\}$, the equation becomes: $$p_{\varphi }(y_w>y_l\mid x)=\sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))\mathrm{.}p\_\backslash phi(y\_w\; >\; y\_l\; \backslash mid\; x)\; =\; \backslash sigma(r\_\backslash phi(x,\; y\_w)\; -\; r\_\backslash phi(x,\; y\_l)).$$

   • $r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)r\_\backslash phi(x,\; y\_w)\; -\; r\_\backslash phi(x,\; y\_l)$: If the winner's score is much higher than the loser's, the probability approaches 1 (the winner is strongly preferred).

Training the Reward Model

To train $r_{\varphi }r\_\backslash phi$, we optimize it to match human preferences as closely as possible. This is done using maximum likelihood estimation (MLE):

• The loss function for the reward model is: $${\mathcal{L}}_R(r_{\varphi },\mathcal{D})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}[\log \sigma (r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l))]\mathrm{.}\backslash mathcal\{L\}\_R(r\_\backslash phi,\; \backslash mathcal\{D\})\; =\; -\backslash mathbb\{E\}\_\{(x,\; y\_w,\; y\_l)\; \backslash sim\; \backslash mathcal\{D\}\}\; \backslash big[\; \backslash log\; \backslash sigma(r\_\backslash phi(x,\; y\_w)\; -\; r\_\backslash phi(x,\; y\_l))\; \backslash big].$$

  • $\mathcal{D}\backslash mathcal\{D\}$: Dataset of human preferences (pairs $y_w,y_{l}y\_w,\; y\_l$ for each prompt $xx$).
  • $\log \sigma (\cdot )\backslash log\; \backslash sigma(\backslash cdot)$: Penalizes predictions that assign low probabilities to actual human preferences.
  • The goal is to minimize the negative log-likelihood, ensuring the reward model aligns its predictions with the collected human feedback.

3. Reinforcement Learning (RL) Optimization

The final step involves fine-tuning the policy ${\pi }_{\varphi }(y\mid x)\backslash pi\_\backslash phi(y\; \backslash mid\; x)$ using reinforcement learning to maximize the reward. However, directly maximizing the reward can lead to excessive deviations from the base policy ${\pi }_{\text{SFT}}\backslash pi\_\{\backslash text\{SFT\}\}$, causing unnatural or overly optimized behavior. To address this, we add a penalty to constrain the policy:

The RL Objective

The optimization objective is: $$\underset{{\pi }_{\varphi }}{\max }{\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\varphi }(y\mid x)}[r_{\varphi }(x,y)]-\beta {\text{D}}_{\text{KL}}[{\pi }_{\varphi }(y\mid x)\mathrm{\parallel }{\pi }_{\text{ref}}(y\mid x)]\mathrm{.}\backslash max\_\{\backslash pi\_\backslash phi\}\; \backslash \; \backslash mathbb\{E\}\_\{x\; \backslash sim\; \backslash mathcal\{D\},\; y\; \backslash sim\; \backslash pi\_\backslash phi(y\; \backslash mid\; x)\}\; \backslash big[\; r\_\backslash phi(x,\; y)\; \backslash big]\; -\; \backslash beta\; \backslash text\{D\}\_\{\backslash text\{KL\}\}\; \backslash big[\; \backslash pi\_\backslash phi(y\; \backslash mid\; x)\; \backslash Vert\; \backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)\; \backslash big].$$

• First Term: $\mathbb{E}[r_{\varphi }(x,y)]\backslash mathbb\{E\}[r\_\backslash phi(x,\; y)]$

  • Encourages the policy to generate responses with higher rewards.

• Second Term: ${\text{D}}_{\text{KL}}\backslash text\{D\}\_\{\backslash text\{KL\}\}$

  • ${\text{D}}_{\text{KL}}(P\mathrm{\parallel }Q)\backslash text\{D\}\_\{\backslash text\{KL\}\}(P\; \backslash Vert\; Q)$: The Kullback-Leibler (KL) divergence, a measure of how much $PP$ differs from $QQ$.
  • Penalizes the policy ${\pi }_{\varphi }\backslash pi\_\backslash phi$ for straying too far from the reference policy ${\pi }_{\text{ref}}\backslash pi\_\{\backslash text\{ref\}\}$ (usually ${\pi }_{\text{SFT}}\backslash pi\_\{\backslash text\{SFT\}\}$).

• $\beta \backslash beta$: A weighting factor controlling the balance between maximizing rewards and staying close to the reference policy.

This process completes the RLHF pipeline, where reinforcement learning ensures the model generates responses that maximize alignment with human preferences while maintaining a natural and task-relevant behavior.

Challenges of RLHF

1. Non-Differentiability of Language Outputs:
   Language generation involves sampling discrete tokens, which breaks the flow of gradients during optimization. This makes it tricky to directly use gradient-based methods (the backbone of deep learning) to adjust the model.

2. Reward Model Struggles to Generalize:
   The reward model learns to predict human preferences, but it’s hard to capture the subtlety and variability of what humans truly prefer. If the reward model fails to generalize, it can lead to misaligned or biased optimization.

3. Computational and Implementation Overhead:
   RL adds significant complexity to the pipeline. From designing the reward function to tuning hyperparameters like the KL penalty, it requires specialized expertise and significantly more compute power compared to simpler fine-tuning methods.

From RLHF to Direct Preference Optimization (DPO)

In this section, we’ll reframe the RLHF objective and introduce a key variable transformation. This reformulation will pave the way to understanding how Direct Preference Optimization (DPO) works and why it’s a simpler, more efficient alternative to RLHF.

Reformulating the RLHF Objective

The RLHF objective begins as:

$$\underset{\pi }{\max }{\mathbb{E}}_{x\sim \mathcal{D},y\sim \pi (y\mid x)}[r(x,y)]-\beta {\text{D}}_{\text{KL}}[\pi (y\mid x)\mathrm{\parallel }{\pi }_{\text{ref}}(y\mid x)]\mathrm{.}\backslash max\_\{\backslash pi\}\; \backslash \; \backslash mathbb\{E\}\_\{x\; \backslash sim\; \backslash mathcal\{D\},\; y\; \backslash sim\; \backslash pi(y\; \backslash mid\; x)\}\; \backslash big[\; r(x,\; y)\; \backslash big]\; -\; \backslash beta\; \backslash text\{D\}\_\{\backslash text\{KL\}\}\; \backslash big[\; \backslash pi(y\; \backslash mid\; x)\; \backslash Vert\; \backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)\; \backslash big].$$

This objective balances two goals:

1. Maximizing reward: Encourage the model $\pi (y\mid x)\backslash pi(y\; \backslash mid\; x)$ to generate outputs $yy$ that align with human preferences, as captured by the reward $r(x,y)r(x,\; y)$.
2. Constraining deviation: Prevent the model from diverging too far from a reference policy ${\pi }_{\text{ref}}(y\mid x)\backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)$ (usually the supervised fine-tuned model), which ensures stability and avoids overly aggressive changes.

Expanding the KL Divergence Term

The KL divergence measures the "distance" between the current policy $\pi (y\mid x)\backslash pi(y\; \backslash mid\; x)$ and the reference policy ${\pi }_{\text{ref}}(y\mid x)\backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)$. Expanding it gives:

$${\text{D}}_{\text{KL}}[\pi (y\mid x)\mathrm{\parallel }{\pi }_{\text{ref}}(y\mid x)]={\mathbb{E}}_{y\sim \pi (y\mid x)}[\log \pi (y\mid x)-\log {\pi }_{\text{ref}}(y\mid x)]\mathrm{.}\backslash text\{D\}\_\{\backslash text\{KL\}\}\backslash big[\backslash pi(y\; \backslash mid\; x)\; \backslash Vert\; \backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)\backslash big]\; =\; \backslash mathbb\{E\}\_\{y\; \backslash sim\; \backslash pi(y\; \backslash mid\; x)\}\; \backslash big[\; \backslash log\; \backslash pi(y\; \backslash mid\; x)\; -\; \backslash log\; \backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)\; \backslash big].$$

This term penalizes $\pi \backslash pi$ for deviating from ${\pi }_{\text{ref}}\backslash pi\_\{\backslash text\{ref\}\}$. Substituting it back into the original objective gives:

$$\underset{\pi }{\max }{\mathbb{E}}_{x\sim \mathcal{D},y\sim \pi (y\mid x)}[r(x,y)-\beta \log \pi (y\mid x)+\beta \log {\pi }_{\text{ref}}(y\mid x)]\mathrm{.}\backslash max\_\{\backslash pi\}\; \backslash \; \backslash mathbb\{E\}\_\{x\; \backslash sim\; \backslash mathcal\{D\},\; y\; \backslash sim\; \backslash pi(y\; \backslash mid\; x)\}\; \backslash big[\; r(x,\; y)\; -\; \backslash beta\; \backslash log\; \backslash pi(y\; \backslash mid\; x)\; +\; \backslash beta\; \backslash log\; \backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)\; \backslash big].$$

Switching to a Minimization Form

For simplicity, we rewrite the objective as a minimization problem (minimizing the negative of the maximization objective):

$$\underset{\pi }{\min }{\mathbb{E}}_{x\sim \mathcal{D},y\sim \pi (y\mid x)}[\log \pi (y\mid x)-\log {\pi }_{\text{ref}}(y\mid x)-\frac{r(x,y)}{\beta }]\mathrm{.}\backslash min\_\{\backslash pi\}\; \backslash \; \backslash mathbb\{E\}\_\{x\; \backslash sim\; \backslash mathcal\{D\},\; y\; \backslash sim\; \backslash pi(y\; \backslash mid\; x)\}\; \backslash big[\; \backslash log\; \backslash pi(y\; \backslash mid\; x)\; -\; \backslash log\; \backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)\; -\; \backslash frac\{r(x,\; y)\}\{\backslash beta\}\; \backslash big].$$

This highlights the trade-offs:

• The \( \log \pi(y \mid x) \) term encourages the policy to focus on likely outputs.
• The \( -\log \pi_{\text{ref}}(y \mid x) \) term ensures outputs remain close to the reference model.
• The \( -r(x, y) / \beta \) term biases the policy toward high-reward responses.

Introducing the Partition Function

Let's introduce a function $Z(x)Z(x)$

$$Z(x)=\sum _y{\pi }_{\text{ref}}(y\mid x)\exp \left[\frac{r(x,y)}{\beta }\right]\mathrm{.}Z(x)\; =\; \backslash sum\_y\; \backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)\; \backslash exp\backslash big[\; \backslash frac\{r(x,\; y)\}\{\backslash beta\}\; \backslash big].$$

Using $Z(x)Z(x)$, we can express $\pi (y\mid x)\backslash pi(y\; \backslash mid\; x)$ as:

$$\pi (y\mid x)=\frac{1}{Z(x)}{\pi }_{\text{ref}}(y\mid x)\exp \left[\frac{r(x,y)}{\beta }\right]\mathrm{.}\backslash pi(y\; \backslash mid\; x)\; =\; \backslash frac\{1\}\{Z(x)\}\; \backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)\; \backslash exp\backslash big[\; \backslash frac\{r(x,\; y)\}\{\backslash beta\}\; \backslash big].$$

Here’s the intuition:

• ${\pi }_{\text{ref}}(y\mid x)\backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)$: Acts as a baseline distribution (our starting point).
• $\exp \left[\frac{r(x,y)}{\beta }\right]\backslash exp\backslash big[\backslash frac\{r(x,\; y)\}\{\backslash beta\}\backslash big]$: Scales outputs $yy$ based on their reward, making high-reward outputs more likely.
• $Z(x)Z(x)$: Normalizes the distribution so probabilities sum to 1.

This formulation enables us to reweight the reference policy by incorporating preferences encoded in the reward model, without requiring direct reinforcement learning.

Derivation of DPO Loss

The key trick of Direct Preference Optimization (DPO) is focusing on pairwise preferences, which simplifies optimization. Let’s break it down:

Pairwise Preferences

For two completions $y_{1}y\_1$ (winner) and $y_{2}y\_2$ (loser), we care about the probability that humans prefer $y_{1}y\_1$ over $y_{2}y\_2$. Using the Bradley-Terry model, this probability is:

$$p(y_1>y_2\mid x)=\sigma (\beta \log \frac{\pi (y_1\mid x)}{{\pi }_{\text{ref}}(y_1\mid x)}-\beta \log \frac{\pi (y_2\mid x)}{{\pi }_{\text{ref}}(y_2\mid x)}),p(y\_1\; >\; y\_2\; \backslash mid\; x)\; =\; \backslash sigma\backslash big(\backslash beta\; \backslash log\; \backslash frac\{\backslash pi(y\_1\; \backslash mid\; x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_1\; \backslash mid\; x)\}\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi(y\_2\; \backslash mid\; x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_2\; \backslash mid\; x)\}\backslash big),$$

where $\sigma (z)=\frac{1}{1+e^{-z}}\backslash sigma(z)\; =\; \backslash frac\{1\}\{1\; +\; e^\{-z\}\}$ is the sigmoid function.

Simplifying $\pi (y\mid x)\backslash pi(y\; \backslash mid\; x)$

Substituting $\pi (y\mid x)\backslash pi(y\; \backslash mid\; x)$ from the earlier reformulation:

$$\pi (y\mid x)=\frac{1}{Z(x)}{\pi }_{\text{ref}}(y\mid x)\exp \left[\frac{r(x,y)}{\beta }\right]\mathrm{.}\backslash pi(y\; \backslash mid\; x)\; =\; \backslash frac\{1\}\{Z(x)\}\; \backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)\; \backslash exp\backslash big[\; \backslash frac\{r(x,\; y)\}\{\backslash beta\}\; \backslash big].$$

When comparing two outputs $y_{1}y\_1$ and $y_{2}y\_2$, the partition function $Z(x)Z(x)$ cancels out (since it’s the same for both), leaving:

$$p(y_1>y_2\mid x)=\sigma (r(x,y_1)-r(x,y_2))\mathrm{.}p(y\_1\; >\; y\_2\; \backslash mid\; x)\; =\; \backslash sigma\backslash big(r(x,\; y\_1)\; -\; r(x,\; y\_2)\backslash big).$$

This simplifies the computation, as we no longer need to compute $Z(x)Z(x)$ explicitly. The sigmoid function ensures that higher rewards correspond to higher probabilities.

DPO Loss

To train ${\pi }_{\theta }\backslash pi\_\backslash theta$ (the parameterized policy), we use maximum likelihood estimation (MLE) over human preferences. The DPO loss becomes:

$${\mathcal{L}}_{\text{DPO}}({\pi }_{\theta },{\pi }_{\text{ref}})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}[\log \sigma (\beta \log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)}\left)\right]\mathrm{.}\backslash mathcal\{L\}\_\{\backslash text\{DPO\}\}(\backslash pi\_\backslash theta,\; \backslash pi\_\{\backslash text\{ref\}\})\; =\; -\backslash mathbb\{E\}\_\{(x,\; y\_w,\; y\_l)\; \backslash sim\; \backslash mathcal\{D\}\}\; \backslash big[\; \backslash log\; \backslash sigma\backslash big(\backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y\_w\; \backslash mid\; x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_w\; \backslash mid\; x)\}\; -\; \backslash beta\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y\_l\; \backslash mid\; x)\}\{\backslash pi\_\{\backslash text\{ref\}\}(y\_l\; \backslash mid\; x)\}\backslash big)\; \backslash big].$$

Why Does This Trick Work?

1. No Reinforcement Learning Required:
   By reframing the problem in terms of pairwise preferences and reweighting the reference policy, DPO eliminates the need for complex reinforcement learning.

2. Simpler Optimization:
   The partition function $Z(x)Z(x)$ cancels out in pairwise comparisons, reducing computational overhead. Training focuses directly on aligning with human preferences.

3. Improved Stability:
   The KL constraint from ${\pi }_{\text{ref}}(y\mid x)\backslash pi\_\{\backslash text\{ref\}\}(y\; \backslash mid\; x)$ ensures that ${\pi }_{\theta }\backslash pi\_\backslash theta$ remains grounded, avoiding extreme behavior changes often seen in RL.

4. Focus on Human Preferences:
   By directly optimizing for pairwise preference probabilities, DPO centers the learning process around human-labeled data, aligning outputs more naturally with human expectations.

Conclusion

Direct Preference Optimization simplifies the alignment of large language models by replacing the RL phase of RLHF with a direct optimization framework. By working with pairwise preferences and avoiding reinforcement learning, DPO achieves alignment with reduced computational and implementation overhead, making it a compelling alternative for aligning large-scale models.

As alignment techniques evolve, DPO exemplifies how simplifying assumptions can lead to practical and effective solutions for real-world challenges in AI.