Direct Preference Optimization (DPO) Loss

Task

Implement `dpo_loss` calculating $-\log \sigma (\beta \log \frac{\pi_\theta(y_w)}{\pi_{ref}(y_w)} - \beta \log \frac{\pi_\theta(y_l)}{\pi_{ref}(y_l)})$.

Function Signature

dpo_loss(policy_chosen_logps: float, policy_rejected_logps: float, ref_chosen_logps: float, ref_rejected_logps: float, beta: float) -> float

Examples

Constraints

• Must compute log-ratios pi_logratio = policy_chosen - policy_rejected and ref_logratio = ref_chosen - ref_rejected.
• logits = beta * (pi_logratio - ref_logratio).
• Return -log(sigmoid(logits)).