Merge a LoRA Adapter Into the Base Weightmedium

Merge a LoRA Adapter Into the Base Weight

medium Inside an LLM 15 min

NumPy allowed · plain Python earns 10 bonus points

Tokens attend to tokens

Q · Kᵀ / √d → softmax → weighted values, computed in the page.

Open lesson →

01tokens

02Q K V

03scores

04attention

modelslearnrelationships

models32%

learn38%

relationships30%

Q · Kᵀ / √d scores normalized with softmax

Implement LoRA adapter merging, the real formula for folding a trained low-rank adapter back into the base model weight for zero-overhead inference.

Task

Implement merge_lora(base_weight, lora_a, lora_b, alpha, rank) (simplified to scalars representing one weight element's low-rank update): return base_weight + (alpha/rank) * lora_a * lora_b.

Function Signature

merge_lora(base_weight: float, lora_a: float, lora_b: float, alpha: float, rank: int) -> float

Examples

Example 1: small adapter update
Input: base_weight = 1
lora_a = 0.1
lora_b = 0.2
alpha = 16
rank = 8
Output: 1.04

Example 2: zero lora effectively no-op
Input: base_weight = 1
lora_a = 0
lora_b = 0.5
alpha = 16
rank = 8
Output: 1

Example 3: higher rank same alpha smaller effective scale
Input: base_weight = 1
lora_a = 1
lora_b = 1
alpha = 16
rank = 16
Output: 2

Constraints

rank > 0

Python3Saved ✓

def merge_lora(base_weight, lora_a, lora_b, alpha, rank):

# Your implementation here

pass

Case 1: small adapter update

Input: base_weight = 1
lora_a = 0.1
lora_b = 0.2
alpha = 16
rank = 8

Expected: 1.04

Case 2: zero lora effectively no-op

Input: base_weight = 1
lora_a = 0
lora_b = 0.5
alpha = 16
rank = 8

Expected: 1

Case 3: higher rank same alpha smaller effective scale

Input: base_weight = 1
lora_a = 1
lora_b = 1
alpha = 16
rank = 16

Expected: 2