Neural Mastery
← Practice
Practice: Rotary Positional Embedding (RoPE) From Scratchmedium

Practice: Rotary Positional Embedding (RoPE) From Scratch

medium Inside an LLM 15–20 min

NumPy allowed · plain Python earns 10 bonus points

New to this topic? LLMs & GenAI Overview
Study this concept first →
RAG step 1/4

A query retrieves context

Embedding similarity ranks the documents before generation.
Open lesson →
01query
02embed
03retrieve
04context
What is RAG?→ embed →top context
retrieval.md44%
rag-systems.md33%
agents.md24%

Cosine-like similarity scores ranked before context assembly

Implement Rotary Position Embeddings (RoPE) to inject relative position information into attention query and key vectors.

Task

Implement rope(x, position, base=10000.0). Rotate consecutive 2D pairs (x[2i], x[2i+1]) by angle theta_i = position * base^(-2i / d).

Function Signature

rope(x: list[float], position: int, base: float = 10000.0) -> list[float]

Examples

Example 1: Position 0 leaves vector unchanged
Input: x = [1, 2, 3, 4] position = 0
Output: [1, 2, 3, 4]
Example 2: Position 1 with standard rotation
Input: x = [1, 0, 0, 1] position = 1
Output: [0.540302, 0.841471, -0.01, 0.99995]

Constraints

  • len(x) is an even integer d >= 2
  • position >= 0
  • Return list of floats rounded to 6 decimal places
Python3Saved ✓
import numpy as np

def rope(x: list[float], position: int, base: float = 10000.0) -> list[float]:
"""Apply Rotary Positional Embedding (RoPE) to 1D vector x at position m.
Return transformed vector as list of floats.
"""
# Your code here
pass

Case 1: Position 0 leaves vector unchanged
Input: x = [1, 2, 3, 4] position = 0
Expected: [1, 2, 3, 4]
Case 2: Position 1 with standard rotation
Input: x = [1, 0, 0, 1] position = 1
Expected: [0.540302, 0.841471, -0.01, 0.99995]