Sinusoidal Positional Encodingmedium

Sinusoidal Positional Encoding

medium Inside an LLM 15 min

NumPy allowed · plain Python earns 10 bonus points

Tokens attend to tokens

Q · Kᵀ / √d → softmax → weighted values, computed in the page.

Open lesson →

01tokens

02Q K V

03scores

04attention

modelslearnrelationships

models32%

learn38%

relationships30%

Q · Kᵀ / √d scores normalized with softmax

Implement the original Transformer's sinusoidal positional encoding formula, the fixed (non-learned) position representation from 'Attention Is All You Need.'

Task

Implement sinusoidal_pe(position, dim_index, d_model). Compute angle = position / (10000 ** (2*(dim_index//2)/d_model)). Return sin(angle) if dim_index is even, else cos(angle).

Function Signature

sinusoidal_pe(position: int, dim_index: int, d_model: int) -> float

Examples

Example 1: position zero even dim
Input: position = 0
dim_index = 0
d_model = 512
Output: 0

Example 2: position zero odd dim
Input: position = 0
dim_index = 1
d_model = 512
Output: 1

Example 3: nonzero position even dim
Input: position = 10
dim_index = 4
d_model = 512
Output: 0.11877648322563235

Constraints

d_model > 0
0 <= dim_index < d_model

Python3Saved ✓

import math

def sinusoidal_pe(position, dim_index, d_model):

# Your implementation here

pass

Case 1: position zero even dim

Input: position = 0
dim_index = 0
d_model = 512

Expected: 0

Case 2: position zero odd dim

Input: position = 0
dim_index = 1
d_model = 512

Expected: 1

Case 3: nonzero position even dim

Input: position = 10
dim_index = 4
d_model = 512

Expected: 0.11877648322563235