Discounted Return From Scratch

Difficulty: Easy · Concept: RL Fundamentals — Markov Decision Processes (MDPs)

An RL agent doesn't just care about the very next reward — it cares about the total reward it can expect going forward, the return. Future rewards are discounted by a factor $\gamma \in [0, 1]$ per timestep, so a reward $k$ steps away counts for $\gamma^k$ of its face value:

$G_t = \sum_{k=0}^{T} \gamma^k r_{t+k}$

$\gamma$ close to 1 means the agent is nearly as concerned with far-future rewards as immediate ones ("far-sighted"); $\gamma$ close to 0 means it barely looks past the next step ("short-sighted"). This return, not any single reward, is what value functions (Policies and Value Functions) are actually trying to predict.

Your task: implement discounted_return(rewards, gamma), where rewards is a list [r_0, r_1, ..., r_T] in time order.

Implement it yourself

assert abs(discounted_return([1, 1, 1], 1.0) - 3.0) < 1e-9
assert abs(discounted_return([1, 0, 0], 0.5) - 1.0) < 1e-9
assert abs(discounted_return([1, 1, 1, 1], 0.5) - 1.875) < 1e-9
assert abs(discounted_return([], 0.9) - 0.0) < 1e-9
assert abs(discounted_return([2, 4], 0.9) - 5.6) < 1e-9

Next: Full Return-to-Go Sequence (a harder variant)