JuniorTheoryCommon

What does this program print?

Python

import numpy as np

counts = np.array([0, 0])
means = np.array([0.0, 0.0])
for arm, reward in [(0, 1.0), (1, 0.0), (0, 0.0), (0, 1.0)]:
    counts[arm] += 1
    means[arm] += (reward - means[arm]) / counts[arm]
print(round(means[0], 2), counts[0])

0.67 3

Expecting 0.5 3, two rewards over all four steps: each arm keeps its own count, and arm 0 was pulled 3 times, so its mean is 2 / 3, rounded to 0.67.
Expecting 2.0 3, the sum of rewards: the update moves the mean by 1 / count of the gap to the new reward, so means[0] stays the average, not the total.

Why does a bandit keep a running mean per arm instead of storing every reward?
What goes wrong if counts[arm] is increased after the mean update instead of before it?

Back to questions Open topic in roadmap →Answer it yourself — Quiz

Back to questions

JuniorTheoryCommon

What does this program print?

Python

import numpy as np

counts = np.array([0, 0])
means = np.array([0.0, 0.0])
for arm, reward in [(0, 1.0), (1, 0.0), (0, 0.0), (0, 1.0)]:
    counts[arm] += 1
    means[arm] += (reward - means[arm]) / counts[arm]
print(round(means[0], 2), counts[0])

0.67 3

Expecting 0.5 3, two rewards over all four steps: each arm keeps its own count, and arm 0 was pulled 3 times, so its mean is 2 / 3, rounded to 0.67.
Expecting 2.0 3, the sum of rewards: the update moves the mean by 1 / count of the gap to the new reward, so means[0] stays the average, not the total.

Why does a bandit keep a running mean per arm instead of storing every reward?
What goes wrong if counts[arm] is increased after the mean update instead of before it?

Back to questions Open topic in roadmap →Answer it yourself — Quiz