JuniorTheoryCommon

What does this program print?

Python

import numpy as np

h = np.array([[2.0, 0.0], [1.0, 2.0]])
p = np.array([0.75, 0.25])
y = np.array([1, 0])
dz = p - y
dw = (h * dz.reshape(-1, 1)).mean(axis=0)
print(dw[0], dw[1])

-0.125 0.25

Expecting 0.125 -0.25: the output gradient is p - y, not y - p, so the first sample, with p below its label 1, gets a negative dz.
Expecting -0.25 0.5: .mean(axis=0) averages the two samples' products for each weight instead of adding them up.

Why is the derivative of log-loss with respect to the sigmoid's input exactly p - y?
How does this gradient reach a hidden neuron that sits behind a ReLU?

Back to questions Open topic in roadmap →Answer it yourself — Quiz

Back to questions

JuniorTheoryCommon

What does this program print?

Python

import numpy as np

h = np.array([[2.0, 0.0], [1.0, 2.0]])
p = np.array([0.75, 0.25])
y = np.array([1, 0])
dz = p - y
dw = (h * dz.reshape(-1, 1)).mean(axis=0)
print(dw[0], dw[1])

-0.125 0.25

Expecting 0.125 -0.25: the output gradient is p - y, not y - p, so the first sample, with p below its label 1, gets a negative dz.
Expecting -0.25 0.5: .mean(axis=0) averages the two samples' products for each weight instead of adding them up.

Why is the derivative of log-loss with respect to the sigmoid's input exactly p - y?
How does this gradient reach a hidden neuron that sits behind a ReLU?

Back to questions Open topic in roadmap →Answer it yourself — Quiz