Why do logistic-regression coefficients blow up on separable data, and what stops it?
A teammate reports that a fitted classifier prints coefficients of order 1e4 and a convergence warning, while its training accuracy is a perfect 1.0. The two classes are linearly separable.
Constraints: keep logistic regression, keep every feature, and do not edit the data.
import numpy as np
from sklearn.linear_model import LogisticRegression
X = np.array([[-2.0], [-1.0], [1.0], [2.0]])
y = np.array([0, 0, 1, 1])
model = LogisticRegression(penalty=None, max_iter=100000)
model.fit(X, y)
print(model.coef_) # order 1e4, and it grows with max_iterFind and fix the error.
Under perfect separation the maximum-likelihood optimum does not exist — scaling the weights up pushes log-loss toward zero without reaching it, so they diverge and the solver stops only on its iteration cap. A penalty pins a finite solution.
- ✗Raising max_iter and treating the vanished warning as a fix
- ✗Blaming float overflow or feature scale instead of a non-existent optimum
- ✗Assuming perfect training accuracy means the fit is healthy
- →Why does raising the iteration cap make the printed coefficients larger, not stabler?
- →Besides a penalty, which other remedy bounds the estimate under separation?
The classes are linearly separable, so without a penalty the likelihood has no optimum: scaling the weights up always makes the predictions more confident and lowers log-loss, and the limit is only reached at infinity. Raising max_iter is pointless — the coefficients simply keep growing. Restore the penalty (which is scikit-learn's default) so the objective is coercive and the minimum is finite.
import numpy as np
from sklearn.linear_model import LogisticRegression
X = np.array([[-2.0], [-1.0], [1.0], [2.0]])
y = np.array([0, 0, 1, 1])
model = LogisticRegression(penalty="l2", C=1.0, max_iter=1000)
model.fit(X, y)
print(model.coef_) # finite coefficient, stable as max_iter growsA smaller C penalizes harder and shrinks the estimate further. A penalty-free alternative is Firth's penalized likelihood, which also yields finite estimates under separation.
