Logistic Regression

Modelling class probabilities with the sigmoid, odds and log-odds, interpreting coefficients as odds ratios, the linear decision boundary, maximum likelihood and the cross-entropy (log) loss, its gradient and gradient-descent training, convexity, separable data and regularisation, multinomial (softmax) regression, and threshold choice.

From linear scores to probabilities

For binary classification, model the probability of class 1 by squashing a linear score through the sigmoid (logistic) function:

P(y = 1 \mid \mathbf x) = \sigma(\mathbf w^\top\mathbf x + b) = \frac{1}{1 + e^{-(\mathbf w^\top\mathbf x + b)}}.

σ maps any real score to (0, 1); σ(0) = 0.5, σ(−z) = 1 − σ(z).

Properties: $\sigma(0) = \frac12$ ; $\sigma(-z) = 1 - \sigma(z)$ ; $\sigma'(z) = \sigma(z)(1 - \sigma(z))$ ; the inverse is the logit $\ln\frac{p}{1-p}$ .

Odds and log-odds

\text{odds} = \frac{p}{1 - p}, \qquad \log\frac{p}{1 - p} = \mathbf w^\top\mathbf x + b.

Logistic regression is linear in the log-odds. Each coefficient is an additive change in log-odds; $e^{w_j}$ is the odds ratio for a one-unit increase in $x_j$ (holding the others fixed).

Decision rule

Predict class 1 if $p \ge t$ . With $t = 0.5$ this is $\mathbf w^\top\mathbf x + b \ge 0$ — a linear decision boundary. Changing $t$ moves the boundary parallel to itself, trading precision against recall (see evaluation).

Training by maximum likelihood

With $p_i = \sigma(\mathbf w^\top\mathbf x_i + b)$ and labels $y_i \in \{0, 1\}$ , the likelihood is $\prod_ip_i^{y_i}(1 - p_i)^{1 - y_i}$ . Maximising its log is the same as minimising the cross-entropy (log) loss:

J(\mathbf w) = -\frac1n\sum_{i=1}^n\left[y_i\ln p_i + (1 - y_i)\ln(1 - p_i)\right].

Its gradient has the clean form

\nabla_{\mathbf w}J = \frac1n\sum_{i=1}^n(p_i - y_i)\,\mathbf x_i = \frac1nX^\top(\mathbf p - \mathbf y).

$J$ is convex, so gradient descent (or Newton's method, called IRLS here) reaches the global minimum.
There is no closed-form solution.
If the classes are perfectly separable, the likelihood can be pushed towards 1 by scaling $\mathbf w$ up forever; add L2 regularisation $\frac{\lambda}{2}\lVert\mathbf w\rVert^2$ (as in ridge) to keep weights finite.

Python

import numpy as np

def fit_logistic(X, y, lr=0.1, epochs=1000, lam=0.0):
    w = np.zeros(X.shape[1]); b = 0.0
    for _ in range(epochs):
        p = 1 / (1 + np.exp(-(X @ w + b)))
        w -= lr * (X.T @ (p - y) / len(y) + lam * w)
        b -= lr * np.mean(p - y)
    return w, b

Multi-class: softmax regression

For $K$ classes, keep one weight vector per class:

P(y = k \mid \mathbf x) = \frac{\exp(\mathbf w_k^\top\mathbf x + b_k)}{\sum_{j=1}^K\exp(\mathbf w_j^\top\mathbf x + b_j)},

trained with the categorical cross-entropy $-\sum_i\ln P(y_i\mid\mathbf x_i)$ . The decision regions are separated by linear boundaries. Binary logistic regression is the case $K = 2$ .