Logistic Regression & the Perceptron

Chapter 1: The Sigmoid — A Squashing Function

We need a function that takes any real number — the unbounded score θ^Tx from our linear model — and gently bends it into the range (0, 1). It should map large positive scores near 1 ("definitely spam"), large negative scores near 0 ("definitely not"), and a score of 0 to exactly 0.5 ("a coin flip"). The function that does this, and does it most naturally, is the logistic function, also called the sigmoid (Greek for "S-shaped"):

Let's sanity-check its behavior by plugging in three values. The whole function lives or dies on the term e^−z in the denominator:

# z = 0:   e⁻⁰ = 1            → g = 1/(1+1)   = 0.5   (maximum uncertainty)
# z = +5:  e⁻⁵ ≈ 0.0067       → g = 1/1.0067  ≈ 0.993 (almost certainly class 1)
# z = −5:  e⁻⁽⁻⁵⁾ = e⁵ ≈ 148  → g = 1/149     ≈ 0.0067 (almost certainly class 0)

As z → +∞, e^−z → 0 so g → 1. As z → −∞, e^−z → ∞ so g → 0. And it passes smoothly through 0.5 at z = 0. No matter how extreme the input, the output is trapped in (0, 1) — exactly the leash we needed. Our hypothesis becomes the linear score, squashed:

We keep every convention from linear regression — including the bias trick x₀ = 1, so θ^Tx = θ₀ + θ₁x₁ + …. The only change is wrapping that weighted sum in g. Everything you learned about θ^Tx as a dot product still holds; we've just added a final squash.

A derivative so clean it feels like a gift

We'll need the derivative of g for training (next chapters). It has a famously beautiful form — the sigmoid's slope can be written entirely in terms of the sigmoid's own value. Let's derive it, every step, because the result is what makes the logistic update rule come out so simply:

So g′(z) = g(z)(1 − g(z)). The slope of the sigmoid at any point is its output times one-minus-its-output. It's biggest (= 0.25) at z = 0 where g = 0.5, and vanishes toward the flat tails where g is near 0 or 1. Remember this identity — it's the hinge that, in Chapter 4, collapses a scary-looking gradient into the elegant (y − h)x.

Chapter 2: From Scores to a Decision Boundary

Now the geometry. With two features x₁, x₂ (say "count of the word free" and "count of links"), every email is a point in a 2D plane. The model h(x) = g(θ₀ + θ₁x₁ + θ₂x₂) assigns each point a probability. Where does it switch from "probably not spam" to "probably spam"? Exactly where the probability crosses 0.5 — and we showed g(z) = 0.5 precisely when z = 0. So the decision boundary is the set of points where

That's the equation of a straight line (in higher dimensions, a flat hyperplane). On one side θ^Tx > 0 so h > 0.5 → predict class 1; on the other side θ^Tx < 0 so h < 0.5 → predict class 0. This is the key picture: logistic regression carves the feature space with a straight cut, and the sigmoid paints a smooth probability gradient that's steepest right at the cut and saturates as you move away from it.

The log-odds interpretation

There's a deeper reading of θ^Tx. Start from h = 1/(1+e^−z) with z = θ^Tx, and solve for z in terms of the probability. A few lines of algebra (invert the sigmoid):

So θ^Tx = log( h / (1−h) ). The quantity h/(1−h) is the odds of class 1 (probability of yes divided by probability of no — "3-to-1 odds" means h = 0.75). Its logarithm is the log-odds or logit. The punchline: logistic regression is a plain linear model in log-odds space. Each weight θ_j says how many units the log-odds of spam shift per unit of feature j. That's why it's "linear" regression — the linearity lives in the log-odds, and the sigmoid is just the inverse map back to probability.

python
import numpy as np
def sigmoid(z): return 1 / (1 + np.exp(-z))

def predict_proba(theta, x):
    return sigmoid(theta @ x)        # P(y=1 | x)
def predict_label(theta, x):
    return 1 if theta @ x >= 0 else 0   # θᵀx ≥ 0  ⇔  proba ≥ 0.5

Chapter 3: The Right Loss — Cross-Entropy from Maximum Likelihood

We have a model that outputs probabilities. Now we need a loss — a score of how wrong the model is — so we can train it. The obvious move is to reuse squared error from the last lesson: penalize (h(x) − y)². Resist it. With the sigmoid inside, squared error becomes a non-convex function of θ — a bumpy landscape full of flat spots and local dips where gradient descent stalls. We need a loss that's convex (one bowl, like before) and that respects the probabilistic meaning of h. Maximum likelihood — the same principle that justified least squares in the last lesson — hands us exactly that.

Write down the probability of the data

Our model is a probability statement. It says P(y = 1 | x; θ) = h(x), and therefore P(y = 0 | x; θ) = 1 − h(x). Here's a slick trick to write both cases in one formula, using the fact that anything to the power 0 is 1:

Check it. If y = 1: the formula is h¹(1−h)⁰ = h — correct. If y = 0: it's h⁰(1−h)¹ = 1−h — correct. One expression, both labels. The exponents act as switches that turn each factor on or off.

Assuming the n training emails are independent, the probability of the entire labeled dataset is the product over all of them. Viewed as a function of θ, that's the likelihood:

As before, products of many small numbers are miserable to differentiate, so take the log — turning the product into a sum and the powers into multipliers. The log-likelihood:

Maximum likelihood says: pick the θ that makes this as large as possible (the data most probable). Equivalently, flip the sign and minimize its negative — and the negative of that expression is the famous cross-entropy loss (also called log loss or binary cross-entropy), the single most-used classification loss in machine learning. For one example:

Why this loss has exactly the right shape

Look at what it does to a single confident-but-wrong prediction. Suppose the true label is y = 1 but the model says h = 0.01 ("almost certainly not spam"). The loss is −log(0.01) ≈ 4.6 — a big penalty. Push the model to h = 0.0001 and the loss rockets to ≈9.2. As h → 0 while y = 1, the loss → +∞. Cross-entropy punishes confident wrongness without mercy — being sure and mistaken is the worst thing a probabilistic classifier can do, and the loss agrees. Meanwhile, when the model is right and confident (h → 1 with y = 1), −log(1) = 0: no penalty. That asymmetry — gentle on correct confidence, brutal on wrong confidence — is exactly what you want, and it falls straight out of the −log.

python
def cross_entropy(h, y, eps=1e-12):
    h = np.clip(h, eps, 1-eps)          # avoid log(0)
    return -(y*np.log(h) + (1-y)*np.log(1-h))

cross_entropy(0.99, 1)   # 0.01  — confident & right: tiny loss
cross_entropy(0.01, 1)   # 4.61  — confident & WRONG: huge loss
cross_entropy(0.50, 1)   # 0.69  — a coin flip: ln(2)

Chapter 4: The Update Rule — A Stunning Coincidence

We have the log-likelihood ℓ(θ) to maximize. As in the last lesson, we climb it by gradient ascent: take a step in the direction that increases ℓ. (Ascent, not descent — note the plus sign — because we're maximizing now, not minimizing.) The update is θ_j := θ_j + α · ∂ℓ/∂θ_j. So we need that derivative. Brace for it to look ugly — and then watch the sigmoid's magic identity from Chapter 1 demolish the ugliness.

The derivative (every step)

Work with a single example (x, y) so we can drop the sum. We differentiate ℓ = y log g(θ^Tx) + (1−y) log(1−g(θ^Tx)), writing g for g(θ^Tx):

The g(1−g) from the sigmoid's derivative is precisely what's needed to cancel the 1/g and 1/(1−g) from the log's derivative. Everything collapses. Since g = h(x), the gradient is simply:

which gives the stochastic gradient-ascent update rule:

Wait — we've seen this exact rule before

Look hard at that update. Compare it to the LMS rule from the linear regression lesson: θ_j := θ_j + α(y − h)x_j. They are character-for-character identical. Same learning rate, same (target − prediction) error, same feature gating. Yet these are completely different algorithms: linear regression's h is the raw line θ^Tx; logistic regression's h is the squashed sigmoid g(θ^Tx). Two different models, two different losses, derived from two different probability assumptions — and out pops the same update rule.

Is this a coincidence? No. It's the first visible crack of a deep structure. Both Gaussian-noise regression and Bernoulli classification are members of one family — the exponential family — and for every member, maximum-likelihood gradient ascent produces this same "(error) × (feature)" update. That unifying framework is Generalized Linear Models, the entire subject of the next lesson. For now, savor the surprise: the rule you learned for predicting house prices also trains a spam filter, unchanged.

One update, by hand

Let's run a single logistic update with numbers. One example: x = (1, 2) (bias + one feature), true label y = 1, current θ = (0, 0), learning rate α = 0.5.

Step 1 — score and squash. z = θ^Tx = 0·1 + 0·2 = 0. h = g(0) = 1/(1+e⁰) = 1/2 = 0.5. (At θ = 0 the model is maximally uncertain — sensible.)

Step 2 — error. y − h = 1 − 0.5 = 0.5. The true label is 1, the model said 0.5, so push the probability up.

Step 3 — update each weight via θ_j += α(y−h)x_j:

θ₀ := 0 + 0.5 × 0.5 × 1 = 0.25
θ₁ := 0 + 0.5 × 0.5 × 2 = 0.50

Step 4 — verify. New z = 0.25·1 + 0.50·2 = 1.25; new h = g(1.25) = 1/(1+e^−1.25) ≈ 0.777. The predicted probability of the (correct) class rose from 0.50 to 0.78 in one step. The model is learning to say "yes."

python
def logreg_step(theta, x, y, alpha):
    h = sigmoid(theta @ x)              # squashed prediction
    return theta + alpha * (y - h) * x    # IDENTICAL shape to LMS

theta = np.zeros(2)
print(logreg_step(theta, np.array([1.,2.]), 1, 0.5))  # [0.25 0.5] ✓

Chapter 5: The Training Lab — Watch a Classifier Learn

Time to put it all in motion. Below is a live logistic-regression trainer. Two clouds of points — class 0 (blue) and class 1 (orange) — sit in the feature plane. The model starts with random weights, so its decision boundary is nonsense. Press Train and watch gradient ascent on the log-likelihood swing the boundary into place, the probability field sharpen, and the log-likelihood climb. Everything from Chapters 1–4 happening at once.

One thing to watch for, because it's the soul of the algorithm: as training proceeds, the points the model is already sure about stop contributing. Recall the update weight is (y − h) — the error. For a point deep in its own territory, h is already ~1 (or ~0) and matches y, so (y − h) ≈ 0: that point stops tugging on the boundary. The only points still exerting force are the ones near the boundary, where the model is uncertain. Logistic regression naturally focuses its attention on the hard, borderline cases — the same instinct that, taken to its logical extreme, gives the support vector machine.

(No quiz — the lab is the test. If you can predict how the boundary will move before pressing Train, you understand logistic regression.)

Chapter 6: The Perceptron — Logistic Regression's Hard-Edged Ancestor

Let's take a short, illuminating detour back in time. Logistic regression's sigmoid gives a soft answer — "73% spam." What if we force a hard verdict instead, replacing the smooth S-curve with a brutal step function that outputs only 0 or 1?

Keep everything else identical — same hypothesis h(x) = g(θ^Tx), same update rule θ_j := θ_j + α(y − h(x))x_j. With this threshold g, the algorithm is the perceptron, invented by Frank Rosenblatt in 1958 and once breathlessly hailed as a model of how neurons in the brain learn. It is the great-grandparent of every neural network alive today.

Because h now outputs only 0 or 1, the error term (y − h) takes only three values, and the update becomes almost comically simple:

• Correct prediction (h = y): error = 0 → no change. The perceptron learns only from its mistakes.
• Predicted 0, true 1 (y − h = +1): add α·x to θ — nudge the boundary so this point falls on the "1" side next time.
• Predicted 1, true 0 (y − h = −1): subtract α·x from θ — push the point to the "0" side.

The perceptron has a celebrated guarantee, the Perceptron Convergence Theorem: if the data is linearly separable, this mistake-driven rule will find a separating boundary in a finite number of steps — full stop. But the flip side is its fatal flaw: if the data is not separable (even one stray point), it never converges — it cycles forever, chasing the unfixable mistake. This brittleness, exposed in Minsky and Papert's 1969 book (the perceptron can't even learn XOR), helped trigger the first "AI winter."

Chapter 7: Many Classes — Softmax Regression

Spam-or-not has two classes. But what about classifying an email into spam, personal, or work — or an image into one of 1,000 object categories? We need to generalize from a yes/no sigmoid to a "pick one of k" model. The generalization is one of the most important functions in all of deep learning: the softmax.

The idea: instead of one weight vector θ, keep k of them — θ₁, …, θ_k, one per class. Each produces a raw score (a logit) t_i = θ_i^Tx measuring how much x looks like class i. But raw scores can be negative and don't sum to 1, so they aren't probabilities. The softmax fixes both problems at once — exponentiate (forcing positivity) and normalize (forcing them to sum to 1):

The exp makes every score positive and amplifies differences (a slightly larger logit becomes a much larger probability — "winner takes more"); dividing by the sum guarantees the k outputs form a valid probability distribution. With k = 2, softmax collapses exactly back to the sigmoid — it's the honest generalization, not a different idea.

The loss and its gorgeous gradient

The loss for the multiclass case is cross-entropy again — the negative log-probability the model assigned to the true class y: ℓ_CE = −log φ_y = −log( exp(t_y) / ∑_j exp(t_j) ). And its gradient with respect to logit i is breathtakingly simple:

where 1{y = i} is the indicator — 1 if i is the true class, 0 otherwise (the "one-hot" label). In words: the gradient is predicted probability minus the true (one-hot) label — written compactly, φ − e_y. This is the exact same "(prediction − target)" shape as the binary case and as LMS. Push down the probabilities of wrong classes, push up the probability of the right one, in proportion to how off you are. Chain it through t_i = θ_i^Tx and you get ∂ℓ/∂θ_i = (φ_i − 1{y=i})·x — the universal update, one more time.

python
def softmax(t):
    t = t - t.max()                  # subtract max for numerical stability
    e = np.exp(t); return e / e.sum()

def softmax_ce_grad(Theta, x, y, k):
    phi = softmax(Theta @ x)         # predicted class probabilities, shape (k,)
    onehot = np.eye(k)[y]            # true label as a one-hot vector
    return np.outer(phi - onehot, x)   # ∂ℓ/∂Θ = (φ − e_y) xᵀ

Chapter 8: Newton's Method — Using Curvature to Go Faster

Gradient ascent is reliable but slow — it only knows the slope, so near a gently-curving maximum it takes many timid steps. What if we also used the curvature — how fast the slope itself is changing — to jump intelligently? That's Newton's method, and for logistic regression it converges in a handful of iterations where gradient ascent needs hundreds.

Start with the classic use of Newton's method: finding a zero of a function f, i.e. solving f(θ) = 0. The trick: at your current guess, replace the curvy f with its tangent line, and jump to where that line crosses zero. The tangent has slope f′(θ), so a little geometry gives the update:

To maximize ℓ(θ) rather than zero a function, recall a maximum sits where the derivative is zero. So apply Newton's zero-finder to f = ℓ′: we want ℓ′(θ) = 0. Substituting f = ℓ′ (so f′ = ℓ″) gives the optimization form:

The update divides the slope by the curvature. Where the function is sharply curved (large ℓ″), it takes cautious steps; where it's nearly flat, it takes bold ones — automatically, with no learning rate to tune. That's the magic: Newton's method has no α. The curvature sets the step size for you.

The multidimensional version: Fisher scoring

With many parameters, the slope ℓ′ becomes the gradient vector ∇_θℓ, and the curvature ℓ″ becomes the Hessian matrix H — the matrix of all second partial derivatives, H_ij = ∂²ℓ / ∂θ_i∂θ_j. The update generalizes to the Newton–Raphson rule:

Applied to logistic regression's log-likelihood, this is called Fisher scoring (or IRLS — iteratively reweighted least squares). It typically converges in well under ten iterations. The cost: each step must build and invert the d×d Hessian, which is O(d³) — cheap when features number in the dozens, but ruinous for the millions of parameters in a neural net. That's the trade-off in one line: Newton uses second-order information for blistering convergence but pays a d³ price per step; gradient descent is first-order, slow but O(d) and scalable. It's exactly why deep learning runs on (stochastic) gradient descent and its cousins, not Newton's method — the same scalability story from the normal-equations-vs-GD discussion in the last lesson.

Chapter 9: When It Breaks, & Connections

You can now build a classifier from zero, derive its loss from first principles, and train it three ways. Before the cheat sheet, the failure modes — because knowing where logistic regression breaks is what makes you trust it where it doesn't.

Perfect separation blows up the weights

If the two classes are perfectly separable, there's no finite "best" θ. The model can always raise the log-likelihood a hair by scaling θ larger — making the boundary sharper and pushing every correct probability closer to a perfect 0 or 1. So gradient ascent drives |θ| → ∞ and never settles (you saw this in the Ch.5 lab with well-separated clouds). The cure is regularization: add a penalty λ|θ|² on the weights' size, so the model balances fitting the data against staying modest. This also gives more sensible, better-calibrated probabilities.

Class imbalance & the non-linear boundary

If 99% of emails are non-spam, a model can score 99% accuracy by predicting "not spam" always — useless. Cross-entropy and class-weighting help, but you must watch precision/recall, not raw accuracy. And remember Ch.2: the boundary is always a straight hyperplane. For tangled classes (XOR, concentric rings), feed in nonlinear features or move to a kernel method or neural net — logistic regression on the right features is astonishingly strong, but on raw tangled inputs it's helpless.

The whole lesson on one page

Full from-scratch trainer

python
import numpy as np
def sigmoid(z): return 1/(1+np.exp(-z))

def fit_logistic(X, y, alpha=0.1, iters=2000):
    theta = np.zeros(X.shape[1])
    for _ in range(iters):
        h = sigmoid(X @ theta)                 # predicted probabilities
        theta += alpha * X.T @ (y - h)         # gradient ASCENT: + (y−h)·x summed
    return theta

# sklearn one-liner (uses a fast solver like L-BFGS + L2 regularization):
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression().fit(X_raw, y)
clf.predict_proba(X_new)    # calibrated P(y=1)

Where to go next

• Linear Regression & LMS — the prerequisite; the identical update rule started here.
• Generalized Linear Models (next) — the framework that explains why linear and logistic regression share an update rule. Both are exponential-family GLMs; the magic stops being magic.
• Loss Functions — cross-entropy in full generality, plus the contrastive and ranking losses that extend it.
• Support Vector Machines — the other great linear classifier; it focuses entirely on the borderline points logistic regression already leans toward.
• Kernels — how to give a linear classifier curved boundaries without manually building features.

"The perceptron may eventually be able to learn, make decisions, and translate languages." — The New York Times, 1958. They were early, but not wrong — the line that learned to say yes became the last layer of every neural network.