Image Formation: The Pinhole Camera Model

Chapter 1: Similar Triangles: x = fX/Z

The pinhole camera model imagines a box with a single tiny hole at one face. Light from the world passes through the hole and lands on the opposite face — the image plane. No lens, no blur, just straight rays.

Setting up coordinates

Place the camera center (the pinhole, also called the optical center) at the origin of the camera frame. The optical axis is the z-axis, pointing into the scene. The image plane is at distance f (the focal length) along the z-axis.

A 3D point in the camera frame has coordinates p^c = (p_x^c, p_y^c, p_z^c). The ray from the origin through p^c intersects the image plane at some 2D location (u_m, v_m), where the subscript m means "in meters" (we convert to pixels later).

Deriving u_m = f · p_x^c / p_z^c

Look at the xz-plane only. The point is at (p_x^c, p_z^c). The image plane is at z = f. The ray from origin through the point hits z = f at:

This is pure similar-triangles geometry: the small triangle (origin to image plane, height u_m, base f) is similar to the large triangle (origin to point, height p_x^c, base p_z^c). By symmetry, the same holds for y:

Worked example

A point is at p^c = (0.3 m, −0.1 m, 2.0 m). Focal length f = 0.05 m (5 cm — a typical wide-angle lens).

The image is 7.5 mm right of center and 2.5 mm above center (y-axis flipped in image coords). Now look at p^c = (0.6 m, −0.2 m, 4.0 m) — double the distance, double all coords. Same pixel! Scale ambiguity confirmed.

Chapter 2: Homogeneous Coordinates

The projection formulas u_m = f·X/Z and v_m = f·Y/Z involve a division by Z, making them nonlinear. Nonlinear equations are hard to chain together, hard to invert, and hard to reason about algebraically. Homogeneous coordinates solve this by embedding a 3D point into 4D space, transforming the nonlinear projection into a linear matrix multiplication — at the cost of one extra dimension.

The homogeneous lift

A 3D point p = (X, Y, Z) becomes the homogeneous vector p̃ = (X, Y, Z, 1)^T. A 2D image point (u, v) becomes (u, v, 1)^T, or more generally (u·w, v·w, w)^T for any w ≠ 0 — all such triples represent the same image point.

Projection as matrix multiplication

Watch the division-by-Z become the third row in a matrix product:

The right side gives (fX, fY, Z)^T. Dividing all entries by Z (the third component) recovers (fX/Z, fY/Z, 1)^T = (u_m, v_m, 1)^T. This "divide by the third entry to get physical coordinates" is called dehomogenization.

More compactly, using the 3×4 canonical projection matrix Π₀:

Worked example

Point p^c = (0.3, −0.1, 2.0, 1)^T in homogeneous form. With f = 0.05:

python
import numpy as np

f = 0.05
K_metric = np.array([[f, 0, 0],
                     [0, f, 0],
                     [0, 0, 1]])

Pi0 = np.array([[1,0,0,0],
                [0,1,0,0],
                [0,0,1,0]])

p_hom = np.array([0.3, -0.1, 2.0, 1.0])
lam_x = K_metric @ Pi0 @ p_hom   # = [0.015, -0.005, 2.0]
u_m = lam_x[0] / lam_x[2]        # = 0.0075 m
v_m = lam_x[1] / lam_x[2]        # = -0.0025 m
print(f"u_m = {u_m:.4f} m, v_m = {v_m:.4f} m")
# u_m = 0.0075 m, v_m = -0.0025 m  (same as hand calc)

Chapter 3: The Intrinsic Matrix K

We have projection in meters (u_m, v_m). Sensors don't return meters — they return pixels. Converting involves two steps: scaling by pixel density and shifting the origin to the image corner.

Step 1: meter → pixel scaling

A digital sensor has s_x horizontal pixels per meter and s_y vertical pixels per meter. Multiplying the metric coordinates by these densities converts to pixels:

where (o_x, o_y) is the principal point — the pixel where the optical axis pierces the image plane (typically near the center of the image). The shift is needed because the metric frame has its origin at the image center, but pixels are counted from the top-left corner.

Step 2: combining into K

Substituting u_m = f·X/Z and the pixel conversion together, and writing it as a single matrix acting on camera-frame coordinates:

where:

Projection formula in pixels

The full expression, step by step:

Worked numbers

Camera: f_x=800, f_y=800, c_x=320, c_y=240, s=0. Point in camera frame: (0.3, −0.1, 2.0).

Chapter 4: Extrinsics [R|t]: World → Camera

The intrinsic matrix K handles the camera's internal geometry. But our 3D points often live in a world frame w, while K expects coordinates in the camera frame c. We need a transform that moves points from world to camera.

The rigid body transform

From VNAV L1, the pose of camera frame c relative to world frame w is the SE(3) matrix T_wc. To transform a world point p^w into camera coordinates p^c, we need the inverse — how to express a world point in camera coordinates. This is T_cw = T_wc⁻¹:

where R_cw is the rotation from world to camera (a 3×3 orthogonal matrix) and t_cw is the camera center position expressed in the camera frame (roughly, where the world origin is, as seen by the camera).

The extrinsic matrix

In homogeneous coordinates, this transform becomes a 3×4 matrix:

It maps a homogeneous world point p̃^w = (X_w, Y_w, Z_w, 1)^T to a (non-homogeneous) camera-frame vector (X_c, Y_c, Z_c)^T.

Physical intuition: what does t_cw mean?

t_cw = −R_cw C, where C is the camera center in world coordinates. It is not the camera center position — it is the world origin expressed in the camera frame. This is a common confusion. To find where the camera is in the world: C = −R_cw^T t_cw.

Chapter 5: Full Pipeline: P = K[R|t]

We now have all the pieces: extrinsics [R|t] to go from world to camera, and K to go from camera 3D to image pixels. Chaining them gives the projection matrix P — a single 3×4 matrix that projects any world point directly to a pixel.

Where p̃^w = (X_w, Y_w, Z_w, 1)^T is the world point in homogeneous coordinates and Z_c is the depth in the camera frame (required for the final dehomogenization step).

Recovering pixel coordinates

Compute q = P p̃^w ∈ R³. Then:

Worked end-to-end example

Camera at world position C^w = (1.0, 0, 0) m, looking along the z-axis (identity rotation). World point at p^w = (1.3, −0.1, 3.0). K: f_x=800, f_y=800, c_x=320, c_y=240.

python
import numpy as np

# Camera at C_w = [1.0, 0, 0], looking along world-z (identity rotation)
R_cw = np.eye(3)
t_cw = -R_cw @ np.array([1.0, 0.0, 0.0])  # = [-1, 0, 0]

K = np.array([[800, 0, 320],
              [0, 800, 240],
              [0, 0, 1]])

Ext = np.column_stack([R_cw, t_cw])   # 3x4
P = K @ Ext                             # 3x4 projection matrix

p_world = np.array([1.3, -0.1, 3.0, 1.0])
q = P @ p_world                         # = [P*p], 3-vector
u, v = q[0]/q[2], q[1]/q[2]
print(f"pixel: ({u:.1f}, {v:.1f})")
# Step by step:
# p_c = R_cw @ p_world[:3] + t_cw = [1.3-1, -0.1, 3.0] = [0.3, -0.1, 3.0]
# K @ p_c = [800*0.3/3+320, 800*(-0.1)/3+240] = [400, 213.3]
# pixel: (400.0, 213.3)

Chapter 6: Back-Projection: Pixel → Ray

Given a pixel (u, v), can we find the corresponding 3D point? Not uniquely — we can only find the ray along which the 3D point lies. This is the inverse of projection, and it is the foundation of triangulation, depth estimation, and structure-from-motion.

Inverting K to get a direction

Starting from u = (f_x X/Z) + c_x, solve for X/Z:

This gives us the normalized image coordinates (x_n, y_n) = ((u−c_x)/f_x, (v−c_y)/f_y). The ray in camera coordinates is proportional to:

Any 3D point on the ray satisfies p^c = λ d^c for some scalar λ > 0. The depth λ is unknown from a single image.

Worked example: back-projecting pixel (440, 200)

K: f_x=800, f_y=800, c_x=320, c_y=240.

Ray direction (in camera frame): d = (0.15, −0.05, 1)^T. Normalize: |d| = √(0.0225 + 0.0025 + 1) ≈ 1.0124. Unit ray: (0.148, −0.049, 0.988).

If we know depth Z_c=2m: p^c = 2 × (0.15, −0.05, 1) = (0.3, −0.1, 2.0) — which matches the original 3D point from Chapter 1. The pipeline is consistent.

python
import numpy as np

K = np.array([[800,0,320],[0,800,240],[0,0,1]])
K_inv = np.linalg.inv(K)

pixel = np.array([440, 200, 1])    # homogeneous pixel
ray = K_inv @ pixel                 # = [0.15, -0.05, 1.0]
ray_unit = ray / np.linalg.norm(ray)
print("Ray direction:", ray)          # [0.15, -0.05, 1.0]

# If depth Z_c = 2m is known:
depth = 2.0
p_cam = depth * ray                  # = [0.3, -0.1, 2.0] ✓

Chapter 7: Lens Distortion

The pinhole model assumes perfectly straight rays. Real lenses bend light. Wide-angle lenses bend it a lot. The result is lens distortion: straight lines in the world appear curved in the image.

Radial distortion

The most common distortion is radial: the further a point is from the image center, the more it shifts outward (barrel distortion) or inward (pincushion distortion). The model in the camera frame (before adding the principal point):

where r² = (u^distort)² + (v^distort)² is the squared distance from the image center. The coefficients a₁, a₂ are called distortion coefficients:

Undistortion in image coordinates

Using the image frame (with principal point), distortion correction becomes:

Worked distortion example

Distorted pixel at (u^d=350, v^d=150), c_x=320, c_y=240, a₁=−0.2, a₂=0.05.

Wait — that blows up! This is why distortion coefficients are always very small (|a₁| < 0.5, |a₂| < 0.1), and r is measured in normalized coordinates (not pixels). Let's redo with normalized coords (after dividing by f):

A gentle inward pull of 0.28% — typical barrel correction. The corrected normalized coords: (0.0375 × 0.99719, −0.1125 × 0.99719) ≈ (0.0374, −0.1122).

python
import numpy as np

def undistort_pixel(u_d, v_d, K, a1, a2):
    """Apply radial undistortion to a single pixel."""
    fx, fy = K[0,0], K[1,1]
    cx, cy = K[0,2], K[1,2]
    # Normalize to camera frame (unitless)
    xn = (u_d - cx) / fx
    yn = (v_d - cy) / fy
    r2 = xn**2 + yn**2
    factor = 1 + a1*r2 + a2*r2**2
    # Undistorted normalized coords
    xn_u = xn * factor
    yn_u = yn * factor
    # Back to pixels
    u = xn_u * fx + cx
    v = yn_u * fy + cy
    return u, v

K = np.array([[800,0,320],[0,800,240],[0,0,1]])
u, v = undistort_pixel(350, 150, K, a1=-0.2, a2=0.05)
print(f"Undistorted: ({u:.2f}, {v:.2f})")  # (349.92, 150.25)

Chapter 8: Showcase: Full Camera Explorer

Now everything comes together. A 3D scene (wireframe cube) is projected through the full pipeline P = K[R|t], with optional radial distortion applied on top. You control K (focal length, principal point), camera pose (translation, yaw), and distortion coefficients. This is the complete model you need to understand to do any visual navigation.

Chapter 9: Connections & Cheat Sheet

You now hold the complete forward model for how cameras see the world. Here is the distilled reference you'll reach for constantly as you go deeper into visual navigation.

Camera Cheat Sheet

What's next in VNAV

Links to related Gleams

• VNAV L1: 3D Geometry & SE(3) — the reference frames and transforms used by [R|t]
• VNAV L2: Lie Groups — optimizing camera poses on the SO(3) manifold
• Multiview Geometry — epipolar constraints, triangulation, and the Essential matrix
• NeRF & 3D Gaussian Splatting — modern 3D reconstruction using the camera model from this lesson
• Classical VIO — fusing camera projection with IMU for visual inertial odometry

"The camera does not capture reality. It records projections of reality. Understanding the projection is the first step toward recovering reality."
— Luca Carlone, MIT 16.485