NN Building Blocks & Where Compute Lives

Chapter 1: Linear Layer: Params & MACs

The Linear layer (also called fully-connected, or FC) is the simplest building block. Every output neuron connects to every input neuron. If the input has C_in features and the output has C_out features, then every output y_j is computed as:

The weight tensor W has shape (C_out, C_in). The bias b has shape (C_out,). Count the elements:

For the MAC count, each output neuron requires C_in multiply-accumulate operations (one per input × weight pair), and there are C_out output neurons:

Concrete example — BERT's final projection: C_in=768, C_out=768.

Notice params ≈ MACs for a Linear layer — one MAC per parameter, because each weight is used exactly once per forward pass (no spatial reuse). This is the key difference from Conv, where each weight gets reused across spatial positions.

AlexNet's two giant FC layers (4096→4096 and 4096→1000) together hold 4096²+4096×1000 = 20,873,216 params — 34% of AlexNet's total 61M — yet do only 20.9M MACs, 2.9% of AlexNet's 724M total. Params are expensive on storage (41 MB at FP32) but cheap on compute. A 4× weight quantization from FP32→INT8 saves 31 MB while leaving MACs unchanged. This is why quantization disproportionately benefits models with large FC layers.

Chapter 2: Conv2d: The Workhorse

A 2D convolution layer takes an input feature map of shape (C_in, H_in, W_in) and produces an output of shape (C_out, H_out, W_out). The kernel has spatial size K_h × K_w. There is one kernel per (input channel, output channel) pair — so the weight tensor has shape (C_out, C_in, K_h, K_w).

Count the parameters: C_out output channels, each needing a kernel of size C_in × K_h × K_w, plus one bias per output channel:

For MACs: each of the H_out × W_out output positions requires computing C_out output values, each needing C_in × K_h × K_w multiply-accumulates:

The crucial insight: params don't depend on spatial size, but MACs do. The same weight tensor is reused at every output position. A layer with output spatial H_out×W_out = 56×56 = 3,136 performs 3,136× more MACs than a layer with output 1×1 — with identical parameter count.

Output spatial size formula

Given input height H_in, kernel K_h, padding p, and stride s, the output height is:

Common cases: "same" padding (output = input size): p = (K−1)/2 with s=1. "Valid" padding (no padding, p=0): H_out = H_in − K + 1. Stride-2 halving: p=1, s=2, K=3 → H_out = H_in/2. For the AlexNet first conv: H_in=224, K=11, p=2, s=4 → H_out = ⌊(224+4−11)/4⌋+1 = ⌊217/4⌋+1 = 54+1 = 55. ✓

This formula matters for counting MACs: if you miscompute H_out, your MAC count is wrong. The padding and stride choices also affect how many times each input pixel contributes to outputs — relevant for understanding why strided conv reduces activations without reducing params.

Now apply the formula to AlexNet's first layer: 3→96 channels, 11×11 kernel, output 55×55 (stride 4 applied to 224×224 input):

That first layer alone costs 105M MACs — 14.5% of AlexNet's total 724M MACs — yet holds only 34K of AlexNet's 61M parameters (0.06%). Classic early-layer profile: cheap on params, expensive on compute due to large spatial maps.

python
def conv2d_cost(c_in, c_out, kh, kw, h_out, w_out):
    """Compute parameter count and MACs for a Conv2d layer."""
    params = c_out * c_in * kh * kw + c_out  # weights + bias
    macs = c_in * c_out * kh * kw * h_out * w_out
    return params, macs

# AlexNet layer 1: 3→96, 11×11 kernel, 55×55 output
p, m = conv2d_cost(3, 96, 11, 11, 55, 55)
print(f"Params: {p:,}  MACs: {m:,}")
# Params: 34,944  MACs: 105,415,200

# 3×3 conv 64→128 on 56×56 (common ResNet block)
p, m = conv2d_cost(64, 128, 3, 3, 56, 56)
print(f"Params: {p:,}  MACs: {m:,}")
# Params: 73,856  MACs: 231,211,008

Here is the complete AlexNet MAC and activation profile, derived from the formula above. Study the per-layer numbers — they reveal patterns you'll see in every CNN:

Key observations: (1) 62% of params are in FC1+FC2 alone. (2) 85% of MACs are in the five conv layers. (3) Activation element count peaks at the input (150,528) — not shown, but the input + first output together = 440,928 elements, which is the peak activation memory AlexNet requires simultaneously.

Chapter 3: Receptive Field & Spatial Reach

The receptive field of an output neuron is the region of the original input that influences its value. A single 3×3 conv layer sees a 3×3 patch. Stack two 3×3 conv layers and the output sees a 5×5 patch of the original input. Stack three and it sees 7×7. The formula for L layers with kernel size k (and stride 1) is:

For L=3, k=3: RF = 3×2+1 = 7×7. That's the same receptive field as a single 7×7 conv — but with dramatically fewer parameters. A single 7×7 conv with C channels has 7²=49 weights per (in,out) channel pair; three stacked 3×3 convs have 3×3²=27 — 45% fewer params, with an identical view of the input. This is why VGG replaced large kernels with stacked 3×3s.

Receptive field matters for efficiency because large RF is needed for scene understanding — but achieving it with a single large kernel is parameter-inefficient. Stacking small kernels also introduces more non-linearity (one ReLU per layer), which improves representational power. The tradeoff: more layers = more activation memory, since each intermediate activation map must be kept live for the residual connection or gradient computation.

VGG vs single large kernel: VGG-16 uses 13 layers of 3×3 convolutions to achieve an effective RF that would require a single 13×13 or larger kernel. The 3×3 stack has 13×(2×3²×C²) = 23.4C² params per unit output; a single 13×13 layer would need 13²×C² = 169C² params — 7.2× more for the same RF. Plus each 3×3 layer introduces a nonlinearity, giving VGG's stacked 3×3s better discriminative power per parameter. This "stack small kernels" insight is now standard across most CNN architectures. Dilated (atrous) convolutions offer another approach: use K=3 with dilation rate r to sample input at positions separated by r pixels, achieving RF = K+(K-1)(r-1) without any increase in params or compute.

Chapter 4: Grouped & Depthwise Separable Conv

Standard convolution treats all input channels jointly for all output channels. What if we could break that coupling? Grouped convolution divides the C_in input channels into g equal groups and applies a separate, narrower convolution within each group. Each group sees only C_in/g input channels and produces C_out/g output channels:

Both params and MACs shrink by exactly g×. AlexNet used g=2 to split across two GPUs in 2012 — it was an engineering hack that became a principled design choice.

Worked example — AlexNet Conv2, g=2: C_in=96, C_out=256, K=5×5. Standard conv params would be 256×96×25 = 614,400. With g=2: params = 614,400/2 = 307,200. MACs similarly halved: 96×256×25×27×27/2 = 223,948,800 (vs 447,897,600 for standard). The g=2 grouping effectively creates two parallel conv towers processing 48 input channels each and producing 128 output channels each — they never interact except at the next layer.

Depthwise Separable Convolution — the MobileNet trick

Take grouped convolution to the extreme: set g = C_in = C_out = C. Now each channel gets its own independent K×K spatial filter. This is a depthwise convolution. It captures spatial structure within each channel but never mixes channels. To mix channels afterward, apply a 1×1 pointwise convolution. Together they form depthwise separable convolution.

Step 1 — Depthwise conv (C channels, K×K kernel, output H_out×W_out):

Step 2 — Pointwise conv (C_in=C → C_out, 1×1 kernel):

Total depthwise separable (C_in=C_out=C for simplicity):

Standard conv (same dimensions):

The FLOP reduction ratio:

For typical values: K=3, C=128 → ratio = 1/128 + 1/9 ≈ 0.119 — roughly 8× fewer MACs. For K=3, C=256 → ratio ≈ 0.115 — still ~9×. The reduction is dominated by 1/K² for small channels but by 1/C for large channels.

MobileNetV1 architecture (Howard et al. 2017) replaces every standard 3×3 conv in a VGG-like network with depthwise separable conv. Result: 28× fewer MACs than VGG-16 (569M vs 15.3B), only 1% top-1 accuracy drop on ImageNet. This single architectural change — one formula substitution — made real-time vision on smartphones feasible. MobileNetV2 and V3 further add inverted residuals and hard-swish activations for even better efficiency.

python
def depthwise_sep_cost(c, k, h_out, w_out, c_out=None):
    """Depthwise separable conv: DW + PW."""
    if c_out is None: c_out = c
    # Depthwise: one k×k filter per channel
    params_dw = c * k * k
    macs_dw = c * k * k * h_out * w_out
    # Pointwise: 1×1 conv mixing channels
    params_pw = c_out * c
    macs_pw = c * c_out * h_out * w_out
    return params_dw + params_pw, macs_dw + macs_pw

def std_conv_cost(c_in, c_out, k, h_out, w_out):
    return c_out * c_in * k * k, c_in * c_out * k * k * h_out * w_out

c, k, h = 128, 3, 56
p_std, m_std = std_conv_cost(c, c, k, h, h)
p_ds, m_ds = depthwise_sep_cost(c, k, h, h)
print(f"Standard:  params={p_std:,}  MACs={m_std:,}")
print(f"DS:        params={p_ds:,}   MACs={m_ds:,}")
print(f"Reduction: {m_std/m_ds:.1f}× MACs  {p_std/p_ds:.1f}× params")
# Standard:  params=147,456  MACs=462,422,016
# DS:        params=17,536   MACs=54,992,896
# Reduction: 8.4× MACs  8.4× params

Chapter 5: Showcase: Conv Cost Comparator

Drag the sliders to set C_in, C_out, kernel size K, and output spatial size H. The canvas computes and compares parameters and MACs side-by-side for standard conv, grouped conv (g=4), and depthwise separable conv. Watch how the relative costs shift as you change the parameters.

The three architectures represent three points on the parameter-accuracy tradeoff frontier:

• Standard conv — maximum cross-channel interaction at every spatial position. Best accuracy per layer but most expensive. Used in ResNets, VGG, early stages of EfficientNet.
• Grouped conv (g=4) — 4× cheaper. Channels see only 1/4 of the input channels. Used in ResNeXt (g=32) and ShuffleNet (g=3–8). Needs channel shuffle or 1×1 convs to recover cross-group communication.
• Depthwise separable — 8–9× cheaper for K=3. Complete spatial-channel factorization. Used in MobileNet, EfficientNet-B0, SqueezeNet. Requires pointwise conv to restore expressiveness.

Chapter 6: Pooling, Normalization & BN Folding

Pooling

Pooling downsamples the spatial dimensions of a feature map. Unlike conv, it has no learnable parameters — it applies a fixed operation (max or average) over a K×K window, typically with stride = K so windows don't overlap. For an input of shape (C, H, W) and pool size K, the output is (C, H/K, W/K), reducing the spatial area by K².

Pooling is cheap — no weights to store or load — but it destroys spatial information irreversibly. A 2×2 max pool after a 256×56×56 feature map produces 256×28×28, cutting activation size by 4× and making all subsequent layers cheaper.

Global Average Pooling (GAP) is the modern alternative to large FC layers for classification. Instead of flattening the final feature map (e.g., 512×7×7 = 25,088 values → 25,088-dimensional FC), GAP computes the average of each channel's spatial map → one value per channel (512 values). The subsequent FC is then 512→1000 (512K params) rather than 25,088→1000 (25.1M params). GAP was popularized by NiN and used in all MobileNets. Cost: C additions per spatial position, no parameters. The spatial structure is lost but the channel-level summary is preserved.

Strided convolution vs pooling: Rather than conv + pool, modern architectures often use a strided conv (stride=2) to downsample. A stride-2 3×3 conv produces the same output spatial size as conv + 2×2 pool, but the conv layer has learned parameters and can learn how to downsample rather than applying a fixed max/avg. ResNet uses stride-2 convolutions; VGG uses max pooling. The tradeoff: strided conv has more params but is potentially more expressive and eliminates a separate memory pass for the pool layer.

Activation Functions

Every layer's output flows through an activation function — a non-linearity that gives deep networks their expressive power. Without activations, a 100-layer network would collapse to a single linear transformation. The cost: one additional operation per activation value, but zero learnable parameters for most activations.

ReLU (Rectified Linear Unit) is the default: y = max(0, x). One comparison per activation, no parameters, no exp() calls. For a 128×56×56 feature map, that's 401,408 comparisons — trivial. ReLU6 (y = min(max(0,x), 6)) is used in MobileNet to prevent very large activations, which aids fixed-point quantization (values stay in [0,6]).

Swish (y = x/(1+e^-x)) and GELU (used in Transformers) are more expensive — each requires an exp() call, ~4–8× more expensive than ReLU. Hard Swish (y = x·(x+3)/6 for x ∈ [−3,3], clamped otherwise) approximates Swish with only integer ops — used in MobileNetV3 for hardware efficiency.

Residual / Skip Connections

Residual connections (introduced in ResNet, He et al. 2016) add the input of a block directly to its output: y = F(x) + x. The function F is typically two or three conv layers. This lets the network learn residuals — small corrections to the identity — rather than full transformations, making deep networks trainable.

Cost of residual connections: zero additional parameters and zero additional MACs (just an element-wise add). But they require that input and output have the same shape. When the spatial size or channel count changes (e.g., stride-2 downsampling), a 1×1 conv projection shortcut is used to match dimensions — adding params = C_out×C_in and MACs = C_out×C_in×H_out×W_out.

Projection shortcut worked example (ResNet-50 first block): The block downsamples 64→256 channels at stride 2 (28×28 output). A 1×1 projection conv is needed: C_in=64, C_out=256, K=1, H=28, W=28.

For comparison, the 3×3 conv in the same block (64→64): MACs = 64×64×9×28×28 = 28,901,376 ≈ 29M. The projection shortcut adds 44% of the main-path compute — not free! But it only occurs once per stage change (typically 4 times in ResNet-50), so its aggregate cost is small relative to the full network's 4B MACs.

Batch Normalization (training)

Batch Normalization normalizes the activations within a mini-batch to have zero mean and unit variance, then applies a learned scale γ and shift β per channel. During training for a conv feature map of shape (N, C, H, W), BN computes the mean and variance across (N, H, W) for each of the C channels:

This adds 2C trainable parameters (γ and β, one scalar per channel) — tiny. But at inference time, μ and σ are fixed (running statistics from training). The operation becomes a simple linear transform: multiply by (γ/σ) and add (β − γμ/σ). And a linear transform after a convolution is itself a convolution — so BatchNorm can be folded into the preceding conv layer.

LayerNorm (used in Transformers) normalizes across the feature dimension (d) for each token independently, rather than across the batch. For a token vector of size d: 2d parameters (γ, β), computed per token. Unlike BatchNorm, LayerNorm statistics change per-sample — it cannot be folded the same way. Its inference cost is 5 ops per element (subtract mean, divide std, scale, shift + the mean/var computation). For a Transformer with N=512 tokens and d=768: LayerNorm costs 512×768×5 ≈ 2M ops per block — negligible compared to the FFN's 2.4B MACs.

BatchNorm Folding (the inference speedup)

After training, the conv + BN sequence is:

Rearranging into a single convolution with new weights W' and bias b':

The result: zero additional MACs at inference. The BN layer simply disappears into the conv's weights. You save both the normalization arithmetic AND the memory reads for γ, β, μ, σ. Most inference engines (TensorRT, ONNX Runtime, TFLite) do this automatically as a graph optimization pass.

Chapter 7: The Transformer Block

The Transformer block has two sub-components: Multi-Head Self-Attention (MHSA) and a Feed-Forward Network (FFN), each preceded by LayerNorm. For a sequence of N tokens with model dimension d (e.g. BERT-base: N=512, d=768), let's derive where every MAC and every parameter lives.

QKV Projections

Three Linear projections map input X (shape N×d) to Queries, Keys, and Values — each of shape N×d. Three weight matrices, each d×d:

For BERT-base (d=768, N=512): Params_QKV = 3×768²=1,769,472; MACs = 3×512×768²=905,969,664 ≈ 906M per block.

Attention computation

Attention scores are Q × K^T (shape N×N), requiring N² × d multiply-accumulates. Then softmax over the N scores, then weighted sum V (another N² × d MACs):

For N=512, d=768: MACs_attn = 2×512²×768 = 402,653,184 ≈ 403M. Notice: these MACs scale as N² — double the sequence length, quadruple the attention cost. This is the quadratic scaling problem of standard Transformers.

Output projection

Another d×d linear map merges all attention heads: Params = d², MACs = N×d². For BERT-base: Params = 589,824; MACs = 512×768² = 301,989,888 ≈ 302M.

Combining all four attention-related components (QKV + attention + O-proj) for one head with d=768, N=512:

At N=512, d=768: MACs = 4×512×768² + 2×512²×768 = 1,207,959,552 + 402,653,184 = 1,610,612,736 ≈ 1.6B MACs.

Feed-Forward Network (FFN)

Two linear layers: d → 4d → d (the 4× expansion is standard). Followed by a nonlinearity (GELU in BERT):

For BERT-base (d=768, N=512): Params_FFN = 8×768² = 4,718,592 ≈ 4.7M; MACs_FFN = 8×512×768² = 2,415,919,104 ≈ 2.4B per block.

Multi-head attention (MHA) vs single-head: Standard Transformers split attention into h heads (h=12 for BERT-base). Each head uses dimension d/h=64. The QKV weight matrices are still each d×d (just logically partitioned into h blocks). So multi-head attention has the same parameter count and MACs as single-head — the split is conceptual, not a cost change. The benefit is multiple attention patterns (different heads learn different relationships) without additional compute.

Chapter 8: Where Cost Lives in Real Networks

Formulas are one thing; seeing them in a real architecture is another. Let's profile a simplified CNN (ResNet-style) and a Transformer block side-by-side, computing params, MACs, and activation memory per layer. The pattern that emerges is the foundation of every efficiency decision in the rest of this course.

The CNN asymmetry

Consider a simplified CNN: Input 3×224×224 → Conv 3→64, 7×7, stride 4 → Conv 64→128, 3×3 → Conv 128→256, 3×3 → Conv 256→512, 3×3, stride 2 → GlobalAvgPool → FC 512→1000. Spatial sizes shrink as we go deeper.

The output spatial size at each layer: 224→56 (stride 4), 56→56, 56→28 (via stride 2 in Conv3), 28→14, 14→1 (GlobalAvgPool). Channel counts grow to compensate: 3→64→128→256→512. This is the canonical ResNet-style "halve spatial, double channels" design.

The pattern is clear: early layers — huge activation maps (401–802 KB), few params (9K–74K), big spatial MACs. Late conv layers — many params (1.2M for Conv4), smaller spatial maps (14×14), same total MACs (spatial shrinks as channels expand). FC layer — half-million params but trivial MACs (no spatial dimension).

Now consider the activation memory peak. The first conv takes a 3×224×224 input (302 KB at FP16) and produces a 64×56×56 output (401 KB). During computation, both must be live simultaneously — that's 703 KB just for that one layer. On a 256 KB MCU, this is impossible without a patched/tiled inference approach. This is exactly the MCUNet problem: MobileNetV2 reduces parameters dramatically but its peak activation memory is still >1 MB.

The Transformer per-layer breakdown (BERT-base, d=768)

A BERT-base Transformer has 12 identical blocks. Each block's cost (N=512):

ResNet-50 full model: 25M params, 4B MACs, 8 MB activation memory at inference (FP32). A MobileNetV2 with ~70% top-1 accuracy: 3.4M params, 300M MACs — 8× fewer MACs, 7× fewer params. The accuracy drop is only ~6 points. This is the efficiency-accuracy frontier: depthwise separable convolutions buy you 8× on the MAC axis at a small accuracy cost. Future techniques (NAS, knowledge distillation) push further.

Key takeaway: FFN holds 67% of block parameters and 60% of MACs at N=512. Attention holds 0% of parameters but grows to dominate MACs as N increases. This is why GPT-style compression research typically focuses on FFN sparsity (sparse attention, MoE routing) for params, and FlashAttention / linear attention for long-context compute.

This insight — that MACs stay roughly constant across stages when channels double and spatial size halves — is why "stage-wise" network design is so common. ResNet, EfficientNet, and MobileNet all use it. It means the network's total compute is distributed roughly evenly across all stages, rather than concentrated in one place. For efficiency engineers, this is convenient: no single stage dominates, so you can apply a uniform compression ratio across all stages without unbalancing the pipeline.

Chapter 9: Connections & Cheat Sheet

You now have the measurement vocabulary for every layer type in modern deep learning. Here is the complete reference — every formula, plugged with real numbers, and mapped to the efficiency techniques that target each cost type.

Complete Parameter & MAC Formula Reference

Which efficiency technique attacks which cost?

The lesson that motivated each row of this table is the formula you now know. Pruning targets params — you know exactly how many each layer has. Quantization targets model size (params × bitwidth) — you can compute the MB savings. Depthwise separation targets MACs — you derived the 8–9× reduction. MCUNet targets peak activation memory — you know which layers dominate (early, large H×W). FlashAttention targets the N²·d attention term — you derived why it grows quadratically. Every efficient ML paper starts with a cost measurement, using exactly these formulas.

python
"""Per-layer profiler: compute params, MACs, activation size for a model spec."""

def profile_model(layers):
    """layers: list of dicts with type and dimension specs."""
    total_params, total_macs, peak_act = 0, 0, 0
    print(f"{'Layer':<30} {'Params':>10} {'MACs':>12} {'Act(KB)':>8}")
    print("-"*62)
    for L in layers:
        t = L['type']
        if t == 'linear':
            p = L['ci'] * L['co'] + L['co']
            m = L['ci'] * L['co']
            a = L['co'] * 2  # FP16 bytes
        elif t == 'conv':
            p = L['co']*L['ci']*L['k']**2 + L['co']
            m = L['ci']*L['co']*L['k']**2*L['h']*L['w']
            a = L['co']*L['h']*L['w']*2
        elif t == 'dw_sep':
            p = L['c']*L['k']**2 + L['co']*L['c']
            m = (L['c']*L['k']**2 + L['c']*L['co'])*L['h']*L['w']
            a = L['co']*L['h']*L['w']*2
        total_params += p; total_macs += m; peak_act = max(peak_act, a)
        print(f"{L['name']:<30} {p:>10,} {m:>12,} {a/1024:>7.1f}")
    print(f"\nTotal params: {total_params:,}  Total MACs: {total_macs:,}  Peak act: {peak_act/1024:.1f} KB")

# Example: simplified MobileNet-style backbone
profile_model([
    {'name':'Conv1 3→32',    'type':'conv',   'ci':3,   'co':32,  'k':3, 'h':112, 'w':112},
    {'name':'DWSep 32→64',   'type':'dw_sep', 'c':32,  'co':64,  'k':3, 'h':56,  'w':56},
    {'name':'DWSep 64→128',  'type':'dw_sep', 'c':64,  'co':128, 'k':3, 'h':28,  'w':28},
    {'name':'DWSep 128→256', 'type':'dw_sep', 'c':128, 'co':256, 'k':3, 'h':14,  'w':14},
    {'name':'FC 256→1000',   'type':'linear', 'ci':256, 'co':1000},
])

python
"""Full per-layer profiler with PyTorch hook-based MAC counting.
Works on any nn.Module — registers forward hooks to intercept layer shapes."""
import torch
import torch.nn as nn

def count_layer_macs(module, input, output):
    """Forward hook that appends MAC count to module._macs."""
    t = type(module)
    macs = 0
    if isinstance(module, nn.Linear):
        # MACs = batch × C_in × C_out
        macs = input[0].numel() * module.out_features
    elif isinstance(module, nn.Conv2d):
        # MACs = C_in × C_out × K² × H_out × W_out / groups
        n, c_in, h_in, w_in = input[0].shape
        n, c_out, h_out, w_out = output.shape
        kh, kw = module.kernel_size
        macs = (c_in // module.groups) * c_out * kh * kw * h_out * w_out
    module._macs = module._macs + macs if hasattr(module, '_macs') else macs

def profile_model_torch(model, input_shape=(1, 3, 224, 224)):
    """Profile a PyTorch model: params and MACs per layer."""
    hooks = []
    for m in model.modules():
        if isinstance(m, (nn.Linear, nn.Conv2d)):
            hooks.append(m.register_forward_hook(count_layer_macs))

    x = torch.zeros(*input_shape)
    with torch.no_grad():
        model(x)

    print(f"{'Layer':<35} {'Params':>10} {'MACs':>12}")
    print("-"*60)
    total_p, total_m = 0, 0
    for name, m in model.named_modules():
        if hasattr(m, '_macs'):
            p = sum(x.numel() for x in m.parameters())
            total_p += p; total_m += m._macs
            print(f"{name:<35} {p:>10,} {m._macs:>12,}")
    print(f"\nTotal: {total_p:,} params  {total_m:,} MACs")
    for h in hooks: h.remove()

# Example: profile a 3-block MobileNet-style backbone
class DWSep(nn.Module):
    def __init__(self, cin, cout, stride=1):
        super().__init__()
        self.dw = nn.Conv2d(cin, cin, 3, stride=stride, padding=1, groups=cin, bias=False)
        self.pw = nn.Conv2d(cin, cout, 1, bias=False)
    def forward(self, x): return self.pw(self.dw(x))

backbone = nn.Sequential(
    nn.Conv2d(3, 32, 3, stride=2, padding=1, bias=False),
    DWSep(32, 64), DWSep(64, 128, stride=2),
    DWSep(128, 256, stride=2), DWSep(256, 512, stride=2),
    nn.AdaptiveAvgPool2d(1), nn.Flatten(),
    nn.Linear(512, 1000)
)
profile_model_torch(backbone)

Quick formula check — BERT-large (d=1024, 24 blocks, N=512)

BERT-large total parameters (ignoring embeddings): per block = 4d² (attention projections) + 8d² (FFN) = 12d². Times 24 blocks: 24×12×1024² = 301,989,888 ≈ 302M. Plus embeddings (30,522 vocab × 1024 = 31M). Total ≈ 333M — matches the published 340M (small discrepancy from biases and LN).

For comparison, GPT-2 medium (d=1024, 24 layers, same architecture as BERT-large but causal): same param count (~340M) and similar MACs per forward pass. The difference is inference mode — BERT processes all N tokens in parallel (one pass), while GPT-2 generates autoregressively (one token at a time, with the KV cache). At generation time, GPT-2's batch=1 MACs per token are dominated by the FFN weights loaded from DRAM (AI ≈ 0.5 FLOPs/byte), making it memory-bandwidth-bound rather than compute-bound. Same formulas, completely different bottleneck.

BERT-large total MACs (N=512): per block = (4Nd² + 2N²d) + 8Nd² = 12Nd² + 2N²d. Block MACs = 12×512×1024² + 2×512²×1024 = 6,442,450,944 + 537,919,488 = 6,980,370,432 ≈ 7B. Times 24 blocks = 167.5B MACs for one forward pass at N=512. This is why BERT is expensive to run — 167B MACs × 2 FLOPs/MAC = 335 GFLOPs, vs ResNet-50's 4 GFLOPs. LLM inference at long contexts is even more expensive, driven by the N² term growing with context window.

How to use this knowledge in practice

When evaluating a new architecture or a compressed model, run this three-step mental audit:

The efficiency techniques you'll learn in subsequent lessons map directly to this table. Structured pruning (L3) removes entire filters from Conv/Linear layers — directly reducing the C_out dimension, shrinking both params and MACs. Quantization (L5) reduces bits per weight — shrinking model size without touching the MAC formula, but reducing the bytes-loaded per weight and enabling INT8 SIMD parallelism that gives ~4× throughput. Knowledge distillation (L7) reduces the overall network width d — replacing BERT-large (d=1024) with BERT-small (d=256) gives 16× param reduction (d² shrinks 16×) and 16× MAC reduction. Neural Architecture Search (L8) automates the tradeoff — searching the space of C_in, C_out, K, and layer count to minimize MACs subject to an accuracy constraint.

One more mental model: arithmetic intensity = MACs / bytes loaded. For a Linear layer (batch=1): AI = C_in·C_out MACs / (C_in·C_out·2 bytes) = 0.5 FLOPs/byte → deeply memory-bound. For a 3×3 Conv on 56×56: AI = 73K×3K MACs / (73K×2 bytes) = 3,136 FLOPs/byte → compute-bound on any modern GPU. This is why convolutions run efficiently on GPUs (high AI) but LLM autoregressive decoding does not (batch=1 Linear → AI=0.5).

"What I cannot create, I do not understand." — Richard Feynman. If you can compute the parameter count and MAC cost of any layer from its shape alone, you understand that layer. Everything else in efficient deep learning is a consequence of those two numbers.