MCUNet: Deep Learning on Microcontrollers

Chapter 1: SRAM vs Flash — Two Constraints, Two Roles

A microcontroller has two fundamentally different types of memory, and they impose two separate constraints on neural network deployment. Confusing them is the most common mistake when reasoning about MCU limitations.

Flash (also called ROM or program memory) is read-only at runtime. It is non-volatile — it persists when power is off. Flash holds your compiled program, your constant data, and crucially: the model weights. For a typical MCU, Flash is 1–2 MB. Flash is the model-size constraint. An INT8 model with W weights needs W bytes of Flash. You check this once per model deployment and it never changes.

SRAM (Static RAM) is read-write working memory. It holds your program's stack, heap, temporary variables, and crucially: the layer activations during inference. SRAM is the peak activation memory constraint. Unlike Flash, SRAM usage is dynamic — it changes layer by layer as the forward pass executes. The question isn't total activation size across all layers; it's the maximum SRAM needed at any single point in the forward pass.

Let's formalize. For a convolutional layer with input feature map H×W×C_in and output feature map H'×W'×C_out, and assuming INT8 (1 byte per value):

The weights don't contribute to SRAM because they can be fetched from Flash row-by-row during computation. Only the activation tensors (input and output of the current layer) must live in SRAM simultaneously.

Chapter 2: Peak SRAM Analysis — Layer by Layer

Let's work through a concrete example from scratch. Consider a small convolutional network with 5 layers processing a 96×96 grayscale image. We'll compute the SRAM footprint for every layer and find the bottleneck.

The network architecture:

Now compute SRAM for each layer (INT8, bytes = H×W×C for each tensor):

The peak is Conv1 and Conv2, tied at 108 KB. With 256 KB of SRAM on an Arduino Nano, this tiny 5-layer net fits — but barely, and only because we used a 96×96 input instead of the standard 224×224. Scale it up to 224×224 and every number increases by (224/96)² ≈ 5.4× — now Conv1 alone needs 596 KB, more than double the MCU's total SRAM.

Chapter 3: Why Standard Models Fail

Now that we can compute peak SRAM analytically, let's understand concretely why every existing efficient model fails on MCUs — and why the failure is fundamental, not fixable by minor tweaks.

Take MobileNetV2. It was designed for smartphones and achieves 72% ImageNet top-1 with only 3.4M parameters (3.4 MB in INT8). On a Pixel phone with 4 GB DRAM, this is extremely efficient. Let's compute its peak SRAM. The early stage of MobileNetV2:

InvRes-2's depthwise expansion layer alone requires 1,367 KB of SRAM — 5.3× the entire 256 KB SRAM of an Arduino. This is the inverted residual bottleneck design at its worst: MobileNetV2 expands channels by 6× within each block to increase representational capacity, but that expansion creates enormous intermediate activations.

What about quantizing more aggressively? Even INT4 quantization only helps model size (Flash), not SRAM layout. The activation tensor shapes don't change with quantization — a 112×112×96 tensor is 1,204,224 values regardless of bitwidth. INT8 = 1.1 MB SRAM; INT4 = 590 KB SRAM; you'd need INT2 to get near 256 KB, and INT2 quantization of activations causes catastrophic accuracy loss.

What about pruning? Structured pruning reduces channel counts, which helps. But to get MobileNetV2 under 256 KB SRAM would require pruning ≥80% of channels in the early expansion layers — destroying accuracy entirely. Pruning alone cannot bridge a 5.3× gap while preserving useful representations.

The conclusion: we need an entirely different architecture that is designed with MCU SRAM constraints as a first-class objective, not a post-hoc compression target. This is what TinyNAS provides.

Chapter 4: TinyNAS — Designing the Search Space

Neural Architecture Search (NAS) automates the hunt for efficient architectures. You define a search space of possible architectures, a search strategy to explore it, and a performance estimator to rank candidates without full training. But here's the key insight the MCUNet paper makes: the search space itself must be designed for MCU constraints, not just the search within it.

The standard NAS approach for mobile models (ProxylessNAS, MnasNet) defines a search space around MobileNetV2-like blocks at (resolution R=224, width multiplier W=1.0). The smallest architecture reachable within this space still needs 4+ GB DRAM. You could run NAS forever inside this space and never find anything that fits a 256 KB MCU — the search space itself is wrong.

TinyNAS introduces a two-stage approach:

Why does Stage 1 matter so much? Because the peak SRAM and total Flash of a network scale predictably with R and W. Given a target MCU (say, STM32F746: 320 KB SRAM, 1 MB Flash), Stage 1 finds the (R*, W*) combination that puts the search space in a "sweet spot" where most of the sub-networks in the space satisfy the memory budget and are large enough to be accurate.

The key metric is the FLOPs distribution of the satisfying sub-networks. Higher FLOPs (within budget) means more expressive models, which means better accuracy. Stage 1 picks the (R, W) pair that maximizes the 80th percentile FLOPs among sub-networks that satisfy the SRAM+Flash constraints.

Worked example from the paper. For a 320 KB SRAM, 1 MB Flash budget:

• w0.3-r160 (width 0.3×, resolution 160): median FLOPs = 32.5M. Small, fits easily, but low capacity.
• w0.5-r144: median FLOPs = 47M. Better capacity, 80th percentile of satisfying models ≈ 50M FLOPs.
• w0.6-r112: median FLOPs = 41M. Wider but lower resolution — less capacity than w0.5-r144.

Stage 1 selects w0.5-r144 (or w0.4-r144, depending on exact Flash budget) as the optimal search space for this MCU. All the NAS in Stage 2 will happen within this sub-space.

Chapter 5: TinyNAS — Resource-Constrained Search

With the search space optimized in Stage 1, TinyNAS Stage 2 runs the actual architecture search within that space. The goal: find the architecture that maximizes accuracy subject to explicit SRAM and Flash constraints.

Stage 2 uses one-shot NAS with weight sharing. Instead of training each candidate architecture from scratch (which would be computationally impossible), TinyNAS trains a single supernet that contains all candidate architectures as sub-networks sharing weights. The supernet covers choices in:

• Kernel size: 3×3 or 5×5 per layer
• Expansion ratio: 3 or 6 (channel expansion within inverted residual blocks)
• Number of blocks per stage: 1, 2, 3, or 4

During training, random sub-networks are sampled from the supernet at each step. All sampled sub-networks share the same weights for their shared operations. This progressive fine-tuning of many overlapping sub-networks gives each sub-network approximately trained weights without requiring separate training.

After supernet training, evaluation proceeds as follows. For any candidate architecture c = (kernel choices, expansion ratios, block counts):

This analytic cost model means we can check SRAM and Flash constraints without running the network at all — just from the architecture specification. Candidates that violate constraints are immediately rejected. The remaining candidates are ranked by their accuracy on a held-out validation set, using the shared weights.

Flash budget check with real numbers. Suppose the search finds a network with these layers (kernel 3×3, INT8):

• Conv-stem: 3×3×3×16 = 432 bytes
• 8 InvRes blocks, avg 4 layers each, avg 3×3×32×32 per layer = 9,216 bytes each → 8×4×9,216 = 294,912 bytes (~288 KB)
• Head (1×1 conv + FC): 1×1×96×10 = 960 bytes
• Total Flash ≈ 289 KB < 1 MB ✓

Chapter 6: Imbalanced Memory Distribution

Even with TinyNAS finding a better architecture, there remains a structural problem that affects all CNNs on MCUs: the imbalanced memory distribution. The first few layers consume disproportionately more SRAM than all subsequent layers combined.

Why? Early CNN layers process large spatial feature maps (high H×W) with relatively few channels. Later layers have smaller spatial maps but more channels. Because SRAM scales as H×W×C, and H×W shrinks rapidly (each stride-2 conv halves both dimensions, cutting H×W to 1/4), the early layers dominate peak SRAM.

Measured on MobileNetV2 processing a 224×224 input on an STM32F746 (320 KB SRAM):

The picture is stark: blocks 1–2 consume 8× more SRAM than the MCU's budget, while blocks 7–17 use <80 KB total. The tail of the network is not the problem. The head is.

This imbalance appears across all standard architectures, not just MobileNetV2. Any CNN with a large input image will have this property because:

For MobileNetV2 with input 224×224: H₀ = 112 after stem, C_expand = 96 after 6× expansion → SRAM = 112×112×96 ≈ 1.1 MB. No amount of pruning or quantization can change this without changing the architecture.

Chapter 7: Patch-Based Inference — MCUNetV2

MCUNetV2 (Lin et al., NeurIPS 2021) introduces a simple but powerful idea to break the early-layer SRAM bottleneck: instead of processing the entire feature map at once, process it one patch at a time.

Here is the standard inference flow for an early conv layer (call it L1) followed by another (L2):

Now consider patch-based inference with a 2×2 patch grid (4 patches):

More patches → lower peak SRAM. With 9 patches (3×3 grid): SRAM ≈ 1,372/9 ≈ 153 KB — under the 256 KB budget.

The halo overhead for N layers of 3×3 convolutions is ≈ 2N pixels on each side. For 4 stacked layers with a 2×2 patch grid on a 224×224 input: halo width = 2×4 = 8 pixels per side. Each patch is 112×112; halo adds (112+16)×(112+16) = 128×128 = 16,384 extra pixels, or about 14% overhead. Empirically, MobileNetV2 with 2×2 patches incurs only ~10% extra MACs — a small price for 4.9× lower SRAM.

MCUNetV2 solves the halo overhead problem with network redistribution: instead of running patch-based inference on a standard network, it redesigns the early-stage architecture to have smaller receptive fields, reducing the halo. The NAS search jointly optimizes architecture AND the patch inference schedule — finding networks that naturally have low-halo early stages and benefit most from patch-based inference.

Measured results on STM32F746 (320 KB SRAM), INT8:

Chapter 8: Showcase — MCUNet Memory Explorer

This showcase integrates everything from Chapters 1–7 into one interactive simulation. Configure a network and memory budget, watch the per-layer SRAM profile update live, and see how patch-based inference slashes the peak.

Use the presets to walk through the MCUNet story:

1. MobileNetV2 preset — see the massive early-layer spikes, far above the 256 KB line.
2. MCUNet preset — TinyNAS found 128px, 0.5× as the optimal space. Bars flatten significantly.
3. MCUNetV2 preset — enable 2×2 patches. The early spikes are divided by ~4. Peak drops below 256 KB.
4. Try the SRAM budget slider — find the minimum budget each preset needs to fit entirely within.

Chapter 9: Connections & Cheat Sheet

MCU Cheat Sheet

The Two-Stage TinyNAS Summary

Flash Budget Check (Code)

python
def flash_budget(layers, bitwidth=8):
    """layers: list of (k, Cin, Cout) tuples. Returns Flash in bytes."""
    bits_per_param = bitwidth
    total_params = 0
    for k, c_in, c_out in layers:
        total_params += k * k * c_in * c_out
    return total_params * (bits_per_param // 8)

def peak_sram(layer_specs, bitwidth=8):
    """layer_specs: list of (H_in, W_in, C_in, H_out, W_out, C_out).
    Returns (peak_sram_bytes, bottleneck_layer_idx)."""
    bytes_per_elem = bitwidth // 8
    peak = 0
    bottleneck = 0
    for i, (Hi, Wi, Ci, Ho, Wo, Co) in enumerate(layer_specs):
        sram = (Hi * Wi * Ci + Ho * Wo * Co) * bytes_per_elem
        if sram > peak:
            peak, bottleneck = sram, i
    return peak, bottleneck

# Example: small MCUNet-like network
layers = [
    (96, 96, 1,  48, 48, 16),   # conv0, s=2
    (48, 48, 16, 48, 48, 32),   # conv1, s=1 ← likely peak
    (48, 48, 32, 24, 24, 64),   # conv2, s=2
    (24, 24, 64, 24, 24, 64),   # conv3, s=1
    (24, 24, 64, 12, 12, 128),  # conv4, s=2
]

peak, idx = peak_sram(layers)
print(f"Peak SRAM: {peak/1024:.1f} KB at layer {idx}")
# Peak SRAM: 108.0 KB at layer 1

def patch_inference_sram(peak_layer_sram, num_patches, halo_overhead=0.12):
    """Estimate peak SRAM after patch-based inference."""
    return peak_layer_sram * (1 + halo_overhead) / num_patches

# 2x2 patches (4 patches) on MobileNetV2 peak of 1,372 KB
mbv2_peak = 1372 * 1024  # bytes
patched = patch_inference_sram(mbv2_peak, num_patches=4)
print(f"After 2x2 patches: {patched/1024:.0f} KB")
# After 2x2 patches: 385 KB — still needs 3x3 patches!

patched_3x3 = patch_inference_sram(mbv2_peak, num_patches=9)
print(f"After 3x3 patches: {patched_3x3/1024:.0f} KB")
# After 3x3 patches: 171 KB — fits in 256 KB!

What's Next: TinyEngine (Lecture 11)

MCUNet finds the right architecture via TinyNAS. TinyEngine (L11) is the companion compiler/runtime that executes that architecture efficiently on MCU hardware. TinyEngine implements:

• Operator fusion — merge conv + BN + ReLU into one pass over the data, eliminating intermediate activation storage
• In-place depth-wise convolution — reuse the input buffer for the output when the shapes match
• Im2col-free convolution — avoid the memory-intensive tensor rearrangement step standard libraries use
• Patch-based execution — schedules the patch-based inference discovered by MCUNetV2 search

Together, TinyNAS + TinyEngine form the full MCUNet co-design loop: the algorithm produces an MCU-fitting architecture, and the runtime executes it with zero wasted memory or compute.

Related Gleams

• TinyML L1 — Efficiency Metrics: activation memory, MACs, roofline — the vocabulary for everything in this lesson
• TinyML L7 — NAS I: Searching Architectures: the NAS framework (search space, strategy, estimator) that TinyNAS extends
• TinyML L8 — NAS II: Hardware-Aware & OFA: ProxylessNAS, Once-for-All — the mobile NAS techniques MCUNet builds on
• TinyML L5 — Quantization I: INT8 quantization reduces Flash usage but SRAM activation analysis holds in INT8

"We want to enable every microcontroller to run deep learning. That means not just fitting the weights — but fitting the activations, in 256 KB, with no OS, no DRAM, and no compromise on accuracy." — Song Han, MIT 6.5940