Docker Layers, the COW Filesystem, and Why Your Dockerfile Order Quietly Decides Your Build Time

The thing nobody tells you about Docker is that the build is a filesystem, not a script. Every RUN, COPY, ADD creates a new layer on top of the last one. Layers are immutable tarballs of file diffs. The image you ship is just a stack of those tarballs plus a small JSON manifest that lists them.

Once that mental model lands, every confusing thing about Docker — slow builds, mysterious cache misses, 800 MB images for a 12 MB app — stops being confusing. It just becomes filesystem math.

1. What a layer actually is

Pull any image and look at it raw:

$ docker save node:20-alpine -o node.tar
$ tar -tf node.tar | head
manifest.json
6c8...d3.json
e9b...fa/layer.tar
2a1...77/layer.tar
…

Every layer.tar is a normal tarball of the files added at that step. Not the whole filesystem — just the diff. When the image runs, the runtime stacks them via a copy-on-write (COW) overlay driver (overlay2 on Linux). Reads merge; writes go to a top scratch layer.

2. Layer caching: the rule that runs your CI bill

Docker keys every layer by:

• The exact instruction (e.g. RUN apt-get install -y curl).
• The hash of any files referenced (for COPY / ADD).
• The hash of the parent layer.

If all three match a previous build, the layer is reused — instantly, no execution. If any of them changes, that layer and every layer after it are rebuilt. This is why ordering matters more than any other Dockerfile decision.

3. The single most common mistake

This Dockerfile rebuilds the whole world on every commit:

FROM node:20-alpine
WORKDIR /app
COPY . .                # <-- changes on every commit
RUN npm install         # <-- always runs
RUN npm run build
CMD ["npm", "start"]

Reorder it like this and your CI bill drops in half:

FROM node:20-alpine
WORKDIR /app
COPY package.json package-lock.json ./   # only changes when deps change
RUN npm ci                                # cached unless deps changed
COPY . .                                  # source, fast tar
RUN npm run build
CMD ["npm", "start"]

The trick: copy what changes least, first. package.json changes ~weekly, source changes ~hourly. Putting deps before source means a typical commit reuses the install layer and only re-executes the cheap build step.

4. Why your image is 1.2 GB

Three culprits, in order of frequency:

• Build artifacts left in the final layer. apt-get install dumps /var/lib/apt/lists into the layer forever. Even if you delete it in the next RUN, the previous layer still has it. You cannot un-add a file from a layer.
• Wrong base image. node:20 is 1.1 GB. node:20-alpine is 130 MB. node:20-slim is 200 MB. Pick the smallest one your code actually runs on (alpine uses musl, which breaks some binaries).
• Source repo copied with node_modules or .git. Always have a .dockerignore. Mine starts: .git, node_modules, *.log, .env*, dist, .next.

The fix for the apt example:

RUN apt-get update \
 && apt-get install -y --no-install-recommends curl \
 && rm -rf /var/lib/apt/lists/*

All three commands in one RUN means they end up in one layer, and the cleanup actually frees the bytes.

5. The layer ceiling

Docker has a soft cap of 127 layers (overlay2 limit). I've never hit it in production, but I've seen Dockerfiles that come close — usually because someone wrote forty RUN echo "step"-style debug lines and forgot to consolidate. Treat layers as a budget. Every layer is a tax on disk, push time, and pull time.

Rules of thumb

• Order from least-changing to most-changing. Cache hits compound.
• One RUN per logical operation. Don't shard apt installs across five lines.
• Always --no-install-recommends on apt, always --no-cache-dir on pip, always npm ci not npm install.
• .dockerignore is mandatory, not optional.
• If your image is over 500 MB and your code is JavaScript, you have a bug, not a constraint.

Once layers click, Docker stops being a black box and becomes a fairly predictable filesystem-with-CI-on-top. Everything else — multi-stage builds, BuildKit, distroless — is just leverage on the same primitive.