[{"data":1,"prerenderedAt":735},["ShallowReactive",2],{"NoscriptNav_XrRK2e2e8meJ0jKVGkb5ULGQDVi3UiFQ9nupAr7Yns":3,"\u002Fideas\u002Fseparating-download-from-install-in-docker-builds":8},["Island",4],{"key":5,"result":6},"NoscriptNav_XrRK2e2e8meJ0jKVGkb5ULGQDVi3UiFQ9nupAr7Yns",{"head":7},{},{"id":9,"title":10,"authors":11,"body":13,"canonicalUrl":721,"canonicalWebsiteName":722,"category":723,"date":724,"description":725,"extension":726,"featured":727,"fullWidthLayout":727,"image":728,"imageAlt":728,"location":728,"meta":729,"metaImage":728,"navigation":730,"path":731,"seo":732,"stem":733,"venue":728,"venueUrl":728,"__hash__":734},"ideas\u002Fideas\u002Fseparating-download-from-install-in-docker-builds.md","Separating Download from Install in Docker Builds",[12],"andrew",{"type":14,"value":15,"toc":712},"minimark",[16,20,23,35,70,75,92,116,129,141,161,165,183,207,220,224,246,276,292,311,315,328,352,367,385,389,408,432,453,457,468,483,486,505,508,512,519,528,531,653,667,670,708],[17,18,19],"p",{},"Docker layer caching works best when each layer's inputs are narrow, and a layer that only depends on a lockfile can survive most builds untouched because you're usually changing application code, not dependencies. Most package managers combine downloading and installing into a single command though, so the layer that fetches from the registry also depends on source files, and any source change invalidates the layer and forces every dependency to re-download even when the lockfile is identical to last time.",[17,21,22],{},"That costs more than build time. crates.io, rubygems.org, and pypi.org all run on bandwidth donated by Fastly, and every redundant download in a Docker build is a cost someone else is volunteering to cover. npm is backed by Microsoft and Go's module proxy by Google, so they can absorb it, but for the community-funded registries it adds up. It feels instant from the developer's side, a few seconds of progress bars, so nobody thinks about the hundreds of HTTP requests firing against those services on every build where the lockfile has changed by even one line, or when you're debugging a failed install and rebuilding the same image over and over.",[17,24,25,26,30,31,34],{},"If package managers exposed a ",[27,28,29],"code",{},"download"," that populates the local cache from the lockfile and an ",[27,32,33],{},"install"," that works offline from that cache, Docker layer caching would handle the rest:",[36,37,42],"pre",{"className":38,"code":39,"language":40,"meta":41,"style":41},"language-dockerfile shiki shiki-themes github-light github-dark","COPY lockfile .\nRUN pkg download\nCOPY . .\nRUN pkg install --offline\n","dockerfile","",[27,43,44,52,58,64],{"__ignoreMap":41},[45,46,49],"span",{"class":47,"line":48},"line",1,[45,50,51],{},"COPY lockfile .\n",[45,53,55],{"class":47,"line":54},2,[45,56,57],{},"RUN pkg download\n",[45,59,61],{"class":47,"line":60},3,[45,62,63],{},"COPY . .\n",[45,65,67],{"class":47,"line":66},4,[45,68,69],{},"RUN pkg install --offline\n",[71,72,74],"h3",{"id":73},"go-mod-download","go mod download",[17,76,77,78,85,86,91],{},"Go modules shipped with Go 1.11 in August 2018, and the community figured out the Docker pattern ",[79,80,84],"a",{"href":81,"rel":82},"https:\u002F\u002Fblog.container-solutions.com\u002Ffaster-builds-in-docker-with-go-1-11",[83],"nofollow","within weeks",". It's now the canonical Go Dockerfile pattern, recommended by ",[79,87,90],{"href":88,"rel":89},"https:\u002F\u002Fdocs.docker.com\u002Fguides\u002Fgolang\u002Fbuild-images\u002F",[83],"Docker's own documentation",":",[36,93,95],{"className":38,"code":94,"language":40,"meta":41,"style":41},"COPY go.mod go.sum .\u002F\nRUN go mod download\nCOPY . .\nRUN CGO_ENABLED=0 go build -o \u002Fapp .\n",[27,96,97,102,107,111],{"__ignoreMap":41},[45,98,99],{"class":47,"line":48},[45,100,101],{},"COPY go.mod go.sum .\u002F\n",[45,103,104],{"class":47,"line":54},[45,105,106],{},"RUN go mod download\n",[45,108,109],{"class":47,"line":60},[45,110,63],{},[45,112,113],{"class":47,"line":66},[45,114,115],{},"RUN CGO_ENABLED=0 go build -o \u002Fapp .\n",[17,117,118,120,121,124,125,128],{},[27,119,74],{}," reads ",[27,122,123],{},"go.mod"," and ",[27,126,127],{},"go.sum"," and fetches everything without doing any resolution or building, and the layer caches when those two files haven't changed.",[17,130,131,132,135,136,124,138,140],{},"Before Go 1.11, ",[27,133,134],{},"GOPATH","-based dependency management didn't have a clean two-file manifest that could be separated from source code for layer caching, and the design of ",[27,137,123],{},[27,139,127],{}," as small standalone files made this Docker pattern fall out naturally once modules landed.",[17,142,143,146,147,150,151,153,154,156,157,160],{},[27,144,145],{},"go build"," can still contact the checksum database (",[27,148,149],{},"sum.golang.org",") after ",[27,152,74],{}," to verify modules not yet in ",[27,155,127],{},". Setting ",[27,158,159],{},"GOFLAGS=-mod=readonly"," after the download step prevents any network access during the build.",[71,162,164],{"id":163},"pnpm-fetch","pnpm fetch",[17,166,167,168,174,175,178,179,182],{},"pnpm is the only JavaScript package manager with a download-only command, and ",[79,169,172],{"href":170,"rel":171},"https:\u002F\u002Fpnpm.io\u002Fcli\u002Ffetch",[83],[27,173,164],{}," was designed specifically for Docker. It reads ",[27,176,177],{},"pnpm-lock.yaml"," and downloads all packages into pnpm's content-addressable store without reading ",[27,180,181],{},"package.json"," at all:",[36,184,186],{"className":38,"code":185,"language":40,"meta":41,"style":41},"COPY pnpm-lock.yaml pnpm-workspace.yaml .\u002F\nRUN pnpm fetch --prod\nCOPY . .\nRUN pnpm install -r --offline --prod\n",[27,187,188,193,198,202],{"__ignoreMap":41},[45,189,190],{"class":47,"line":48},[45,191,192],{},"COPY pnpm-lock.yaml pnpm-workspace.yaml .\u002F\n",[45,194,195],{"class":47,"line":54},[45,196,197],{},"RUN pnpm fetch --prod\n",[45,199,200],{"class":47,"line":60},[45,201,63],{},[45,203,204],{"class":47,"line":66},[45,205,206],{},"RUN pnpm install -r --offline --prod\n",[17,208,209,210,213,214,216,217,219],{},"The download layer only depends on the lockfile, and the install step uses ",[27,211,212],{},"--offline"," so it never touches the network. In monorepos this is particularly useful because you don't need to copy every workspace's ",[27,215,181],{}," before the download step, and pnpm's authors thinking about container builds when they designed the CLI is the same kind of design awareness that made ",[27,218,74],{}," standard in Go.",[71,221,223],{"id":222},"cargo-fetch","cargo fetch",[17,225,226,120,232,235,236,239,240,124,243,245],{},[79,227,230],{"href":228,"rel":229},"https:\u002F\u002Fdoc.rust-lang.org\u002Fcargo\u002Fcommands\u002Fcargo-fetch.html",[83],[27,231,223],{},[27,233,234],{},"Cargo.lock"," and downloads all crate source into the registry cache. After fetching, ",[27,237,238],{},"--frozen"," (which combines ",[27,241,242],{},"--locked",[27,244,212],{},") prevents any network access during the build:",[36,247,249],{"className":38,"code":248,"language":40,"meta":41,"style":41},"COPY Cargo.toml Cargo.lock .\u002F\nRUN mkdir src && touch src\u002Fmain.rs\nRUN cargo fetch --locked\nCOPY . .\nRUN cargo build --release --frozen\n",[27,250,251,256,261,266,270],{"__ignoreMap":41},[45,252,253],{"class":47,"line":48},[45,254,255],{},"COPY Cargo.toml Cargo.lock .\u002F\n",[45,257,258],{"class":47,"line":54},[45,259,260],{},"RUN mkdir src && touch src\u002Fmain.rs\n",[45,262,263],{"class":47,"line":60},[45,264,265],{},"RUN cargo fetch --locked\n",[45,267,268],{"class":47,"line":66},[45,269,63],{},[45,271,273],{"class":47,"line":272},5,[45,274,275],{},"RUN cargo build --release --frozen\n",[17,277,278,279,282,283,285,286,291],{},"The dummy ",[27,280,281],{},"src\u002Fmain.rs"," is needed because ",[27,284,223],{}," requires a valid project structure even though it's only reading the lockfile, and there's been an ",[79,287,290],{"href":288,"rel":289},"https:\u002F\u002Fgithub.com\u002Frust-lang\u002Fcargo\u002Fissues\u002F2644",[83],"open issue"," about removing that requirement since 2016.",[17,293,294,295,297,298,303,304,307,308,310],{},"Almost nobody uses ",[27,296,223],{}," in Dockerfiles. The Rust community skipped straight to caching compilation with ",[79,299,302],{"href":300,"rel":301},"https:\u002F\u002Fgithub.com\u002FLukeMathWalker\u002Fcargo-chef",[83],"cargo-chef",", because compiling hundreds of crates is where builds spend most of their wall-clock time and downloads feel cheap by comparison. But every ",[27,305,306],{},"cargo build"," without a prior ",[27,309,223],{}," is still hitting crates.io for every crate whenever the layer rebuilds, and Fastly is absorbing that traffic whether it takes three seconds or thirty.",[71,312,314],{"id":313},"pip-wheel","pip wheel",[17,316,317,323,324,327],{},[79,318,321],{"href":319,"rel":320},"https:\u002F\u002Fpip.pypa.io\u002Fen\u002Fstable\u002Fcli\u002Fpip_wheel\u002F",[83],[27,322,314],{}," builds wheels for all dependencies into a directory, and ",[27,325,326],{},"pip install --no-index --find-links"," installs from that directory offline:",[36,329,331],{"className":38,"code":330,"language":40,"meta":41,"style":41},"COPY requirements.txt .\nRUN pip wheel -r requirements.txt -w \u002Ftmp\u002Fwheels\nCOPY . .\nRUN pip install --no-index --find-links \u002Ftmp\u002Fwheels -r requirements.txt\n",[27,332,333,338,343,347],{"__ignoreMap":41},[45,334,335],{"class":47,"line":48},[45,336,337],{},"COPY requirements.txt .\n",[45,339,340],{"class":47,"line":54},[45,341,342],{},"RUN pip wheel -r requirements.txt -w \u002Ftmp\u002Fwheels\n",[45,344,345],{"class":47,"line":60},[45,346,63],{},[45,348,349],{"class":47,"line":66},[45,350,351],{},"RUN pip install --no-index --find-links \u002Ftmp\u002Fwheels -r requirements.txt\n",[17,353,354,357,358,363,364,366],{},[27,355,356],{},"pip download"," also exists but has a ",[79,359,362],{"href":360,"rel":361},"https:\u002F\u002Fgithub.com\u002Fpypa\u002Fpip\u002Fissues\u002F7863",[83],"known bug"," where build dependencies like setuptools aren't included, so packages that ship as source distributions can fail during the offline install. ",[27,365,314],{}," avoids this by compiling everything into wheels up front, so the install step never needs a build backend.",[17,368,369,370,374,375,380,381,384],{},"Neither Poetry nor uv have download-only commands. Poetry has had an ",[79,371,290],{"href":372,"rel":373},"https:\u002F\u002Fgithub.com\u002Fpython-poetry\u002Fpoetry\u002Fissues\u002F2184",[83]," since 2020, and uv has ",[79,376,379],{"href":377,"rel":378},"https:\u002F\u002Fgithub.com\u002Fastral-sh\u002Fuv\u002Fissues\u002F3163",[83],"one"," with over a hundred upvotes. Both suggest exporting to ",[27,382,383],{},"requirements.txt"," and falling back to pip.",[71,386,388],{"id":387},"bundle-cache","bundle cache",[17,390,391,392,395,396,399,400,403,404,407],{},"Bundler has ",[27,393,394],{},"bundle cache --no-install",", which fetches ",[27,397,398],{},".gem"," files into ",[27,401,402],{},"vendor\u002Fcache"," without installing them, and ",[27,405,406],{},"bundle install --local"," installs from that cache without hitting the network:",[36,409,411],{"className":38,"code":410,"language":40,"meta":41,"style":41},"COPY Gemfile Gemfile.lock .\u002F\nRUN bundle cache --no-install\nCOPY . .\nRUN bundle install --local\n",[27,412,413,418,423,427],{"__ignoreMap":41},[45,414,415],{"class":47,"line":48},[45,416,417],{},"COPY Gemfile Gemfile.lock .\u002F\n",[45,419,420],{"class":47,"line":54},[45,421,422],{},"RUN bundle cache --no-install\n",[45,424,425],{"class":47,"line":60},[45,426,63],{},[45,428,429],{"class":47,"line":66},[45,430,431],{},"RUN bundle install --local\n",[17,433,434,435,440,441,444,445,448,449,452],{},"In practice this has enough rough edges that it rarely gets used in Dockerfiles. Git-sourced gems ",[79,436,439],{"href":437,"rel":438},"https:\u002F\u002Fgithub.com\u002Fruby\u002Frubygems\u002Fissues\u002F6499",[83],"still try to reach the remote"," even with ",[27,442,443],{},"--local",", and platform-specific gems need ",[27,446,447],{},"--all-platforms"," plus ",[27,450,451],{},"bundle lock --add-platform"," to work across macOS development and Linux containers. The command was designed for vendoring gems into your repository rather than for Docker layer caching.",[71,454,456],{"id":455},"npm-and-yarn","npm and yarn",[17,458,459,460,463,464,467],{},"npm has no download-only command. ",[27,461,462],{},"npm ci"," reads the lockfile and skips resolution, but downloads and installs as one atomic operation with no way to separate them, and there's no ",[27,465,466],{},"--download-only"," flag or RFC proposing one.",[17,469,470,471,476,477,482],{},"Yarn Classic has an offline mirror that saves tarballs as a side effect of install, but no standalone download command. Yarn Berry has no fetch command either, despite ",[79,472,475],{"href":473,"rel":474},"https:\u002F\u002Fgithub.com\u002Fyarnpkg\u002Fberry\u002Fissues\u002F4529",[83],"multiple"," ",[79,478,481],{"href":479,"rel":480},"https:\u002F\u002Fgithub.com\u002Fyarnpkg\u002Fberry\u002Fissues\u002F5998",[83],"open"," issues requesting one.",[17,484,485],{},"The standard JavaScript Docker pattern is still:",[36,487,489],{"className":38,"code":488,"language":40,"meta":41,"style":41},"COPY package.json package-lock.json .\u002F\nRUN npm ci\nCOPY . .\n",[27,490,491,496,501],{"__ignoreMap":41},[45,492,493],{"class":47,"line":48},[45,494,495],{},"COPY package.json package-lock.json .\u002F\n",[45,497,498],{"class":47,"line":54},[45,499,500],{},"RUN npm ci\n",[45,502,503],{"class":47,"line":60},[45,504,63],{},[17,506,507],{},"When the lockfile hasn't changed the layer caches and nothing gets downloaded, but when it has changed every package re-downloads from the registry, and pnpm is the only JavaScript package manager where you can avoid that.",[71,509,511],{"id":510},"buildkit-cache-mounts","BuildKit cache mounts",[17,513,514,515,518],{},"Docker BuildKit has ",[27,516,517],{},"--mount=type=cache",", which persists a cache directory across builds so package managers can reuse previously downloaded packages even when the layer invalidates:",[36,520,522],{"className":38,"code":521,"language":40,"meta":41,"style":41},"RUN --mount=type=cache,target=\u002Froot\u002F.npm npm ci\n",[27,523,524],{"__ignoreMap":41},[45,525,526],{"class":47,"line":48},[45,527,521],{},[17,529,530],{},"Cache mounts solve the problem from the wrong end. The package manager has the lockfile and knows the cache format, but Docker doesn't know any of that, which is why the Dockerfile author has to specify internal cache paths that vary between tools and sometimes between versions of the same tool. Not every build system supports BuildKit cache mounts either, and not every CI environment preserves them between builds, so a download command in the package manager itself would be more broadly useful.",[532,533,534,556],"table",{},[535,536,537],"thead",{},[538,539,540,544,547,550,553],"tr",{},[541,542,543],"th",{},"Registry",[541,545,546],{},"Funding",[541,548,549],{},"Download command",[541,551,552],{},"Offline install",[541,554,555],{},"Used in practice?",[557,558,559,578,598,617,636],"tbody",{},[538,560,561,565,568,572,575],{},[562,563,564],"td",{},"Go module proxy",[562,566,567],{},"Google",[562,569,570],{},[27,571,74],{},[562,573,574],{},"implicit",[562,576,577],{},"Yes, canonical",[538,579,580,583,586,591,595],{},[562,581,582],{},"npm registry",[562,584,585],{},"Microsoft",[562,587,588,590],{},[27,589,164],{}," (pnpm only; npm and yarn have nothing)",[562,592,593],{},[27,594,212],{},[562,596,597],{},"pnpm yes, others no",[538,599,600,603,606,610,614],{},[562,601,602],{},"crates.io",[562,604,605],{},"Fastly (donated)",[562,607,608],{},[27,609,223],{},[562,611,612],{},[27,613,238],{},[562,615,616],{},"Rarely",[538,618,619,622,624,629,634],{},[562,620,621],{},"PyPI",[562,623,605],{},[562,625,626,628],{},[27,627,314],{}," (pip only)",[562,630,631],{},[27,632,633],{},"--no-index --find-links",[562,635,616],{},[538,637,638,641,643,647,651],{},[562,639,640],{},"rubygems.org",[562,642,605],{},[562,644,645],{},[27,646,394],{},[562,648,649],{},[27,650,443],{},[562,652,616],{},[17,654,655,656,659,660,659,663,666],{},"Most package managers were designed around a persistent local cache on a developer's laptop, ",[27,657,658],{},"~\u002F.cache"," or ",[27,661,662],{},"~\u002F.gem",[27,664,665],{},"~\u002F.npm",", that warms up over time and stays warm. Ephemeral build environments start clean every time, and Docker layers are the only caching mechanism available, which means the network-dependent part of a build needs to be isolated from the rest for caching to work.",[17,668,669],{},"Opportunities:",[671,672,673,685,696,702],"ul",{},[674,675,676,677,680,681,684],"li",{},"npm could add an ",[27,678,679],{},"npm fetch"," that reads ",[27,682,683],{},"package-lock.json"," and populates the cache without installing",[674,686,687,688,691,692,695],{},"Poetry has had an ",[79,689,290],{"href":372,"rel":690},[83]," requesting a download command since 2020, and uv has ",[79,693,379],{"href":377,"rel":694},[83]," with strong community interest",[674,697,698,699,701],{},"Bundler's ",[27,700,394],{}," would work if it handled git gems and cross-platform builds more reliably",[674,703,704,705,707],{},"Cargo's ",[27,706,223],{}," shouldn't need a dummy source file to run a command that only reads the lockfile",[709,710,711],"style",{},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":41,"searchDepth":54,"depth":54,"links":713},[714,715,716,717,718,719,720],{"id":73,"depth":60,"text":74},{"id":163,"depth":60,"text":164},{"id":222,"depth":60,"text":223},{"id":313,"depth":60,"text":314},{"id":387,"depth":60,"text":388},{"id":455,"depth":60,"text":456},{"id":510,"depth":60,"text":511},"https:\u002F\u002Fnesbitt.io\u002F2026\u002F02\u002F15\u002Fseparating-download-from-install-in-docker-builds","nesbitt.io","package-management","2026-02-15","Most package managers could separate download from install for better Docker layer caching.","md",false,null,{},true,"\u002Fideas\u002Fseparating-download-from-install-in-docker-builds",{"title":10,"description":725},"ideas\u002Fseparating-download-from-install-in-docker-builds","lJk-VriAKSFJW8DdwnepkA0NPylri9DmwboFmHoj_lQ",1780596104478]