0062. Workspace export: lazy materialisation without a node driver

• Status: Accepted
• Date: 2026-07-28
• Deciders: thulasi.ram (architect)

Amended 2026-08-03 by 0066 — the lazy feed is CANCELLED on measurement, and the machinery built for it is being deleted. Read this before reading anything below. This ADR’s entire purpose is to deliver laziness — a Step transferring what it reads rather than what it inherits — without a node driver. git-bug 4ce7f2c measured the leg laziness was buying, on the real workspaced::router over the real WorkspaceClient, and the premise inverted:

leg	per file	at 50k files
feed (Cas::materialize — the exact call scarab-wsfetch makes)	0.54 ms	~27 s
cold drain	2.68 ms	~134 s

The feed is ~10× cheaper than 0061 s0 measured, and the drain is now 5× the feed — so the leg this ADR optimises is the smaller one. Worse for the case: of the feed’s 1076 ms (2000 files), only 434 ms is blob GETs; 55% is local filesystem writes, which a lazy mount pays anyway on first read. Laziness could at best defer a minority share of an already-cheap leg, in exchange for a privileged mount, a per-node prerequisite and an NFS server. The eager scarab-wsfetch init container stays.

Being deleted (0066 point 10): crates/scarab-server/src/export.rs, farm.rs, and the Export half of settle.rs — ~10k lines with no control-plane or executor caller. Dormant is not free: the live /v1/exports* routes are attack surface (any fence token could DELETE another fence’s Export until they were Browse-gated), every refactor pays to keep it compiling, and it invites someone to wire it up without reading why it was shelved.

Three carve-outs. (a) The fence-residue sweep is extracted from sweep_exports_once first — the ledger and drain-record TTLs live inside it and are load-bearing. (b) changeset.rs and its test tier stay: it is the only code anywhere that reads an overlayfs upper layer, it was finally executed 2026-08-02, and it is exactly what a revived laziness effort would want. (c) The reflink primitive (reflink-copy) stays — a future local cache wants it.

What survives of this ADR: the substrate facts — the measured kernel/PSA/overlayfs/NFS findings, including 20e8786’s ganesha semantics — and the change-set tier. They are why the revival path is a known quantity rather than a fresh investigation.

The named revival candidate is JuiceFS plus its CSI driver — not a rebuild of the ganesha path. Triggers, so this is not re-litigated on taste: a read-fraction measurement showing Steps touch a small share of their workspace, or workspaces growing until 27 s matters. It needs the privileged CSI DaemonSet this ADR already priced out, so the cost is known.

Context

0061 decided a four-part workspace data path. Part 2 was a Scarab node driver, shipped and installed as standard, which would mount a Workspace Snapshot as a read-only lower layer with a pod-local writable upper layer, materialising lazily so a Step transfers what it reads rather than what it inherits. Part 2 is the part 0061’s own measurement section identifies as load-bearing:

Consequence for sequencing. Part 2 (lazy materialisation) is the load-bearing part, not part 3.

A deployment constraint arrived after 0061 was accepted and it forbids the mechanism:

No DaemonSet from Scarab at all — privileged or not. The only nodes Scarab may touch are the node its server runs on and the nodes its Step Pods run on.

The constraint protects against a cluster-wide footprint that Scarab installs and versions: something present on every node, outliving every Run, that a platform team must accept, upgrade and trust. It is not a blanket prohibition on privilege, and it is not a prohibition on touching a node a Step Pod is already running on.

Amended 2026-08-02 — the rule is the rationale, and stating it as “no DaemonSet” hid an option. The sentence above is the constraint; “Scarab installs and versions” is what makes it one. A DaemonSet an operator chooses, installs and upgrades on its own schedule is outside it for the same reason a lazy snapshotter is (the third row of the table below) — it is someone else’s node change. The concrete consequence: csi-driver-nfs is a live alternative to nfs-utils as the per-node prerequisite of part 2, not something this rule excludes. Its node plugin ships its own NFS client and mounts in the host mount namespace, so it makes the delivery path work on nodes whose image has no mount.nfs at all (Bottlerocket, hardened AMIs — exactly the population red-team #1 names as the risk). The choice an operator faces is therefore “a node package Scarab cannot install” vs “an upstream chart Scarab does not version”, and this ADR deliberately does not make it for them.

The trap, recorded so whoever picks it up does not discover it in a benchmark. csi-driver-nfs declares a real fsGroupPolicy, which invites kubelet to recursively chown the whole mount on Pod start — walking every inherited path and thereby destroying exactly the laziness the Export exists to buy (and defeating the caching argument below). Using it requires fsGroupChangePolicy: OnRootMismatch, or no fsGroup at all. This is the same fact as red-team #4 seen from the other side: the in-tree NFS plugin’s Managed: false is why fsGroup does not apply there, and a CSI driver that manages it is not a free upgrade.

0061 did not test part 2 against this, because it asserted the problem away:

…it is the only mechanism that does — intercepting reads requires a privileged mount, which an unprivileged Pod cannot arrange for itself.

Both halves of that sentence are true and the conclusion drawn from them is not, because the Pod was never going to be the thing arranging the mount. In Kubernetes the privileged mounter is kubelet: a Pod declares a volume and kubelet performs the mount on its behalf. So an entirely unprivileged Pod can have an exotic filesystem at /workspace without holding any capability at all. 0061’s ten “Alternatives considered” are all about transport and topology and none of them asks where the mount could come from instead.

The complete space, which is what makes this decidable

On a cold Pod, moving less than the whole workspace requires either interception or foreknowledge. There is no third source of laziness. Foreknowledge means inferring a Step’s read set, which fails silently when wrong (a missing file makes a test skip, not fail). That leaves interception, and interception has exactly three possible owners:

owner	verdict
a privileged mount inside the Step Pod (FUSE, fuse-overlayfs)	Dead. Measured: rejected by PodSecurity at both baseline and restricted.
kubelet, mounting a network filesystem	This ADR. kubelet holds the privilege; the Pod holds none.
the container runtime (lazy snapshotters — stargz, nydus)	Not ours to install or version; a platform team adopts it on its own schedule.

That table is the decision. Everything below is the consequence of picking row two.

Measured facts

All on colima, k3s v1.35.0+k3s1, runtime docker://29.5.2, node filesystem ext4. Recorded because several of them contradict what this design was expected to need.

Mount propagation. A mount made in one container can reach a sibling:

unprivileged sidecar + mountPropagation: Bidirectional
      → Forbidden: Bidirectional mount propagation is available only to privileged containers
privileged sidecar + Bidirectional, alongside an unprivileged `restricted`-baseline step container
      → valid Pod (server dry-run)

and the mount stacks directly on the shared volume’s own root, which is what keeps /workspace at /workspace. From the step container’s /proc/self/mountinfo:

1187 1179 0:153 / /wsa rw,relatime master:498 - tmpfs tmpfs rw,inode64     # stacked ON the volume root
1188 1179 253:1 …/empty-dir/wsb /wsb … ext4                                 # plain emptyDir, for contrast
1189 1188 0:154 / /wsb/nested … tmpfs                                       # nested one level under

Nesting would have forced the workspace to move to a deeper path and every pipeline with a hardcoded /workspace would have paid for it. It does not.

Unmount does not propagate back. When the sidecar unmounted, the step kept the mount and merely lost its master: link — contents still readable. So a daemon that dies mid-Step leaves a live mountpoint that fails every operation. This is designed for below, not discovered later.

PodSecurity is what kills the in-Pod mount, and it kills it at baseline:

Pod shape	enforce=baseline	enforce=restricted
plain emptyDir	allowed	allowed
privileged sidecar + Bidirectional	denied	denied
inline nfs: volume	allowed	denied — restricted volume types
persistentVolumeClaim	allowed	allowed
image: volume	allowed	allowed
hostPath	denied	denied
inline csi: volume	allowed	allowed
generic ephemeral (inline PVC template)	allowed	allowed

Amended 2026-08-02 — three rows the first version of this table did not have, and one of them changes what part 2’s delivery may be. restricted’s volume-type allow-list is exactly configMap, csi, downwardAPI, emptyDir, ephemeral, persistentVolumeClaim, projected, secret. So csi: and generic ephemeral are PSA-legal at restricted too — the table as written implied a PVC was the only legal envelope for an exotic volume, and it is not; it is one of three. That matters because it is what makes csi-driver-nfs (amendment above) and an inline per-Step ephemeral volume claim reachable without weakening the posture. And hostPath is denied at baseline, not merely at restricted — it is one of baseline’s own prohibitions, which is why the LocalPath delivery below reaches a production cluster as a local PV and only ever as hostPath in dev.

Two things follow. baseline — not restricted — is the level orgs actually set as a cluster default, because restricted breaks too much; so the population that cannot run a privileged sidecar is every enforcing cluster, not a minority of them. And a PVC is the PSA-legal envelope for a volume type that is otherwise forbidden, because persistentVolumeClaim is on restricted’s allow-list and the PV behind it is an admin-mediated object. Inline nfs: is denied at restricted; the same NFS wearing a PVC is allowed.

There is a mechanical escape from PSA — it is per-namespace, a namespace label overrides the cluster default, and Scarab owns its step namespace — and it is rejected. It would make the default data path depend on being permitted to weaken the posture of our own namespace, which any org that takes PSA seriously blocks with Kyverno or Gatekeeper. A default path resting on that is not a path.

overlayfs on ext4 protects its lower layer — but read the next block before concluding anything about hardlinks from this one. One privileged pod, directories on a real ext4 volume (overlayfs cannot be stacked on the container rootfs, which is itself overlayfs):

fs type            : ext2/ext3                    # ext4 family — no reflink
hardlink farm      : blob link count = 2          # free, zero bytes copied
plain overlay (lowerdir = the farm)   : MOUNTED
  step writes through it →
    merged sees      : step wrote this
    CAS blob now     : original content           # UNCHANGED — the store cannot be corrupted
    upper holds      : main.rs                    # the EXACT change set
overlay + index=on,nfs_export=on      : MOUNTED   # exportable over NFS

A hardlink cannot carry per-snapshot metadata, and that is the real constraint on the Farm. This was measured after the paragraph above was first written, and it corrects it:

set the snapshot's recorded mtime on the FARM entry → CAS blob mtime moved too
chmod 755 the farm entry                            → CAS blob mode changed too
reflink (APFS clonefile) instead of hardlink        → CAS blob metadata independent

A hardlink is a second name for one inode, and mode and mtime live on the inode. 0061 s7 made mode/mtime fidelity a pinned contract (crates/scarab-storage-s3/tests/fidelity.rs) after measuring that dropping it silently degraded cross-Step incremental compilation, and s8 rejected letting a first-writer’s timestamp win for a shared blob. So restoring a snapshot’s metadata onto a hardlinked farm entry would mutate the CAS blob — the precise corruption this design claims to prevent — and two snapshots sharing content but not timestamps would fight over it.

Therefore the Farm materialises by reflink where the filesystem offers it and by plain copy where it does not; hardlinks are not the mechanism. The earlier claim that “reflink is a bonus, not a dependency” is withdrawn. What survives is the part that matters: even the copy rung is a local copy at disk speed, against 4–6 ms per file of network round-trip in the measured status quo — for 50 000 files that is seconds versus minutes. The ordering argument is untouched; only the size of the win moves, and only on filesystems without reflink.

The verified overlayfs result above still stands on its own terms — copy-up does not modify the lower layer, so a Farm built by any rung is safe against a Step writing through it. That was what the probe tested; it did not test metadata, and saying so is the point.

This inverts one warning and it is worth keeping the corrected form: the local dogfood disk is ext4, which has no reflink, so the dogfood exercises the copy rung and the reflink rung is the one it cannot reach. Any benchmark must therefore state which rung it took.

The governing principle, unchanged

0061’s principle still decides the arguable calls:

Minimise the substrate idiosyncrasies an author must know.

/workspace stays at /workspace, holds the same bytes, and a Step cannot tell which of the configurations below produced it. Where the substrate is expensive, the system pays.

Decision

Five parts.

1. A Snapshot Farm — the CAS given tree shape, without leaving the disk. The workspace service materialises a Workspace Snapshot into a directory tree on its own disk whose entries are reflink clones of CAS blobs where the filesystem supports it, and local copies where it does not. Either way no round-trip is made and nothing crosses a network. A Farm is immutable and read-only, keyed by snapshot root, built once and shared by every Step that inherits that snapshot — so fan-out costs nothing. The Farm is a warm-tier cache object: bounded by space, LRU-evicted, and a miss is slower and never wrong.

Not hardlinks: see the measurement above — a hardlink shares its inode, so restoring a snapshot’s mode and mtime onto a farm entry mutates the CAS blob, and 0061 s7 makes that fidelity a pinned contract.

This is where the per-file cost goes to die, and the reason is the CAS, not the transport: the store already holds exactly one copy of each content on the same filesystem as the Farm, so building a tree never has to ask the network for a byte. 0061 measured the CAS legs at 4–6 ms per file against loopback object storage. A reflink is a local metadata operation in the tens of microseconds; a local copy is disk bandwidth. For a 50 000-file node_modules or target/ that is about a second on reflink, a few seconds on a plain copy, and minutes today.

2. A Workspace Export — the Step’s Workspace, mounted by kubelet. Per Step, the service mounts overlayfs with the Farm as lowerdir and a per-Step directory as upperdir, and exports the merged view. The Step Pod receives it as a PersistentVolumeClaim bound to a per-Step PersistentVolume with an NFS source, mounted at /workspace.

The Step Pod holds no privilege, no capability, and no PSA exception. It does not fetch, and there is nothing to copy back. Reads fault over the network only for what the Step actually opens — which is the laziness part 2 of 0061 wanted, obtained from kubelet instead of from a driver. Delivery as a PVC rather than an inline nfs: volume is load-bearing, not stylistic: the table above shows inline nfs: is denied at restricted and the PVC is not.

The PV must pin mountOptions: [nfsvers=4.2], and that is a correctness requirement rather than a tuning choice (probed 2026-08-02 on this machine). A kind node — kindest/node, which just up runs Steps on — ships /sbin/mount.nfs, mount.nfs4 and rpc.statd, but has no systemd to run rpc.statd, so a mount that defaults to v3 fails with “rpc.statd is not running”. -o vers=4 and an unqualified mount that negotiates 4.2 both succeed. An unpinned PV therefore works or does not depending on what the client’s default happens to be on that node — the substrate idiosyncrasy the governing principle exists to keep off an author’s desk. (nolock is the alternative for a deployment that must stay on v3; the pin is the default.)

Amended 2026-08-02 — delivery is a seam with three arms, and NFS is only one of them. Part 2 above reads as if “the Export” and “NFS” were the same decision. They are not, and separating them is what makes the correctness half testable before the transport half exists. Export prepare returns a Delivery, and the executor’s single flagged volume-kind line (crates/scarab-executor-k8s/src/lib.rs, the “THE one line that chooses the volume kind” comment) is its one consumer:

• Eager — today’s emptyDir plus the scarab-wsfetch copy. Unchanged, and the fallback the ladder already names.
• LocalPath — the Depot’s overlay merged/ made visible in the host mount namespace (the Depot’s data mount carries CAP_SYS_ADMIN and Bidirectional propagation) and handed to a co-located Step as a local PV — PSA-legal at restricted, unlike hostPath, which the amended table above shows is denied at baseline. No network, no NFS, no server. The Step is pinned to the Depot’s node by affinity, which is precisely why this is a dev/single-node posture and not the production default: it caps a Run’s throughput at one node and it writes a placement constraint onto every Step, which is the fight with 0055 part 4 already refuses to pick.
• Nfs — part 2 as written, once there is a server.

A deployment that asks for a delivery its topology cannot serve refuses at launch, in the same fail-closed shape the cutover knob already required — never a silent fall back to Eager.

LocalPath cannot exist in proc mode: the Depot there is a darwin host process and darwin has no overlayfs. It is a just local-helm capability only, and any recipe or test that claims otherwise is claiming something the substrate cannot do.

The Export mount carries redirect_dir=on, and that is a correctness requirement rather than a tuning choice. Without it, rename(2) of a directory that exists only in the lower layer returns EXDEV — measured: rename failed: Cross-device link, with the module default redirect_dir=N. Directory renames are not exotic; git, cargo, npm, pip and maven all do them, and today they work because /workspace is a plain emptyDir. An Export that broke them would hand authors “Invalid cross-device link” as substrate knowledge, which the governing principle forbids. Worse, mv masks it by recursively copying the subtree, so the failure mode is not an error but a silent full copy of an inherited tree — which then lands in the upper layer and makes “the change set is the upper layer, exactly” re-ingest a tree nothing changed. Verified that redirect_dir=on coexists with nfs_export=on — which metacopy does not, that being the actual reason metacopy is unavailable to an Export — so the fix is available.

Amended 2026-08-02 (stage-2a substrate spike) — the mount is no longer a probe’s mount carried into production by inference; it is the one the code writes, and the option string is load-bearing. crates/scarab-server/src/export.rs:501-508’s exact option string mounted first try from a privileged Pod on the colima node — Ubuntu 24.04.4, kernel 6.8.0-117-generic, k3s v1.35.0, docker runtime, hostPath on ext4 — with the kernel echoing back +uuid=on,nouserxattr. On that node the overlay module parameters redirect_dir, index and nfs_export are all default N, so every behaviour parts 2 and 3 depend on comes from that string and from nothing else: a delivery path that drops, reorders or re-derives it returns EXDEV on the first mv of an inherited directory, silently masked into a full subtree copy by mv. index=on,nfs_export=on also mounted cleanly (an actual NFS export was not exercised — 2a does not need one).

This forces part 3’s reader to support trusted.overlay.redirect, not refuse it. A directory rename under redirect_dir=on records the old path in that xattr, so a change-set reader that treats redirect as unsupported would refuse every renamed directory. The two halves have to agree, and the ADR previously implied they could each be decided alone — it even said in part 3 that “only directory renames need refusing” while part 2 required supporting them, which is the shape of mistake that ships as a defect rather than a disagreement.

3. The change set is the upper layer, exactly. An overlayfs upper directory contains precisely the paths the Step touched, put there by the kernel. The drain reads the upper, hashes only those files, folds them into the CAS and returns a new snapshot root — all on the service’s own local disk, with no network in the path.

“Hashes only those files” is true of the snapshot root and false of the content identity, and that was measured by building the fold rather than reasoned about here. The two coordinates are not symmetric: a root is an address, so an untouched subtree can be carried forward by naming its hash and nothing needs reading. A content identity is not an address — nothing is stored under one and nothing can be fetched by one (see CONTEXT.md §4.2) — so a subtree carried by hash has a known root and an unknown identity, and every rebuilt directory level must walk its untouched siblings to compute theirs: one tree_entries per directory, recursively.

The consequence is shaped opposite to intuition and is worth stating plainly, because it inverts which edit is cheap: a change deep in the tree costs little and a change near the root costs an identity walk of most of the parent. Editing README.md at the workspace root is the expensive case, not editing one file under target/.

This is local work against the warm tier, so “no network in the path” stands unharmed — but it is work, and an ADR whose entire argument is about per-file cost should not have left it unbooked. The fold memoises within one settle, which bounds it to one walk per directory rather than one per reference; that is a constant factor, not a structural fix. The structural fix is a persisted snapshot root → content identity index in the warm tier, which is filed rather than decided here.

Reading a change set must fail closed on missing privilege, and the reason is nastier than it looks. An upper layer records deletions as whiteouts (a character device with rdev == 0) and wholesale directory replacements as the trusted.overlay.opaque xattr. trusted.* xattrs are readable only with CAP_SYS_ADMIN — and the kernel answers an unprivileged read with ENODATA, which is indistinguishable from “the attribute is not set” (measured while implementing s2: kernel 6.8.0, an unprivileged reader saw nothing on an upper that demonstrably carried the markers). So a drain that merely tried to read them would report “no opaque directories, no renames”, be believed, and publish a snapshot missing every deletion. The reader therefore checks its own effective capabilities up front and refuses rather than answering. A silent “nothing changed” is the worst available failure and it is one syscall away from being the default.

Two further measured refinements, because they decide what has to be supported rather than merely tolerated. A file rename is exact — it appears as a whiteout plus a full copy, with no redirect — so redirect is a directory phenomenon only, and a redirect on anything else is a shape we do not understand and must refuse. And redirect_dir=on is honoured when passed at mount time even though the module default is off, so a directory rename really does reach the drain in production. Copy-up also stamps origin on files and impure on their parent directories, which must be tolerated as bookkeeping — treat them as unsupported markers and every real drain fails on its first edited file.

So a change set is not quite “the upper layer”. It is the upper layer plus a graft instruction, and the graft is the one thing in it that means nothing on its own: a renamed directory’s redirect names a path in the parent snapshot, not in the upper. Three consequences the drain must honour, each of which fails silently if it does not:

• Resolve grafts before deletions. A rename also whiteouts the old path, so a drain that applied deletions first would delete the graft’s own source.
• Every other path is a merged-view path, including the renamed directory’s own — its upper name is its new name. Only redirect is in parent-snapshot coordinates, and mixing the two silently grafts the wrong subtree.
• Redirects compose. A relative redirect under an already-renamed ancestor resolves against that ancestor’s lower coordinate, not its merged one; resolving against the merged parent grafts nothing and loses a subtree without erroring. A leading / is the kernel’s encoding for overlay-root-relative, which is how a cross-parent rename is recorded — so an absolute redirect must be resolved, not rejected, or the design refuses precisely the renames part 2 requires it to support.

Durability is not local, and the cold upload stays on the critical path. The paragraph above says the fold happens “with no network in the path”, which is true of the fold and was allowed to imply something false about the Attempt. The service’s own disk is the warm tier, and 0061 part 4 requires a Workspace Snapshot to be durable before an Attempt is Succeeded, while 0061’s retention table says warm is bounded by space, evicts, and promises nothing. 0061 also rejected exactly the shortcut this wording invited: “writing warm-first and letting the service tier onward would make the warm tier load-bearing for durability, which part 4 forbids.”

Decided: 0061 part 4 stands verbatim. A change set is folded into the CAS locally and then uploaded to cold before the Attempt may reach Succeeded. A Step that exits 0 whose service dies immediately afterwards must not leave a green Attempt whose evidence is gone, and warm cannot promise otherwise — so a green Attempt always has its snapshot in the tier that makes a promise.

The reason this is affordable, and it is the whole argument: what gets uploaded is now only the change set. Today’s drain ships the entire workspace over the exec tar tunnel whether it changed or not. So an Export that folds locally and then uploads what the Step actually wrote moves strictly less than the status quo, while keeping a durability guarantee the status quo also has. The cold leg is a real cost and it is booked here rather than discovered in a benchmark: it is one upload per Step, sized by the change set, on the critical path to Succeeded.

Two consequences worth naming. The upload cannot be skipped for content cold already holds without an existence primitive on the ObjectStore port, which is a filed gap — so the first implementation pays for content cold may already have. And the laziness this ADR buys is on the read side; the write side is bounded by the change set, not by interception, which is why part 5 exists.

This is not an optimisation of the drain, it is a different kind of answer. 0061’s deferred overlay-diff drain (git-bug 66fc0e3) wanted this and believed it needed a privileged mount on the node. A stat cache — comparing each file’s (size, mtime) against the input manifest, git’s index-cache trick — was the approved unprivileged approximation, and its failure mode is silently publishing a stale hash on an mtime race. The upper layer makes the change set exact, so that failure mode does not exist rather than being defended against. The stat cache survives as the drain for configurations without an Export (part 5’s ladder, and the local executor), where it is the correct fallback and not the mechanism.

Stage 1 landed 2026-08-01, and its trusted egress helper SHARES the wsfetch image. 0061’s s3-drain put the drain in-Pod ahead of any Export: every workspace Step Pod’s egress barrier is now the scarab-wsfetch image (hold + the exec’d drain), one knob with the fetcher — chart workspace.fetcherImage / SCARAB_WSFETCH_IMAGE — not a second image to version. When stage 2 puts the change set on the service side, the helper’s drain is the thing that stops walking /workspace and starts reading the upper; the container, its token, its record rendezvous and its exit-code contract are already in place and do not move.

4. Zone preference is soft, and a miss is a cost line rather than an outage. A Workspace Export lives on one service replica’s disk, and the volume must be named in the Pod spec before the scheduler places anything. Scarab picks the replica at launch, prepares the Export there, and expresses a preferred (never required) zone affinity on the Pod. If the scheduler honours it the mount is in-zone; if capacity says otherwise the Pod mounts cross-AZ — slower, metered, never wrong.

Hard-pinning is rejected: it hands the scheduler a zone constraint on every Step, which fights the spot capacity 0061 names as the operating environment, and it silently amends the placement overlays 0055 makes the operator’s to own. When a snapshot is warm nowhere, every zone is equally good and the preference has nothing to say.

5. An optional node-side accelerator, for write-heavy Steps on fast local disk. Writes in part 2 cross the network. Where a cluster permits privilege and its nodes have fast local storage, a Step may instead run with the same primitive in a different place: a privileged Scarab-owned native sidecar mounts overlayfs with the Export as lowerdir (read-only) and a node-local emptyDir as upperdir, and propagates the merged view to /workspace via Bidirectional propagation. Reads still fault over the network; writes never touch it during the Step; at exit the sidecar ships the upper layer — again the exact change set — to the service.

This is opt-in and it is an accelerator, not a second architecture. Both configurations use the same Farm, the same CAS, the same snapshot roots, the same /workspace, and in both the upper layer is the change set. The only difference is where writes land in the interim.

The privilege ladder

The service needs CAP_SYS_ADMIN to mount overlayfs. That is one operator-installed StatefulSet — the exception storage operators are routinely granted, on the node the constraint explicitly permits — and it is preferred, not required. The interface is “give me a tree-shaped writable view of a snapshot on the service’s disk”, with three backends:

The two axes are independent — how the Farm is built, and whether an overlayfs sits on top of it — so they are two ladders, not one:

Farm build	needs	cost, 50k files	metadata fidelity
reflink clone per blob	XFS (reflink=1), btrfs, APFS	~1 s, no bytes	exact
plain local copy per blob	nothing	seconds, at disk bandwidth	exact
hardlink per blob	—	free	unsafe — mutates the CAS blob

Export build	needs	change set
overlayfs, Farm as lower	service pod CAP_SYS_ADMIN	exact (the upper layer)
writable copy of the Farm	nothing	stat cache — (size, mtime, ctime) approximation

Every live rung produces an identical tree, and the slowest is a local copy at disk speed instead of 50 000 network round-trips. A build must report which rung it took; a benchmark that silently drops a rung reports a number the real deployment would never produce.

Two corrections to that paragraph, both found while building the rungs.

“Identical tree” has one exception, and it is a refusal rather than a wrong tree. The copy rung reads the Farm and the overlay rung never does — it only names it as a lowerdir and lets the kernel read it. So a Farm entry whose recorded mode denies read to the service’s uid (a 0o000 file, which the fidelity contract obliges the Farm to reproduce faithfully) builds on the overlay rung and fails on the copy rung. Loud, and therefore acceptable; recorded because “identical” invited the opposite assumption.

The rung table above is missing a row, and the omission looks accidental. The Farm ladder’s top rung is reflink, and a Farm and its Exports live on one filesystem by construction — part 1 puts farms on the warm volume precisely so a clone cannot cross a filesystem boundary. So “a writable copy of the Farm” could be a CoW clone of it: nearly free, metadata-exact per entry, and correct for exactly the reason the overlay rung is correct, since copy-on-write independence is what a reflink provides. That would make the unprivileged Export rung nearly as cheap as the privileged one wherever the filesystem clones, which materially weakens the case for CAP_SYS_ADMIN being preferred. What it would not change is the change set: that stays the stat-cache approximation, which remains the real reason to prefer the overlay. Filed, not built.

And the two columns of this table must not be independently reachable. The change-set column is not a note about what the drain tends to do, it is a constraint on what it can do: an upper layer read from the copy rung reports every inherited file as written and no deletion at all, because nothing on that disk distinguishes the two. Reading it as an overlay upper therefore publishes a snapshot in which everything the Step deleted has silently returned. This shipped as a real defect — the seam carried an overlay-marker value across both rungs — and the fix was to make the wrong pairing unrepresentable in the type rather than to document against it. The lesson generalises to every row of every ladder in this ADR: if two rungs need different drains, the rung must choose the drain.

What refusing the capability actually costs. The first version of this sentence said “never correctness, and never a second code path”, which was wrong on both halves and was corrected once to say the rungs differ in correctness. That was true of the stat cache as first built and is no longer true of it, so here is the settled position.

(size, mtime) alone has a hole that is not a race but a determinism: any writer that preserves both length and timestamp — cp -p, touch -r, rsync -a, tar -xp — defeats it every time, reproducibly, and those are ordinary CI operations rather than pathological ones. A racily-clean check does not help, because nothing about it is racy.

ctime closes it, and this is why git’s index records more than size and mtime. No syscall sets ctime; utimensat bumps it as a side effect. So comparing the observed ctime against the moment materialisation completed catches every one of those writers. Verified on APFS, ext4, tmpfs and an overlayfs upper: rewrite+utimensat, cp -p, touch -r and tar -xpf all moved ctime past the capture; a pure read did not.

Two things that only measurement would have produced, both now contractual. The mtime cutoff needs a one-second slack for coarse filesystems and the ctime cutoff must not have one — materialisation finishes in the milliseconds before the capture, so a slack there puts every file inside the untrusted window and the cache degrades to hashing everything (measured: every file re-read, nothing ever reused). The asymmetry is principled: the mtime cutoff compares a recorded producer timestamp against a different clock, while the ctime cutoff compares two readings of the same clock milliseconds apart. And the capture must be stamped at least a millisecond after the last file is written, or millisecond truncation leaves materialisation’s final files inside the capture’s own tick.

So the rungs do not differ in correctness. They differ in what their correctness rests on: the upper layer is the kernel’s own record of what was touched, while the stat cache is sound only given an unforgeable ctime, a capture stamped after materialisation, a filesystem whose ctime is not coarse, and a Step that cannot move the clock backwards (ADR-0039 drops ALL capabilities, so no CAP_SYS_TIME). Those assumptions hold in every configuration Scarab ships, and they are assumptions where the exact path has none. It remains two drains, which 0061 part 1 would still count against this design; the defence is that only one of them is ever used in a cluster, and the other exists where there is no Export to read — the local executor.

The threat model is worth stating plainly, because it bounds all of this: the stat cache guards against accident, not malice. A Step that deliberately forges timestamps to poison its own change set corrupts only its own fenced evidence.

The Farm rung is per-file, not per-build. A clone can fail on an individual file while succeeding on its neighbours, so Mixed is a real outcome and the counters — how many were cloned, how many copied — are the reportable truth rather than a single label. Discovered while implementing s1; the two-rung table above reads as if a build picks one rung and keeps it, and that is a simplification.

Vocabulary

Snapshot Farm — the immutable, shared, tree-shaped view of one Workspace Snapshot on the workspace service’s disk, built by reflink where the filesystem offers it and by local copy where it does not. (Not hardlinks — see the measurement above; the first draft of this ADR said hardlinks and was wrong.) Defined here and deliberately kept out of CONTEXT.md: it is a cache implementation, not domain language, and the glossary is a glossary.

Two terms do reach the glossary, because they are distinctions a reader must not conflate:

• Workspace Export — the delivered form of a Workspace: the per-Attempt, writable, network-mounted view a Step Pod receives. Losing an Export fails that Attempt, which retries; losing a Snapshot widens a rerun’s scope. Different words, different consequences.
• Change set — the exact set of paths an Attempt wrote, known from the upper layer rather than inferred. Where there is no Export it is approximated from (size, mtime), and that is the one place in the data path where a wrong answer is possible rather than merely slow.

Workspace also loses its “Pod-local, dies with the Pod” claim, which was an implementation leak in a glossary: a Step cannot tell whether its bytes sit on its node, on the service, or across both, and the lifetime unit is the Attempt.

The fence

An Export’s identity is a capability, not an assertion. NFS authenticates with AUTH_SYS — the client asserts a uid and the server believes it — so per-Step isolation cannot come from the protocol. It comes from the export path being an unguessable 256-bit secret, TTL’d to the Step’s deadline, pinned to the first client that mounts it, and revoked when the Attempt settles. That is structurally the same capability the existing per-Pod HMAC workspace token is, delivered through a different channel; Scarab already creates a Pod and a Secret per Step, so a PV and a PVC per Step is the same order of object churn.

Consequences

• The service becomes a network filesystem server. nfs-ganesha in userspace is the trodden path. It is a real new dependency implementing a protocol with decades of edge cases, and it is the largest single risk in this ADR. It is also the part 2a does not need (see the delivery seam), and as of 2026-08-02 it is verifiable on this dev machine rather than on a box we do not have.
• A privileged workspace-service Pod is the preferred configuration. Per the ladder this is degradable, so it is not an install prerequisite — which is the material difference from 0061’s node driver, and the reason this design satisfies the constraint at all.
• /workspace does not move, measured, so no authored pipeline changes.
• Writes cross the network in the default configuration. This is the last structural cost and it is smaller than it sounds: today’s drain ships the entire workspace over the exec tar tunnel whether it changed or not, where an Export ships only the bytes the Step actually wrote, once, at write time. The risk is operation count, not byte volume — NFS charges a round-trip per file operation, and a build that creates, writes and fsyncs many small files pays latency per file that no bandwidth fixes. That is what part 5 exists for and what the spike must measure.
• Aggressive client attribute caching is safe here and is not in general. An Export has exactly one client and one writer, and its lower layer is immutable by construction, so a high actimeo cannot serve a stale answer. This is the specific lever against the many-small-files metadata cost that makes EFS bad at node_modules.
• A dead FUSE-or-NFS mount fails the Attempt, and must say so. Measured: unmount does not propagate back to the Step, so a failed daemon leaves a live mountpoint returning errors on every operation. Hard mounts plus the existing Step deadline plus durable retry is the right handler — a stall becomes a deadline-exceeded Attempt that retries — but the Attempt must name the cause rather than surface a mystery I/O error (0027: smart never means mysterious).
• Rolling the service is now a drain-then-roll operation. With per-AZ replicas, a Step mounts one replica, so an upgrade must stop accepting new Exports and wait for in-flight Steps (minutes) before rolling. Blast radius of a replica loss is the Steps mounted on it, not all of them.
• Warm-tier eviction stops being optional. Farms and Exports are directories on a bounded disk, and today the warm tier has no eviction at all (git-bug 24476bc). That ticket becomes blocking.
• Per-Step PV/PVC objects and per-Step exports are a reaping obligation. A leaked Export is a leaked directory and a leaked capability.
• 0061’s cross-AZ claim is amended. “Cross-AZ traffic is confined to the archive drain” is false once a Step can mount a replica in another zone; part 4 makes that a priced, preferred- affinity miss instead of an invariant.
• The eager fetcher becomes the no-Export fallback, not the standard route. scarab-wsfetch and the HTTP CAS path shipped on PR #93 remain the path for the local executor and for any mode without an Export. They stop being the way a Step gets its inputs in a cluster.

Alternatives considered

• FUSE (or fuse-overlayfs) inside the Step Pod, via a privileged Scarab sidecar. The closest drop-in, and the union would not even need overlayfs — a daemon we write is the union (lower = fetch-on-open, upper = pod-local, copy-up on write). Rejected on the measured PSA table: denied at baseline, which is the common cluster default. Survives only as part 5, opt-in, where the privilege buys local writes rather than laziness.

• FUSE inside the Step’s own container via 0039’s governed add_capabilities. Rejected: it hands SYS_ADMIN to the user’s container and guts 0039’s restricted baseline. Named because it is the obvious move and it is worse than a DaemonSet, not better. Mount propagation makes it unnecessary in any case — a sibling container can own the mount.

• Inline nfs: volume instead of a PVC. Simpler and denied at restricted (measured). The PVC is not ceremony.

• A warm, content-indexed pool of block PVCs handed to Step Pods — better than laziness on its best axis, since a pre-populated volume moves 0% of a tree where laziness moves 5%, and it is PSA-clean. Rejected because it re-binds volumes to short-lived Pods, which is precisely what 0061 part 1 designed away: “Every problem PVCs have here… comes from binding volumes to short-lived pods. Bind them to the long-lived thing instead and all of it goes away.” Attach is 5–20 s on EBS-class storage and detach wedges when a spot node goes; volumes are AZ-pinned, so they constrain scheduling and fight 0055; per-node attachment quotas cap concurrent Steps in the mid-20s; RWO means a fan-out needs a clone per consumer; and the write-back needs a detach-and-reattach before the volume can return to the pool. A pool genuinely fixes provisioning and population — the part 0061’s “PVC per Step” rejection did not consider — and fixes none of the rest.

• EFS (or any managed RWX filesystem) mounted directly. 0061 rejected “RWX PVC per Run” partly as “a much heavier prerequisite than the driver”. That reasoning is now backwards and is corrected here: the EFS CSI driver is a DaemonSet and therefore forbidden, while EFS reached through an NFS-sourced PV needs no driver at all. It still loses, for two reasons that are not the availability argument. It makes a managed filesystem product a prerequisite, which 0061 declines to do even for object storage (“an operator’s cost decision, not an engine decision”). And it puts the per-file cost back on the wire: EFS charges roughly a millisecond per metadata operation, where a Snapshot Farm charges tens of microseconds locally. Running our own server is what buys the Farm; a managed one cannot be given hardlinks into our CAS.

• A hardlink Farm, so the Farm is free on every filesystem. This is what the ADR originally decided and it is withdrawn on measurement: mode and mtime live on the inode, a hardlink is a second name for one inode, so writing a snapshot’s recorded metadata onto a farm entry mutates the CAS blob. 0061 s7 pinned that fidelity after measuring real damage from losing it, and s8 rejected first-writer-wins timestamps for shared content. Kept in the record because the mistake is reusable: the overlayfs probe verified that copy-up does not modify the lower layer and was read as verifying the Farm was safe, which it never tested. A probe proves what it exercised.

• A metadata-variant hardlink pool — one materialised copy per distinct (content, mode, mtime) and hardlinks from there, so repeats are free. Rejected for the first cut: mtimes vary per file in a real build, so the sharing rate collapses toward one copy per file and the pool buys complexity rather than speed. Worth revisiting only if measurement shows real trees cluster into few distinct metadata variants.

• A custom NFS server implementing copy-on-write in userspace, so a hardlink farm could be exported directly with no overlayfs and no CAP_SYS_ADMIN. Technically sound — we would break the link on first write ourselves — and rejected because it means writing the CoW semantics the kernel already has, in the write path of every Step, in exchange for a capability an operator-installed StatefulSet can reasonably hold.

  Amended 2026-08-02 — read the scope of this rejection before citing it against 2b. What is rejected here is implementing copy-on-write in userspace, which is a property of the hardlink-farm-without-overlayfs shape. It says nothing about a userspace NFS server sitting above an already-mounted overlay: there the kernel is still doing the CoW and the server is a protocol adapter over an ordinary directory. So “we rejected a custom NFS server” is not an argument against 2b’s linked-in-server option, only against making one do the union.

• OCI image volumes as the transport. PSA-clean at restricted (measured), node-cached, parallel and runtime-managed, and it becomes lazy for free the day a platform team adopts a lazy snapshotter — a node change Scarab neither installs nor versions, which is the cleanest possible fit to the constraint. Not chosen because the laziness is contingent on someone else’s adoption and the read-only union problem returns (a copy into emptyDir reads every file; working directly in a read-only tree changes the authoring model). Also currently untestable locally: the runtime here is docker://29.5.2, and cri-dockerd does not implement image volumes. Retained as a possible internal representation, which is where 0061 already had it.

• Fusing the boundary instead of making it lazy — automatically co-locating a linear chain of Steps into one Pod when placement profiles match and no gate or approval intervenes, so no bytes move at all. Distinct from 0061’s rejected “coarsening the unit”, which is an authoring change; this is invisible. Not rejected on merit — it attacks boundary count where this ADR attacks boundary cost — but deferred: it trades away per-Step restart granularity inside a fused group, which collides with 0027 and 0056 and would have to be governed and visible. Filed separately.

• Inferring a Step’s read set (profile-guided prefetch). Foreknowledge instead of interception. Rejected as unsafe in this direction: a wrong guess does not fail, it succeeds with a file missing. Write-side inference is a different and safe proposition and is what the stat cache is.

Red-team findings this ADR does not yet answer

A red team was run against this ADR before code rested on it, with a live cluster. It confirmed the design’s load-bearing claims — an overlay does survive being exported and re-read through a second fresh client with modes and mtimes intact, /workspace does stay put, per-Step PV/PVC churn is cheap (80 objects in 2.3 s, 40 PVs Bound in under 5 s, no quota trouble), and PSA restricted does admit an unprivileged Pod with an NFS-backed PVC including the bind. The PVC-as-envelope argument is sound.

It also found the following, which are recorded here rather than quietly fixed because several change what the design must do. Each is filed.

1. kubelet needs an NFS client on every Step node — CONFIRMED, and it dents the headline claim. The delivery path had never actually been mounted. When it was: PV bound, PSA admitted the Pod, the scheduler placed it, then kubelet failed with “you might need a /sbin/mount.nfs helper program” and the Pod hung in ContainerCreating forever. The colima node has no nfs-utils. So “the Pod needs nothing and kubelet does the mount” is true of privilege and false of packages: an NFS-sourced PV requires an NFS client on each node that runs a Step. That is not a DaemonSet and not something Scarab installs, but it is a per-node prerequisite, and it must be stated as one — it is present on most distributions and its absence on minimal images (Bottlerocket, hardened AMIs) is a real risk. It also means the s9 spike cannot run on just up or local-helm until the colima VM gets nfs-common.

   Amended 2026-08-02 — the prerequisite is confirmed, but the last sentence was wrong and the gate it gates is now MET. “The dev substrate cannot mount an Export, so verification needs a separate privileged Linux box” was recorded here and in the stage-2 table yesterday, and it survived one day. Probed on this machine:

   • The colima VM is Ubuntu 24.04.4, kernel 6.8.0-117-generic, k3s v1.35.0+k3s1, aarch64/vz/virtiofs. mount.nfs was indeed absent — but nfs, nfsv4 and nfsd are all on disk and modprobe cleanly, and apt-get install -y nfs-common pulls mount.nfs 2.6.4 from ports.ubuntu.com. One command, no custom image, no separate host. It is installed now; it survives colima stop/start and is lost on colima delete.
   • The delivery path then mounted, on colima k3s. A PV with an nfs: source (server = the node’s own INTERNAL-IP, 192.168.5.1) + PVC + consumer Pod went Running → Succeeded in ~6 s, mounted vers=3, and the Pod read and wrote through it. kubelet did the mount, exactly as part 2 claims.
   • kind needs nothing at all. The default kindest/node (v1.36.1, what deploy/local-proc/kind.yaml creates) already ships /sbin/mount.nfs, mount.nfs4 and rpc.statd — no install, no custom node image, no kind.yaml change. The one catch is the nfsvers=4.2 pin recorded in part 2: v3 fails there for want of a running rpc.statd.

   So this finding’s own gate — “a Step Pod on just up / just local-helm mounts an NFS-backed PVC and reads a file through it” — is met for local-helm and one cluster-create away for proc/kind. What remains true, and is the whole of what this finding should ever have claimed: an NFS-sourced PV needs an NFS client on every Step node, nfs-common is not in the colima image by default, and there is no colima provisioning hook in this repo — so it must be either a documented prerequisite with a fail-fast check or a recipe that installs it. (Nothing about the 2a LocalPath delivery needs any of this; see part 2.)

2. Evicting a Farm under a live Export is silent corruption — CONFIRMED, and worse than “must not happen”. With lower entries deleted while the overlay was mounted, ls of the merged directory returned empty while cat of already-cached paths still returned content, and a write into the vanished directory returned rc=0. A Step would see an empty tree, build nothing, and exit 0 — the fail-silently class this ADR rejects read-set inference for. Nothing refcounts Farms against live Exports today, and the actimeo argument (“the lower layer is immutable by construction”) inherits the same assumption. Reaping is therefore a correctness mechanism, not housekeeping.

   Addressed in s4, and the same failure turned out to have a second route this finding did not name. Farms now carry on-disk leases and an eviction fails closed while one is held. But an empty tree served to a Step does not require an eviction at all: a service restart destroys the overlay mount with the old mount namespace, and an Export whose record is then adopted from disk points at a merged/ directory that is no longer a mountpoint — empty, writable, rc=0, exit 0. Identical symptom, no Farm touched. So the invariant to hold is not “do not evict a leased Farm” but “never hand out a Workspace that is not backed by what its record says”, which costs a mountpoint check at claim time and a re-mount on adoption. Recorded because the narrower framing is what let it hide: the guard was built, tested, and the hole was beside it.

   Amended 2026-08-02 (stage-2a substrate spike) — restart is SAFER than this reasoned, and it LEAKS, which is the failure the remount-invariant above does not cover. Measured, not argued. Killing the mounter left every overlay alive in the host mount namespace: the Step Pod kept reading and writing through it throughout, and a fresh mounter that adopted the record could read a change the Step had written while the mounter was dead. So a Depot restart is not destructive to an in-flight Step on the LocalPath delivery — the mount outlives the process that made it, and the “empty, writable, rc=0” symptom above comes from an adoption that re-mounts blindly or fails to check, not from the restart itself.

   The cost is on the other side: kubelet never unmounts these, so nothing reaps them. A node accumulates one leaked overlay per restarted-through Export until mount-table exhaustion. Requirement: the Depot must reconcile at boot — adopt the mounts backing live Exports (verifying each is still a mountpoint of the expected shape) and unmount every orphan whose Export record is gone. This is not housekeeping either; it is the other half of the “never hand out a Workspace that is not backed by what its record says” invariant, and it is the s4 restart-adoption seam’s remaining obligation.

3. Part 3’s “no network in the path” collides with 0061 part 4 — CONFIRMED (documents). Folding a change set into the CAS on the service’s own disk lands it in the warm tier, and 0061 part 4 requires durability before Succeeded while 0061 explicitly forbids making warm load-bearing for durability. So either part 4 weakens or a cold upload stays on the critical path — and this ADR books neither. Unbooked cost, exactly what 0061’s own “price of seeding the warm tier” section exists to avoid repeating.

   Answered: part 4 stands verbatim and the cold upload is on the critical path, booked in part 3 above. The claim that moved is “no network in the path”, which was true of the fold and false of the Attempt.

4. fsGroup does not survive an NFS volume — PLAUSIBLE, strong, and possibly a blocker for part 2. The executor sets fs_group: WORKSPACE_GID and comments that the restored /workspace is “owned by WORKSPACE_GID and only group-writable”. The in-tree NFS plugin reports Managed: false, so kubelet applies no fsGroup ownership. Write access then rests on AUTH_SYS uid/gid against the modes 0061 s7 preserves exactly — and 0644 is not group-writable. The Step may be unable to write to its own Workspace. This ADR discusses AUTH_SYS only in the context of the fence and never as an access-control problem for the Step itself.

   Amended 2026-08-02 (stage-2a substrate spike) — the other side of this finding is now MEASURED, and it is worse than a permission problem: on a delivery where kubelet does manage fsGroup, fsGroup ALONE DESTROYS LAZINESS. The DaemonSet amendment at the top of this ADR called csi-driver-nfs’s fsGroupPolicy a trap “recorded so whoever picks it up does not discover it in a benchmark”. It was discovered anyway, on the local PV: with fsGroup: 1000 and no policy set, kubelet’s recursive chown walked the volume and copied up all 200/200 lower files before the container started. Laziness gone, before a single line of the Step ran. This is red-team #4 inverted — NFS’s Managed: false is what spares it, and any delivery that manages fsGroup pays it.

   Proven mitigation, and the recipe is exact — both halves are required:

   1. fsGroupChangePolicy: OnRootMismatch on the Pod, and
   2. the Depot pre-sets upper/ to uid:gid = <step uid>:<fsGroup> with mode 2777.

   The setgid bit is the load-bearing part. OnRootMismatch with mode 0775 still copied all 200 files: kubelet’s root-mismatch test compares the setgid bit as well as owner and gid, so a root that is otherwise correct but not setgid reads as a mismatch and triggers the full walk. Anything that reconstructs this recipe from memory and drops the 2 re-introduces a total copy-up that no test asserting correctness will catch.

   Requirement, separately measured: upper/’s ownership and mode must be set BEFORE mount(2), not after. Chmod-ing the upper once the overlay is mounted leaves stat on the merged root reporting the new mode while the kernel’s permission check still uses the old one — an unprivileged rename(2) of a lower directory fails EACCES against a merged root that displays 0777. Chmod through the merged path fixes it immediately, which is the tell; ordering the change before the mount avoids the state entirely.

5. The fence is weaker than the word “capability” implies — PLAUSIBLE. A userspace NFSv4 client needs no privilege, no PV and no kubelet, and can assert any uid; a probe mounted with noresvport, so the privileged-port defence never engaged. First-mount pinning pins the node, because kubelet does the mounting — so co-tenant Steps on one node share an identity. And path secrecy is the only remaining barrier while the path sits in a cluster-scoped PV object and in the node’s /proc/mounts. “Structurally the same capability as the HMAC token” overstates it: the token is per-Pod and unreadable by other Pods; this is not. Needs a real answer — resvport, a NetworkPolicy, per-Step uid squashing — or an honest downgrade of the claim.

6. Revocation and restart are unhandled — CONFIRMED. Revoking an export under a live client gives ESTALE then EACCES, and nfsd start logs a 90-second grace period plus “Unable to initialize client recovery tracking … Is nfsdcld running?”. This ADR handles a dead daemon but not revocation racing a Pod already in SIGTERM grace, nor the post-restart grace window, nor nfsdcld as a dependency.

7. Part 4’s zone preference has no merge story with 0055 — PLAUSIBLE. affinity appears nowhere in the Rust sources, and a placement overlay is applied as a block, so whichever writer goes last replaces the other’s affinity wholesale. A preference Scarab adds and an overlay the operator owns cannot both survive without a defined merge.

8. The s9 spike as specified cannot falsify the design — PLAUSIBLE. Operations-per-second under latency exercises none of: EXDEV/redirect correctness, whiteout and opaque handling in the drain, eviction under a live mount, restart with live Exports, or cross-AZ behaviour. Its rung guard covers reflink and overlayfs but not redirect_dir, not which drain ran, and not whether the mount was actually NFS. A spike shaped only to produce a number will produce one.

   Amended 2026-08-02 — a spike shaped to falsify was run instead, and it falsified two things. The stage-2a substrate spike produced no ops/sec number at all and was worth more than one: it exercised the mount, propagation escape, PSA admission, redirect_dir through a live mount, and all four change-set forms in a real upper (chain and verdicts in the 2a table below). Two beliefs died — fsGroup silently defeating laziness (#4 above) and mounts leaking across a Depot restart (#2 above) — and both were invisible to the metric this row warned about. The ops/sec row stays open and stays 2b’s; it is now the only remaining reason to run a number-shaped spike.

Open — deliberately not decided here

Stage 2 — what still stands between here and an Export-delivered /workspace

Stage 1 (the in-Pod drain, 0061 s3-drain) is live. Stage 2 — a Step Pod mounting a Workspace Export instead of an eager copy — is not, and the remaining work is enumerable rather than vague. Each row names what it is blocked on, the gate that proves it, and its size.

Rewritten 2026-08-02, because the sequencing was wrong. The first version of this table ordered the work behind an NFS server and then behind a machine to run it on, on the strength of the “cannot be verified here” claim the red-team-#1 amendment above retracts. Both premises are gone, and with them the reason to do the two halves together. Split stage 2 at the delivery seam: the correctness of the Export — the overlay rung executing, a real kernel-produced upper reaching read_change_set, redirect_dir/EXDEV through a live mount — has nothing to do with NFS, and waiting on a transport to test it is what kept the exact path unexecuted (the most important caveat on this ADR, below). 2a buys that today on local-helm. 2b is then a transport change against a path already known to be correct.

2a — the delivery seam and a co-located, NFS-free path. No server, no NFS client, no new protocol. This is what unblocks the correctness half.

work	blocked on	gate (what proves it)	size
The Delivery seam — Delivery::{Eager, LocalPath, Nfs} returned by Export prepare, with the one flagged volume-kind line in crates/scarab-executor-k8s/src/lib.rs as its only consumer	nothing	a deployment asking for a delivery its topology cannot serve refuses at launch (proc mode asking for LocalPath is the test case), never silently feeding Eager	small
LocalPath delivery — the Depot’s overlay merged/ published into the host mount namespace (CAP_SYS_ADMIN + Bidirectional on its data mount) and handed to a node-pinned Step as a local PV (hostPath dev-only — denied at baseline, per the amended PSA table)	the seam row	on just local-helm: a Step Pod writes /workspace through a REAL overlay and the Depot reads the change set out of the upper	medium — local-helm only, the darwin Depot in proc mode has no overlayfs
0ad393c — the exact-path overlay test — read_change_set has never met a kernel-produced upper; every green test covers the copy rung’s approximation	the LocalPath row, which is the place to run it — not a separate privileged host	whiteouts, opaque dirs, file renames, and absolute+relative redirect resolution asserted against a REAL upper produced by a REAL Step	the tier, and 2a supplies it
Export/Farm lifecycle and reaping — the per-Step volume objects and the Farm underneath them (eviction under a live Export is silent corruption, red-team #2)	git-bug 24476bc (on the critical path, not beside it)	no orphaned volume object, Export or Farm survives Run teardown; a reaped Farm under a live Export is refused, not corrupted	medium
eager|lazy cutover config — the volume-kind line becomes a per-deployment posture	the rows above	an explicit knob, fail-closed; eager_fetch_total goes to zero and stays there on a lazy deployment	small, once the substrate is proven

Amended 2026-08-02 — the 2a chain was EXECUTED on this machine, end to end, and it holds. This is measurement, not a plan. Substrate: colima node, Ubuntu 24.04.4, kernel 6.8.0-117-generic, k3s v1.35.0, docker runtime, ext4. Overlay module parameters redirect_dir, index, nfs_export all default N. The chain, in the order it was proven:

1. The mount. export.rs:501-508’s exact option string mounted first try from a privileged Pod with hostPath + mountPropagation: Bidirectional (that Pod’s /warm was already shared); the kernel echoed +uuid=on,nouserxattr.
2. Propagation escapes the Pod — the load-bearing unknown. The overlay appears in the node’s /proc/1/mounts with findmnt PROPAGATION=shared. The whole LocalPath delivery rested on this and it had never been observed.
3. PSA restricted admits the delivery. A local PV (nodeAffinity → colima) + PVC was admitted and delivered the real overlay as /workspace — fstype overlay in the container’s own /proc/self/mounts, not a copy of it.
4. A restricted Step produces all four change-set forms. With runAsNonRoot, runAsUser 1000, drop: [ALL], seccompProfile: RuntimeDefault, the upper held: a new file (uid 1000); a modified file (trusted.overlay.origin, lower nlink U-1); a deletion (whiteout chardev 0:0); and a renamed directory (trusted.overlay.redirect='docs' on an empty upper dir, trusted.overlay.impure='y' on its parent). The lower layer stayed byte-for-byte pristine, and a file the Step never read was never copied up — laziness holds under a real Step, which is the claim this ADR is built on.

De-risked by this spike — the LocalPath delivery row’s substrate (rows 1–4 of the chain are exactly its gate’s preconditions), and the 0ad393c exact-path row’s inputs: a kernel-produced upper carrying every marker shape the reader must handle now demonstrably exists and has been characterised on the substrate local-helm runs on.

Still open, and not weakened — the Delivery seam row (refuse-at-launch is untouched by this spike), the second half of the LocalPath gate (the Depot reading the change set with its own code, as opposed to the shapes being present for it to read), the whole of 0ad393c (read_change_set has still never met this upper), the cutover-config row, and Export/Farm lifecycle and reaping — which this spike ENLARGED, see the mount-leak amendment at red-team #2.

Requirements this puts on production code. Each is a constraint, not a note:

• read_change_set requires CAP_SYS_ADMIN or it is silent data loss. Without it listxattr omits the trusted.* names entirely — no error, no hint — so a renamed directory reads as “delete docs, create empty documentation” and the contents are lost, because upper’s documentation genuinely is empty and the files live in lower behind the redirect. This is the omission-shaped sibling of part 3’s ENODATA discussion, and the existing guard does cover it: changeset.rs’s require_marker_privilege() reads CapEff from /proc/self/status and returns NoSysAdmin before walking whenever Markers::Overlay is requested — a capability check, not a getxattr probe, so silent omission cannot slip past it. The obligation that remains is on the deployment: the Depot must actually hold the capability wherever an overlay-rung drain runs, or every such drain refuses.
• hostPath is refused under PSA restricted — exact message: restricted volume type "hostPath". The local PV + PVC is the only admissible delivery; hostPath is a dev-only shortcut and will not survive a restricted namespace. (Consistent with the amended PSA table, which already records hostPath as denied at baseline too.)
• Never reconstruct upperdir by parsing host mounts. The host’s /proc/1/mounts records the Pod’s paths for lowerdir and upperdir, not host paths, because that is the namespace the mount was made from. Code that re-derives an upper from the host mount table reads a path that does not exist there.
• The fsGroup recipe (OnRootMismatch + upper/ at uid:gid with mode 2777) and the set-ownership-before-mount(2) ordering: both at red-team #4 above.
• Boot-time mount reconciliation: red-team #2 above.

Refuted, so nobody re-litigates it. A redirect_dir directory rename by an unprivileged, zero-capability uid WORKS — proven directly, through a bind mount, and through a PSA-restricted Pod’s PVC. It does not need CAP_SYS_ADMIN; only reading the resulting xattr does. The two are different privileges on different sides of the boundary and this ADR should never again imply otherwise.

Still untested, plainly. Multi-node — the cluster is single-node, so the local PV’s nodeAffinity was never really exercised and node-pinning remains asserted rather than shown; concurrent Steps on one Export; and kubelet restart mid-Step.

Node pinning is what makes 2a a dev/single-node posture rather than the production default, and it must not be allowed to look like a shipping answer: it caps a Run at one node’s throughput and writes a placement constraint onto every Step, which fights 0055 in precisely the way part 4 refuses to. 2a is a test and dogfood delivery that happens to be real.

2b — the transport. Everything only a network filesystem can answer, now locally verifiable (red-team #1 amendment: colima mounts NFS, kind’s node image already can).

work	blocked on	gate (what proves it)	size
A userspace NFSv4 server — nfs-ganesha, or a linked-in server, exporting the overlay merged view; the rung that needs no kernel nfsd on the Depot	2a (export something already known correct)	a Step Pod mounts the export and round-trips a file, with the harness failing LOUDLY on any silent rung fallback	spike, days
Delivery::Nfs — the PV, its mountOptions: [nfsvers=4.2] pin, and per-Step bind/reap churn	the server row	churn stays cheap at Run scale; the mount is asserted to be NFS at the version pinned, not merely “a mount”	medium
nfs-common on Step nodes — confirmed per-node prerequisite; not in the colima image, and this repo has no colima provisioning hook	a choice: document it with a fail-fast preflight, install it from a recipe, or take csi-driver-nfs (see the DaemonSet amendment, and its fsGroupPolicy trap)	just local-helm either mounts an Export or fails naming the missing package — never ContainerCreating forever	small
fsGroup over NFS — kubelet applies no fsGroup to NFS volumes (Managed: false, red-team #4), so a non-root Step may be unable to write its own Workspace	a decision: per-Step uid squash on the export vs group-writable modes stamped at Farm build (each changes what s7 fidelity means for modes)	a non-root Step under the ADR-0039 restricted baseline writes an Export-backed /workspace	medium, and a potential part-2 blocker
The fence over AUTH_SYS — resvport, NetworkPolicy, uid squashing (red-team #5: a userspace client needs no kubelet and can assert any uid; first-mount pinning pins the node)	the server row	a co-tenant Pod on the same node cannot read another Step’s Export, demonstrated rather than argued — or the “capability” claim is honestly downgraded	medium
Revocation, ESTALE and the restart grace window (red-team #6: 90-second grace, nfsdcld)	the server row	revoking under a live client, and restarting the server with live Exports, both produce a named Attempt failure rather than a mystery I/O error	medium
Ops/sec under latency for a cargo-shaped write load against a real in-cluster Export	everything above	the one unpriced number in the design; it decides whether part 5 is an accelerator or a requirement	spike

None of the 2b rows can be answered by 2a, and that is the reason 2b exists rather than being folded away: they are all properties of a network protocol, and LocalPath has no network.

• The exact path has never executed, and every green test exercises the approximate one. This is the most important caveat on this ADR’s status and it is easy to miss, because the code is written, reviewed and tested. Part 3’s argument — that the rungs “do not differ in correctness” because the upper layer is the kernel’s own record — rests entirely on the overlay rung. Reaching it needs a privileged Linux host with a Rust toolchain, which nothing in this repo’s loop has: the dogfood node has no Rust, GitHub Actions is quota-blocked, and darwin cannot mount overlayfs at all. So the mount, the resulting upper layer, and read_change_set against a real one are all unexecuted, while the copy rung — whose change set is the (size, mtime, ctime) approximation — is what every passing test covers. A reader should treat “the change set is exact” as designed and unverified, not as working. Tracked as the tier problem rather than as a bug in any one slice (git-bug 0ad393c), because the missing thing is a place to run it.

  Amended 2026-08-02: the status is unchanged, the diagnosis is not. The exact path still has never executed. But “nothing in this repo’s loop has a privileged Linux host with a Rust toolchain” is the wrong reason — the Rust toolchain is not needed on the host, and just local-helm already runs the Depot as an in-cluster StatefulSet on a Linux node with a real ext4 disk. 2a’s LocalPath delivery makes a Step write through that overlay with no NFS anywhere, which is where read_change_set finally meets a kernel-produced upper and where redirect_dir/EXDEV runs through a live mount. The blocker was never the tier; it was having filed this behind a transport. That is what the resequenced stage-2 table above fixes.

  Amended 2026-08-02 (stage-2a substrate spike) — the substrate half of this caveat is now DISCHARGED; the code half is not, and the distinction is the whole of the remaining status. The overlay was mounted from the exact option string the code writes, its propagation was observed escaping into the node’s mount namespace, a PSA-restricted Step Pod received the real overlay through a local PV, and that Step produced a kernel-made upper containing every marker shape part 3 enumerates — new file, copy-up origin, whiteout, and a directory redirect with its parent’s impure (chain and substrate in the 2a amendment above). So “the mount, the resulting upper layer” are no longer unexecuted, and the reasons given for why they could not be are all gone. What is still unexecuted is read_change_set against one of these uppers. Scarab’s own reader has met no kernel-produced upper; every green test still covers the copy rung’s approximation. A reader should now treat “the change set is exact” as designed, with its substrate verified and its reader unverified — a narrower claim than yesterday’s and still not working. 0ad393c stays open and is no longer blocked on anything but doing it.

• Operations-per-second under latency, measured, for a cargo-shaped write workload against a real in-cluster Export. This is the one unpriced number in the design and it is the number that decides whether part 5 is an accelerator or a requirement. Byte throughput is not the thing to measure. The harness must fail loudly if it silently falls back a rung on the ladder — a spike that reports a flattering number because reflink or overlayfs was unavailable is the failure mode this repo has already paid for.

• Farm-and-Export reaping and the warm-tier space bound. Blocked on git-bug 24476bc, which is now on the critical path rather than beside it.

• overlayfs with an NFS mount as lowerdir, which part 5 requires. Documented as working with caveats; unverified here, and load-bearing for the accelerator only.

• Whether overlayfs metacopy=on can restore a hardlink Farm. Answered: not for a Workspace Export — but the first answer recorded here was wrong, and the correction matters.

  What was wrong. This ADR briefly claimed metacopy=on “mounts and then does a full data copy-up anyway”, citing upper entry size 17, blocks 8 on a 17-byte file. That is not evidence of a data copy: metacopy preserves size by design, and 8 blocks is ext4’s 4 KiB minimum allocation. Re-run on an 8 MiB lower file the answer inverts — size=8388608 with 8 blocks allocated and trusted.overlay.metacopy="" on the upper entry. metacopy works, from the mount option alone, with the module parameter still N. A hardlink Farm under metacopy was also verified safe: link count 2, CAS blob still 644, merged view reading 755.

  The claim that the module parameter made it “accepted and ineffective” was also a category error: the service-side overlay is mounted on the workspace service, so a module parameter would be one node — the one the constraint already permits — not every Step node.

  The real reason it cannot be used, which is a different fact entirely:

  mount -o index=on,nfs_export=on,metacopy=on  →  refused: conflicting options

  An Export must be exportable, and nfs_export and metacopy are mutually exclusive. So on the service side the Farm cannot use hardlinks, and part 1’s reflink-or-copy stands — for this reason, not the one first written down.

  And it opens something. Part 5’s node-side accelerator mounts an overlay with no nfs_export, so metacopy is available there. That is worth following up rather than burying: it would let the accelerator’s upper carry metadata with no data copy.

  Kept in full because the mistake is the reusable part. The first probe measured a real thing and the conclusion drawn from it was unsound, for the third time in this ADR’s short life — and each time the tell was the same, a probe whose result was read as answering a question it had not asked.

• Automatic Cache detection for lockfile-derived trees — recognising node_modules, .venv, vendored ~/.m2 and the pip/uv caches from their lockfiles and routing them through 0007’s Cache instead of the Workspace, so that less is subject to the write cost at all. Deliberately not decided in this ADR: it is a heuristic whose failure mode is wrong output (target/ is Cache-shaped when the next Step rebuilds and an output when the next Step consumes a binary), and it deserves its own trade-off record rather than riding along on a transport decision.

References

• 0004 — pod-per-step; this ADR changes how the workspace reaches a Pod
• 0007 — Workspace / Result / Artifact / Cache
• 0027 — content-addressed invalidation; “smart never means mysterious”
• 0029 — per-file merkle CAS; the Farm is its tree-shaped view
• 0039 — the restricted step baseline this ADR does not weaken
• 0050 — retention and GC
• 0055 — placement profiles; part 4 must not silently amend them
• 0061 — supersedes its part 2; amends its line on privileged mounts, its node-driver install prerequisite, and its cross-AZ confinement claim