0064. Durability tiering and the write path: warm-first, flush before `Succeeded`, and the promise is the backing

• Status: Accepted
• Date: 2026-07-28
• Deciders: thulasi.ram (architect)
• Builds on: 0061 (amends its part 4), 0062 (the drain this write path serves), 0063 (the Data Depot), 0027, 0029

Amended 2026-08-24 by 0067 — parts 1–2 retire: no warm-first write, no flush RPC. The durable write happens in one pass at drain time, as an atomic pack written straight to the object store, so warm stops being a durability tier at all. The pinned ordering invariant is made vacuous, not weakened. Part 4’s warm-only deployment mode and part 5’s stamp (attempts.output_durability, the boot banner, the st_dev probe, the GC suppression) are deleted; warm-only survives only as a runtime degradation in which a drain fails rather than succeeding with a smaller promise. Part 3’s “backing is measured, not assumed” test stands.

Context

0061 part 4 states the rule:

4. An Attempt is not Succeeded until its Workspace Snapshot is durable. On spot, a node can vanish between “the Step exited 0” and “its evidence is safe”. Declaring success before durability puts a claim in the durable record that the record cannot back — the one thing this product may not do.

That rule is implemented, and it is implemented as a write ordering. TieredCas::put_blob and put_tree await cold first and propagate its error; a warm failure is counted and swallowed. The comment at tiered.rs:250 says so in as many words: “Cold first: this is the leg that licenses Succeeded.”

Two problems with that mechanism came out of building 0062’s drain, and one came out of checking what actually backs each tier.

The per-blob cold round-trip is the cost this whole line of ADRs exists to remove. TieredCas::ingest is:

let snapshot = self.cold.ingest(path).await?;   // full walk, one cold round-trip per blob
match self.warm.ingest(path).await { ... }      // a SECOND, independent full walk

Two complete walks of the tree, and the cold walk pays a network round-trip per file — the 4–6 ms per file that 0061 measured as 81–88% of the cost it set out to fix. There is even a tripwire for the case where the two independent walks disagree on the root. The change-set fold is better, since only changed files are written, but each changed file still carries an interleaved cold round-trip on the critical path.

“Durable” is a property of the backing, not of the tier’s name. StoreConfig is S3 | LocalDir, and LocalDir may point anywhere — including a directory on the warm volume itself, or the Helm chart’s scratch emptyDir. In those configurations “written to cold” licenses nothing: cold is the same disk as warm, or worse, a volume that dies with the Pod. The word “cold” implies a promise that in those deployments nothing is making.

And the deployments that matter do have a real cold tier, which is worth recording because an earlier reading of this said otherwise. deploy/local-helm/deploy.sh defaults S3_BUCKET and stands up in-cluster MinIO on a PVC; just up runs MinIO under compose; a real chart install has the operator set scarab.s3.bucket. The degenerate case is the chart’s unset default, labelled “dev only”, which no shipped deployment path uses. So this is a misconfiguration hazard rather than the normal case — but it is silent, and silence is what this ADR family refuses.

Decision

1. The write path is warm-first with a single batched flush. A drain writes to the Data Depot’s warm tier locally — one walk, no network — and then performs one batched archival flush to cold. This replaces per-blob interleaved cold round-trips and the second independent walk. It is strictly faster than the status quo and it is also simpler: one writer, one walk, one flush.

2. The flush gates Succeeded wherever a real cold tier exists. Part 4’s guarantee is kept in full: the Attempt is not Succeeded until the flush completes. What changes is how the bytes get there, not what must be true before success. So “success only when written to cold” now means the flush completed, not that every blob individually round-tripped during the fold.

3. A cold tier is one whose backing is independent — measured, not assumed. The test is st_dev of the cold directory against the warm directory, which is one stat:

cold	verdict
S3 / MinIO	genuine second tier
LocalDir on a separate volume (a second PVC)	st_dev differs → genuine second tier
LocalDir on the warm volume	st_dev matches → not a tier; no independent durability
LocalDir on an emptyDir of its own	st_dev differs — the probe reads it as a second tier — yet it dies with the Pod (corrected 2026-08-01, see below)

(Corrected 2026-08-01, while building the probe.) The original row here claimed the chart’s emptyDir is “same volume” and therefore caught by st_dev. That is only true when the cold LocalDir shares the warm volume; a cold LocalDir on an emptyDir of its own is a different device, so the probe answers separate-volume while the backing dies with the Pod. st_dev measures independence, not persistence — persistence of the backing is the chart’s check (refuse or loudly label a cold path that resolves to an emptyDir), not a stat’s. Follow-up: the chart-side check is not yet built.

“Is it object storage?” was the obvious test and it is the wrong one: it would reject a second PVC, which is a perfectly good cold tier, and accept nothing else. The backing is the promise — and st_dev is one measurable half of it.

4. Where there is no independent cold tier, the Depot degrades gracefully rather than refusing. Warm gates Succeeded, and the deployment is loudly a warm-only-durability deployment: stated at startup, and stamped on the record (below). Refusing to boot was considered and rejected — it re-imposes object storage as a prerequisite, which 0061 declines to do even for the CAS (“an operator’s cost decision, not an engine decision”).

5. The durability that backed a snapshot is stamped on the Attempt, beside output_snapshot and output_identity. A startup log tells whoever read logs that day; it cannot explain a Run a month later. Deployments change — an operator adds S3 next month, and then Attempts from before and after have different guarantees with identical records. Stamping it means the UI can say “this Run’s evidence was never archived; this deployment had no independent durable tier” instead of showing a mystery gap. This is 0027’s rule applied to durability rather than to invalidation.

(Named 2026-08-01, as built.) The column is attempts.output_durability and its values are the three tier strings the Depot serialises everywhere — FlushResponse.tier, SettledExportDto.tier, GET /v1/tier: object, separate-volume, warm-only. NULL means unknown (an Attempt from before this slice, or a flush answered by an older Depot during deploy skew) — recorded as unknown, never defaulted.

The amendment to 0061 part 4, stated plainly

Part 4 as written is unconditional, and allowing Succeeded on warm alone contradicts it — 0061’s own retention table says warm “promises none”. So this amends part 4, and the amendment is marked in 0061 in place.

What the amendment preserves is the invariant; what it changes is the mechanism. Read part 4’s own reasoning: “Declaring success before durability puts a claim in the durable record that the record cannot back — the one thing this product may not do.” The invariant is never make a claim the record cannot back. Cold-before-Succeeded was one mechanism for it, and a good one where a cold tier exists.

A deployment that declares “durability here is warm’s” and then succeeds on warm is not making an unbacked claim. It is making a smaller, true one. What 0061 was protecting against is the silent version — success implying an archive that was never written. Disclosure plus the per-Attempt stamp preserves the invariant under a weaker mechanism, which is why this is an amendment rather than a retreat.

0061 also explicitly forbade a related thing, and that prohibition stands:

“writing warm-first and letting the service tier onward would make the warm tier load-bearing for durability, which part 4 forbids.”

This ADR writes warm-first and does not let the service tier onward asynchronously. The flush is synchronous with respect to Succeeded. Warm is the write path; cold remains the promise wherever it exists. Asynchronous archival was considered and rejected — see below.

The control plane’s write leg (added 2026-08-01)

The decision above was built Depot-side first, and its “one walk, no network” phrasing is true of exactly one topology: the Depot’s own drain, whose warm tier is its local disk. This section extends warm-first to the control plane, whose warm tier is the Depot over HTTP, and states what the words mean there — because the same slogan quietly overclaims on that leg.

Warm-first on the control plane means one deduped HTTP walk plus one flush RPC — not “no network”. The per-Step drain uploads through the Depot’s PUT verbs (WorkspaceClient::ingest: one batched /have, then parallel PUTs of only what is missing), and then calls POST /v1/cas/flush with the snapshot root. The win over the cold-first shape is real but different from the Depot’s: the double walk and the double cold write of 0061’s “price of seeding the warm tier” bullet go, and the cold tier sees one batched, probed flush instead of a round trip per object — but every object still crosses the network once to reach the Depot’s warm tier.

The Depot’s PUT handlers are warm-only, and the flush RPC is the only cold writer. The PUTs used to write cold-first through TieredObjectStore::put — that was the old durability leg, and part 4’s invariant rode on it. The invariant did not weaken; it migrated into the flush contract: “an Attempt is not Succeeded until its snapshot is durable” is now “the caller awaits the flush — the settle path’s internal flush phase, or the /v1/cas/flush RPC — before reporting Succeeded”. The flush’s refusals split on the one axis a caller acts on: 422 retryable: false for an addressing disagreement between the tiers (the only fatal class — no retry converges two binaries that canonicalise differently), 503 retryable: true for everything else, including a warm miss — a wiped Depot warm tier is recoverable, because the re-driven drain re-uploads via /have before the retried flush, so answering fatal there would strand a provably repairable state. The flush requires the control plane’s own token scope (browse): unlike a content-addressed PUT it is not harmless under any valid token — it commands cold round trips for arbitrary roots.

A Depot outage MAY fail Attempts — promptly and with a named cause, never as a timeout. This is the availability question 0061 answered by forbidding warm-first on the control plane, decided the other way (architect’s decision): the Depot is now on the write path and the durability path of every workspace Step, so when it is down, Steps whose evidence cannot be drained fail, saying so. What is bought with that coupling is the end of the double write and of the tar-tunnel-era shape; what is kept is legibility — the failure names the Depot and the leg (upload vs flush), and the retryable verdict tells the caller whether re-driving can ever help. 0061’s prohibition is marked superseded in place, with the reasoning preserved.

Reads keep the fall-through. Nothing here touches the read side: the control plane’s TieredCas still opts into fall_through_on_warm_error, so a rolling or unreachable Depot degrades Browse, the GC mark walk and rerun widening to slower, never wrong. The write leg is where the coupling was accepted; the read leg still refuses it.

Alternatives considered

• Keep cold-first per blob (status quo). The invariant comes free and needs no disclosure mechanism. Rejected on cost: a network round-trip per file on the critical path, plus a second full walk, is precisely the per-file cost 0061 and 0062 exist to eliminate.
• Warm-first with asynchronous archival, Succeeded on warm. The fastest option, and the one first proposed in the discussion that produced this ADR. Rejected: it makes warm load-bearing for durability, which is what part 4 forbids and what 0061 named explicitly. It also requires the machinery that falls out of that choice — an eviction pin for un-archived snapshots, and a state for an Attempt whose evidence was lost after it went green, which would mean a Succeeded Attempt becoming un-green after downstream Steps consumed it and the forge already showed a checkmark. That is a large amount of vocabulary and cascade semantics bought for a latency win that the batched flush delivers anyway.
• Refuse to boot without an independent cold tier. Honest and simple, and it makes object storage a hard prerequisite. Rejected per (4).
• Test for cold-ness by asking “is it S3?”. One line, and wrong in both directions: it rejects a second PVC and cannot detect a LocalDir sitting on the warm volume. Rejected in favour of st_dev.
• Record the degraded guarantee only at startup. Free. Rejected: it cannot explain an old Run, and a deployment’s guarantee changes over time while its old records do not.

Consequences

• TieredCas’s ordering is superseded, including the “Cold first” comment at tiered.rs:250, which is currently the clearest statement of the old mechanism and must be rewritten rather than deleted — a reader who finds it will otherwise trust it.
• The two-walk ingest goes, and with it the root-disagreement tripwire between two independent walks, which existed only because there were two walks. The canonicalisation-skew tripwire on put_tree is a different hazard and stays.
• A flush that fails fails the Attempt where a cold tier exists. The Step exited 0 and its evidence did not reach the archive, so it is a retryable failure that must name the cause rather than surfacing a mystery I/O error.
• One new column on attempts for the durability tier that backed the snapshot — attempts.output_durability, holding object / separate-volume / warm-only (part 5).
• A warm-only deployment is a supported configuration with a stated, weaker guarantee — not a misconfiguration to be rejected, and not a silent downgrade either.
• The batched flush is a new failure surface: a partial flush must not report success, and the batch must be idempotent, because the CAS is content-addressed and a retried flush will re-offer the same keys.

References

• 0027 — content-addressed invalidation; smart never means mysterious
• 0029 — per-file merkle CAS; why a retried flush is idempotent
• 0050 — retention classes; cold is the tier with the TTL
• 0061 — amends part 4; the warm/cold tiers and the retention table
• 0062 — the change-set drain this serves
• 0063 — the Data Depot; logs take the opposite decision, and the asymmetry is that logs cannot be re-derived
• 0065 — what is kept, and what is recomputed instead