0024 — Retention defaults, what each boundary destroys, and cold storage
Status: Accepted · 2026-08-09
Decided WITHOUT the owner. See How to overturn this, below — it is designed to be cheap.
Relates to: 0014 (the scale envelope this is derived
from, and the DETACH → Parquet escape hatch), 0003
(why the projections survive what the event stream does not),
0001
Amends: SPEC §D.1 (raw_retention_days default), §D.3, §D.4, §D.11
Resolves: git-bug 344bf68 — “Retention defaults delete the history the product is selling, on
no stated requirement”, and docs/ORCHESTRATION.md open question 5. Also git-bug 747bde8 — the
thirty-day replay-safety story — in
Amendment 4.
⚠️ The
raw_retention_daysderivation in the Decision below is RETIRED.oto replaygates on supersession, not on age, so the 30 is a chosen number and its justification lives in Amendment 4. The number itself did not change.
Context
Section titled “Context”oto shipped two retention defaults — raw payloads 14 days, events 13 months — traceable to no stated
requirement. docs/ORCHESTRATION.md listed them as an open question for the owner. The charge in the
issue is serious and specific: oto’s pitch is “for every alert that has ever fired”, 14 days is not
“ever”, deletion is by DROP PARTITION and therefore irreversible, and the first buyer who needs a
year of history for an audit discovers it was deleted by a number nobody chose.
The issue also asserts something that turns out not to be true, and establishing that was the whole job: retention does not delete the history the product is selling. The rest of this ADR is the evidence, then the numbers that follow from it.
1. What is actually reaped — the complete list
Section titled “1. What is actually reaped — the complete list”Four tables age out, and only four. Everything else in the schema is unpartitioned and has no reaper.
| Table | Grain | Window | Dropped by |
|---|---|---|---|
ingest_batches |
daily | raw_retention_days |
partitions.manage → oto_drop_partitions_before |
ingest_rejections |
daily | raw_retention_days |
same |
alert_events |
monthly | event_retention_months |
same |
ui_events |
hourly | 24 h (SSE resume buffer, ADR 0010) | same |
Plus three small side tables pruned by row: ingest_dedup (10 min), sessions (on expiry),
enrichment_cache (on expiry). (alert_event_keys is now a fourth — Amendment 1 — and
idempotency_claims, added later, a fifth at 24 hours.)
Never reaped, by anything, at any setting: alerts, alert_occurrences, alert_groups,
alert_group_members, rule_snapshots, notifications, notification_deliveries,
channel_threads, enrichments, silences, alert_quality_daily,
(⚠️ alert_event_keysalert_event_keys is now pruned by row — see
Amendment 1).
This list is not a backlog of tables waiting to be partitioned. Partitioning them was evaluated and declined, because on each one the constraint a partition key would have to join is the product invariant itself — see Amendment 3.
2. What survives each boundary — the reconstructibility answer
Section titled “2. What survives each boundary — the reconstructibility answer”Line up that second list against README.md’s own sentence. The promise is that for every alert that
has ever fired, oto shows when it first appeared, every episode since, what the rule said at that
moment, who was told, on which channel, in which thread, who acknowledged it, and how it ended.
Every clause of it is served by a table retention never touches:
| The promise | Where it lives | Reaped? |
|---|---|---|
| when it first appeared | alerts.first_seen_at |
never |
| every episode since | alert_occurrences (seq, started_at, ended_at, reopen_of) |
never |
| what the rule said at that moment | rule_snapshots via alert_occurrences.rule_snapshot_id |
never |
| who was told, on which channel | notifications + notification_deliveries |
never |
| in which thread | channel_threads (channel_id, ts) |
never |
| who acknowledged it | alert_occurrences.acked_by_label, acked_at, ack_note |
never |
| how it ended | alert_occurrences.state + resolve_reason |
never |
After the 14-day (now 30-day) raw boundary you can still answer every one of the above, plus every
label and annotation, the flap score, and the daily hygiene rollups. You can no longer answer: what
were the exact bytes Alertmanager sent?, which elements of that payload were rejected and why?, and
you can no longer replay a stored batch after a parser fix. The raw tables have no read API at all
— no path in ⚠️ That stopped being true. Both
raw tables are now read by the contract — api/openapi/openapi.yaml touches ingest_batches or ingest_rejections — so they are
today a psql-only debugging artifact, not a product surface.GET /api/v1/sources/{id}/rejections and
GET /api/v1/sources/{id}/failed-batches (openapi.yaml:2897, :2957;
internal/sources/api/feeds.go) — and neither takes a time-window parameter, so this setting IS the
depth of both feeds. The boundary now empties a screen, not just a psql session
(Amendment 4).
After the 13-month event boundary you keep the whole table above. What you lose is the
instant-by-instant narrative: the ordered transitions with an actor on each, the pre-rendered
summaries, enrichment.completed, group.storm_started, delivery.* per attempt, and — the one that
matters — comment.added and the note on occurrence.unacknowledged, which live nowhere but
alert_events and cannot be rebuilt from anything. alerts/api/handlers.go says it plainly: “a
comment is an event like any other: it cannot be edited or deleted, because the timeline IS the
record.” When the partition goes, the human writing goes with it. The API surface that empties is
GET /alerts/{id}/events, GET /occurrences/{id}/events, GET /alert-groups/{id}/timeline.
So the honest summary is: retention deletes the narrative, never the record. That is a real loss, and a much narrower one than the issue supposed.
3. What it actually costs — measured, not estimated
Section titled “3. What it actually costs — measured, not estimated”A defence that rests on “storage” has to say how much. Measured on Postgres 17 with the real DDL and
its real indexes, synthetic rows modelled on the actual event mix and a three-alert Alertmanager v4
webhook body (9 labels, 2 annotations, generatorURL, per alert):
alert_events: 752 bytes/row all-in (393 B heap + 357 B across its four indexes), at 200 000 rows with every optional FK populated — an upper bound, sincegroup_idandoccurrence_idare NULL on many real rows.ingest_batches: 1 698 bytes/row for a 3-alert batch. The 3 305-byte JSON body compresses to 1 165 B inline; TOAST never engages. That is ~566 B per alert instance persisted.
At ~8 events per firing episode and ~3 payload appearances per episode (initial plus repeats):
| Alert firings/day | alert_events @ 13 mo |
ingest_batches @ 14 d |
@ 30 d | @ 90 d | @ 13 mo |
|---|---|---|---|---|---|
| 1 000 | 2.4 GB | 24 MB | 51 MB | 153 MB | 0.7 GB |
| 10 000 | 24 GB | 238 MB | 510 MB | 1.5 GB | 6.7 GB |
| 100 000 | 237 GB | 2.3 GB | 5.0 GB | 15 GB | 66 GB |
The storage defence of 14 days is worth 27 MB at 1 000 firings a day and 270 MB at 10 000. It does not survive contact with a number. ADR 0014 itself calls 100 000 firings a day a broken alerting configuration rather than a scale target.
The event side reads differently. ADR 0014 puts one org’s pessimistic ceiling at 10M events/month and names 50–100M rows as where a single Postgres starts to hurt. 13 months × 10M = 130M rows ≈ 98 GB, which is the top of that band exactly.
Decision
Section titled “Decision”raw_retention_days: 14 → 30
Section titled “raw_retention_days: 14 → 30”⚠️ The number stands; the derivation below does NOT — oto replay gates on supersession, not on
age. See Amendment 4, which is
where the justification for the 30 now lives.
The one stated requirement a stored raw payload exists to serve is SPEC acceptance criterion 36 —
“replaying a stored ingest_batch after a parser fix reproduces the same state without duplicate
Slack messages”. That replay is idempotent only while the batch’s event dedupe keys are still in
alert_event_keys, which SPEC §D.4 prunes at 30 days. Past that horizon a replay appends the
timeline a second time, so a payload kept longer cannot be used for the thing it is kept for.
Thirty days is therefore the exact width of the window in which a raw payload is useful, and it is
derived from another number in the system rather than picked — the same relationship
MinRefireGraceSeconds has with ingestion/domain.DedupTTL. The two are coupled: if the
alert_event_keys pruner moves, this moves with it. The coupling is stated on
DefaultRawRetention so a future change to one cannot silently invalidate the other.
14 days was still strictly worse: traceable to nothing, and defended by a storage figure that turns out to be tens of megabytes.
event_retention_months: 13, kept, on a different reason than the one written down
Section titled “event_retention_months: 13, kept, on a different reason than the one written down”13 stays, but the recorded justification was wrong. “Thirteen so year-on-year comparisons work” is the
justification for alert_quality_daily — and that table is never reaped, so year-on-year works at
any event retention. The real constraint is the one from §3: 13 months is the longest default that
keeps one org inside ADR 0014’s own scale envelope at ADR 0014’s own stated ceiling. Longer would
ship every install past the point ADR 0014 says to revisit the datastore.
The existing bound (1–120 months) already lets a compliance buyer configure ten years, and that is the answer to “we need a year for an audit” — a supported setting, not a code change. Above roughly 13 months at high volume, expect ADR 0014’s revisit triggers to fire; that is a documented consequence, not a surprise.
The per-org setting now reaches the reaper
Section titled “The per-org setting now reaches the reaper”This was the load-bearing defect underneath the issue. partitions.manage read
Config.Retention — a process-global value — and never looked at orgs.settings at all. The
per-org keys added in a827f4a were validated, bounded, rendered with an origin and enforced nowhere.
“It is configurable” was not a mitigation, because it was not true.
A partition holds every tenant’s rows, so per-org retention cannot be honoured by dropping one. There are exactly two honest options, and oto takes the second:
- shortest wins — silently deletes data an org configured itself to keep. Rejected outright.
- longest wins — retention is a floor, never a ceiling.
partitions.managenow drops at the maximum of the deployment’s configured window and every org’s effective setting. An org that raises its window gets what it asked for; an org that lowers its window may keep rows a while longer, which costs disk and destroys nothing. Every failure in that fold — unreadable org, failed list — widens the window rather than narrowing it, because reading a setting must never be able to cost data.
For a self-hosted single-org install, which is what oto is, the maximum is the org’s value.
Cold-storage export: SCOPED, not built. Until it exists, the product says so plainly
Section titled “Cold-storage export: SCOPED, not built. Until it exists, the product says so plainly”Ruling it out is untenable. The one thing retention destroys irreversibly is human-authored writing —
comments and unack notes — and no setting recovers it; a buyer who needs the narrative beyond their
window has nothing to buy. ADR 0014 already designed the hatch (“a cold partition can be
DETACH PARTITION’d and exported … a bolt-on, not a migration”). Scoping it is finishing a
sentence ADR 0014 started.
The shape, deliberately minimal — this is a retention hook, not an analytics pipeline:
- Trigger.
partitions.manage, before its drop step. When export is enabled, a partition with no export receipt is not dropped — the job logs and retries next hour. That single rule is what turns a silent deletion into a visible, blocking one. Plus an operator-invokedoto export events --from --tofor a one-off pull. - Shape. One file per partition, the partition’s rows verbatim, no joins and no reshaping. A dropped partition is a self-contained month; keeping it that way is what makes the export trivially re-loadable and the receipt trivially checkable.
- Format. gzipped JSONL, not Parquet. Parquet needs a Go writer dependency and a schema
registry to stay honest across migrations; JSONL is a streaming row scan with zero new dependencies
and loads into DuckDB (
read_json_auto) or ClickHouse in one statement. Parquet stays the upgrade ADR 0014 names, reachable later from the same files. - Where it lands. A configured directory,
OTO_RETENTION_EXPORT_DIR— a mounted volume or PVC. Not S3 in v1: an object-store client is a dependency and a credential surface, and a directory is what a Helm chart can mount and what every existing backup tool already covers. - Receipt. A small
retention_exportstable —(parent, partition_name, row_count, bytes, sha256, path, exported_at)— which is both the drop gate and the operator’s proof the month left the building. - Default: OFF. A product that writes files nobody asked for is worse than one that does not.
- Explicitly out of scope: restore/import (documented as a DuckDB one-liner against the files), S3, Parquet, encryption at rest beyond the volume’s own, and any per-org filtering — a partition is the unit, and it is multi-tenant by construction.
What oto tells a buyer who asks today: “The alert record — identity, every episode, the rule text at fire time, who was told, on which channel, in which thread, who acknowledged it and how it ended — is kept indefinitely and is never deleted by retention. The instant-by-instant timeline is kept for 13 months by default and is configurable up to 10 years. Raw upstream payloads are kept 30 days, and because the rejections and failed-batches feeds take no time window, those 30 days are how deep both of those screens go. There is no cold-storage export yet: if you need the timeline beyond your configured window, raise the window. The export is designed and not built.” That is a true sentence, which is the only kind worth giving a buyer.
Consequences
Section titled “Consequences”- Every install gets 16 more days of raw payloads: tens to hundreds of megabytes at realistic volume.
- The two
orgs.settingsretention keys do something for the first time. On an existing multi-tenant deployment where some org set a longer window, the firstpartitions.managetick after this change will stop dropping partitions it previously dropped, and disk will grow to the new maximum. - No migration is needed for the numbers themselves — they are Go constants and a config default. A
companion migration corrects the stale
14inoto_partitions_manage’s unused parameter default and in threeCOMMENT ON TABLEstrings, so\d+stops disagreeing with the code. - Until the export exists, the settings UI and
docs/setup/tuning.mdstate what each boundary destroys, in operator language, including that comments and unack notes are unrecoverable. - Two defects surfaced and are NOT fixed here, because both are separable and neither changes the
decision:
- ✅ CLOSED — see Amendment 1.
Nothing prunesThe pruner is now implemented, so the coupling is live, and Amendment 1 states what that made the horizon: a floor of 30 days that widens to the longestalert_event_keys. SPEC §D.4 and its own table comment promise 30 days;pruneRetentionprunes onlyingest_dedupandsessions. The table grows forever. Today this makes the 30-day raw default conservative rather than wrong — replay is currently safe indefinitely — but the moment the pruner is implemented the coupling above becomes live.raw_retention_daysany tenant set. ingest_rejectionshas no read API. Its table comment calls the per-source rejection feed “the whole point of the table” and the SPEC’s “we never silently drop” promise rests on it, but no route inopenapi.yamlreads it. A rejection nobody can look at, deleted 30 days later, is a silent drop with extra steps.
- ✅ CLOSED — see Amendment 1.
How to overturn this
Section titled “How to overturn this”This was decided without the owner, on the evidence in §1–§3 and nothing else. No customer was asked, no deployment was measured, and the alert-volume model (8 events and 3 payload appearances per firing) is reasoned from the schema rather than observed. If the owner has a real buyer with a real number, that beats all of it.
The decision is deliberately cheap to reverse, in ascending order of cost:
- The numbers are two Go constants (
identity/domain.DefaultRawRetention,DefaultEventRetention) and one config default. Changing them changes nothing else. - “Longest wins” is three functions in
internal/app:Container.effectiveRetention(the entry point, and the config floor it seeds the fold with),foldRetention(the reduce over every tenant’s settings) andwidenToSettingsCeiling(where a failed read lands — see Amendment 2). Deleting the three together restores the previous behaviour exactly. - Cold storage is scoped and unbuilt, so ruling it out costs deleting a section of this ADR and amending the buyer answer to say the product does not have one and will not.
What is not cheap to reverse is the direction: any decision that shortens a retention window and ships deletes data that cannot come back. Widening is free; narrowing is not. That asymmetry is why every fallback in the code widens. (It did not, when this was written — see Amendment 2.)
Alternatives rejected
Section titled “Alternatives rejected”Leave 14 days and document it. Rejected: nothing derives 14, the storage it saves is a rounding
error, and it is shorter than the window in which a raw payload is still replayable — so it fails the
one acceptance criterion the table exists to serve it halves the window in which criterion 36’s
replay can be run at all, on no stated reason
(Amendment 4).
90 days or 13 months of raw payloads. Affordable (1.5 GB and 6.7 GB at 10 000 firings/day), and
rejected because they buy nothing: past the nothing has asked for them. Neither of the two original reasons survives: a replay is
refused by supersession, not by age, so a 90-day payload is exactly as replayable as a 1-day one, and
the raw tables now have a read API — alert_event_keys horizon the payload cannot be replayed
safely, and there is no read API through which anyone would look at it. Storage for bytes nothing can
act on.GET /sources/{id}/rejections and GET /sources/{id}/failed-batches
(Amendment 4). An operator who
wants a longer window should raise it; widening is free and this ADR’s own asymmetry favours it.
Raise event retention to make comments outlive the timeline. Delays the loss, never fixes it, and walks straight into ADR 0014’s revisit triggers. The correct fix for irreversible loss is an export, not a longer fuse.
Move comments out of alert_events into their own unpartitioned table. Would make comments
immortal for free, and is rejected on doctrine: ADR 0003 makes the event stream the truth and every
other table a projection. A comment that is not an event is a second kind of history, and the next
question is which one the timeline renders. If cold storage is ever ruled out, this becomes the
cheapest remaining mitigation and should be reconsidered as an amendment to 0003.
Shortest-wins for the per-org fold. Honours a low setting exactly, and silently deletes another tenant’s data to do it. Never acceptable for an irreversible operation.
Parquet, or S3, in the first export. Both are the right long-term shape and both add a dependency
and a failure mode to a feature whose entire value is that it runs before a DROP TABLE. A file in a
directory can be verified by ls.
Amendment 1 — the alert_event_keys pruner exists, and the horizon is a floor
Section titled “Amendment 1 — the alert_event_keys pruner exists, and the horizon is a floor”Open defect 1 above is closed. retention.prune now sweeps alert_event_keys alongside
ingest_dedup, idempotency_claims and sessions — the rows no partition drop can reclaim.
The index alert_event_keys_prune_idx (created_at) had been carried since 00007 for exactly this
statement and served no other; migration 00043 rewrites the table comment that stated the sweep as
fact for thirty-six migrations while nothing performed it.
The horizon is not flatly 30 days, and this is the part that changed the decision above. The body
of this ADR reads the coupling in one direction — raw_retention_days derives from the key horizon —
which is right, and incomplete the moment the pruner is real. raw_retention_days is a per-org
setting bounded 1..365, and “longest wins” means the raw partitions of every tenant are kept to the
widest window any single tenant asked for. A key horizon fixed at 30 days would therefore leave an org
on 365 days holding replayable payloads for eleven months after the keys that make them replayable
were deleted, and every replay of one would append the timeline a second time and report success.
So the sweep deletes at the wider of alerts/domain.DedupeKeyRetention (30 days) and the same
Container.effectiveRetention window the partition dropper already computes — plus one day of
partition grain. That last day is not slack. oto_drop_partitions_before drops a DAILY raw partition
only once its whole range is past now − rawDays, so a payload that landed at any hour of day D stays
readable until D+rawDays+1, while a key dies exactly at created_at + horizon. Taking rawDays
straight would kill the key up to 24h before the batch it guards, and a partial batch replayed from
the failed-batch feed inside that window appends its whole timeline a second time and reports success.
The 30 remains the floor, and the path that reaches it is the deployment’s own
OTO_RETENTION_RAW_PAYLOADS, not the per-org setting: effectiveRetention seeds the fold with
Config.Retention.RawPayloads (shipped at 30 days) and only widens it, so an org on
raw_retention_days=1 never pulls the horizon down at all. An operator who drops the install-wide raw
window under 720h must not thereby unclaim the keys of episodes that are still open, whose transitions
the reconciler re-applies and which nothing else dedupes. Both directions are pinned by
TestTheDedupeKeyHorizonIsAFloorThatOnlyWidens, the grain by
TestAKeyOutlivesThePartitionHoldingItsPayload (which asserts the two jobs’ clocks against each other
rather than restating the arithmetic), and the constants’ equality by
TestTheShippedKeyHorizonCoversTheShippedRawWindow — prose stating a relationship does not fail
a build.
What it costs. One tick is bounded at 20 000 rows, which is arithmetic rather than taste: this
ADR’s own volume model is ~8 events per firing episode at 10 000 firings a day, every lifecycle
transition carries a dedupe key, and retention.prune is hourly — so the generic 500-row sweep limit
would delete 12 000 a day against 80 000 written, which is a table that still grows forever with a job
in front of it saying otherwise. 20 000 an hour is ~6× the modelled write rate: ~480 000 deletions a
day GROSS, so against the same model’s ~80 000 daily arrivals a backlog drains at ~400 000 a day NET,
while remaining a sub-second statement. That is the number to plan with — an install that has been
accumulating keys since it shipped holds tens of millions of them, so it converges over 50 to 75
days, not over “days” and certainly not in one transaction. The bound is chosen to keep each tick
cheap, not to make the first drain fast.
What it does not change. alert_event_keys leaves this ADR’s “never reaped” list at §2 and joins
the row-pruned side tables. Nothing about the event partitions, the raw partitions or the export
scope moves.
Amendment 2 — “longest wins” is three functions, and the fallbacks widen for real
Section titled “Amendment 2 — “longest wins” is three functions, and the fallbacks widen for real”“How to overturn this” named Container.effectiveRetention as the one function. The reduce now sits
beside it as foldRetention, with widenToSettingsCeiling as its fallback; the container method is
still the only entry point and still seeds the fold with Config.Retention, so deleting the three
together restores the previous behaviour exactly.
The split exists because this ADR’s closing claim — “every fallback in the code widens” — was not
true when it was written. Two reads stand between the fold and its answer, and both of them
NARROWED on failure. An unreadable tenant list returned the config floor, 30 days as shipped, because
the per-org loop never ran: one Scopes() timeout dropped 335 days of raw payloads belonging to an
org configured at the 365-day bound. An unreadable org row was worse-hidden — it continued with the
maximum it had, which is the correct answer for every tenant except the only one whose absence can
change a maximum, the one asking for the longest window, and it logged “keeping the wider window”
while doing it. Both narrowed a drop this ADR records as irreversible: oto_partitions_manage issues
DROP PARTITION, there is no soft delete, and cold-storage export is still scoped rather than built.
Both now widen to the settings ceiling — Bounds(KeyRawRetention).Max and
Bounds(KeyEventRetention).Max, the 365 days and 120 months of identity/domain.settingBounds. It is
the one number reachable without the read that cannot be narrower than the answer the read would
have given, because UpdateOrgSettings clamps every org to those same bounds. It raises rather than
assigns, so a deployment whose own OTO_RETENTION_* already exceeds a per-org bound keeps its wider
window, and a bound the table stops carrying leaves the window untouched instead of handing the
dropper a zero. The cost of the fallback is one hour of disk: the next tick reads the settings again.
TestEveryFailureInTheRetentionFoldWidensTheWindow pins both failure paths against the widest
configured org, and pins the all-readable case to an exact answer — an inequality alone is satisfied
by a fold that widens to the ceiling on every tick, which is ten years of alert_events partitions
nobody asked for. TestTheSettingsCeilingOnlyEverWidens pins the raise-never-assign direction. The
two reads are ports (app.retentionTenants, app.retentionSettings) for exactly this reason: a pool
and a service cannot be asked to fail, and a rule about failures that no test can reach is a comment —
which is what these two were.
Amendment 3 — the never-reaped list is not a partitioning backlog, and the blocker is uniqueness
Section titled “Amendment 3 — the never-reaped list is not a partitioning backlog, and the blocker is uniqueness”The §1 list gets read as a backlog. It is not one. Partitioning the high-growth tables the way the event tables are partitioned was evaluated end to end, and the answer is no for eight of them, and for most of them not ever.
The blocker is not volume and not query shape. It is that Postgres requires a unique index on a
partitioned table to include every partition key column, and on these eight the uniqueness constraint
is the product invariant. SPEC §C14 hit this once and solved it once: “Idempotency moves to two
small unpartitioned side tables: alert_event_keys and ingest_dedup.” The comment above
alert_event_keys (00007_alerts.sql:286) makes the argument in one line, about alert_events:
“because it would have to include recorded_at, and the whole point is to suppress a SECOND write at
a DIFFERENT time.”
Do not take that on trust — two statements against an empty Postgres 11 or later settle it. Declaring
UNIQUE (org_id, alert_key) on a table PARTITION BY RANGE (first_seen_at) gives
ERROR: unique constraint on partitioned table must include all partitioning columns. Add
first_seen_at to the tuple and it succeeds — and so does the partial form, where a
UNIQUE INDEX (alert_id, started_at) WHERE ended_at IS NULL partitioned on started_at is accepted
without complaint and then holds two rows with ended_at IS NULL for one alert_id, one either side
of a boundary. There is no third option: the only DDL that compiles converts a GLOBAL guarantee
into a PER-PARTITION one, which makes this a schema decision and not a tuning one.
Nine constraints across the eight tables break this way. Each row is the whole argument for that table:
| Table | The constraint a partition key would have to join | What a date buys the bug |
|---|---|---|
alerts |
alerts_key_uniq UNIQUE (org_id, alert_key) (00007_alerts.sql:54) |
The dedup identity itself. 00007_alerts.sql:101: “Dedup is enforced by alerts_key_uniq, NEVER by a read-then-write check.” An alert re-seen after its partition rolls becomes a second Alert with first_seen_at reset. |
alert_occurrences |
occ_one_open_idx UNIQUE (alert_id) WHERE ended_at IS NULL (00007_alerts.sql:187) |
Exactly two open episodes, one per side. Its own comment: “An Occurrence is a CONTIGUOUS episode; two open at once is definitionally a bug.” |
alert_occurrences |
occ_seq_uniq UNIQUE (alert_id, seq) (00007_alerts.sql:155) |
Two episodes both claiming seq = 4. |
notifications |
notifications_idem_uniq UNIQUE (org_id, idempotency_key) (00011_notification.sql:219) |
The §C.7 intent identity. An at-least-once notify.evaluate retried across a boundary mints a second intent and fans out a second Slack post. |
notification_deliveries |
deliveries_fanout_uniq UNIQUE (notification_id, channel_id, mode) (00024_delivery_mode_uniq.sql:26) |
A duplicate Slack message for the same fact on the same channel in the same mode. |
channel_threads |
threads_subject_uniq UNIQUE (channel_id, subject_kind, subject_id) (00011_notification.sql:168) |
One generation owns two threads, against ADR 0005. |
rule_snapshots |
rule_snapshots_content_uniq UNIQUE (org_id, source_id, rule_name, rule_group, rule_file, rule_fingerprint) (00040_rule_snapshots_per_key_uniq.sql:57) |
Content addressing, destroyed. §C.6 drift is “the newest snapshot differs from the one bound to the previous occurrence”, so every boundary manufactures false drift. |
alert_groups |
groups_key_gen_uniq UNIQUE (org_id, group_key, generation) (00008_grouping.sql:47) |
The widest blast radius of the set: three inbound FKs (alert_group_members.group_id, alert_occurrences.group_id, notifications.group_id). |
alert_group_members |
PRIMARY KEY (group_id, occurrence_id) (00008_grouping.sql:100) |
Weakens “one membership row per (group, occurrence)” to gain joined_at, and buys nothing. |
Relocating an invariant would still buy nothing, because none of these tables’ hot queries carry a
lower time bound. alerts_list_idx (org_id, state, last_seen_at DESC, id DESC)
(00007_alerts.sql:74) serves the §D.12(a) keyset with no floor at all, so partitioning alerts
prunes zero partitions and turns that index into an all-partition merge append. Only
del_dead_idx (org_id, created_at DESC) WHERE status = 'dead' (00011_notification.sql:313) has
org_id as its sole equality, and so is the only index in the set a range bound could prune.
What is NOT declined: notification_deliveries — it has that one prunable index and, inferred
from its growth shape rather than measured, the highest growth rate of the eight. The intuitive growth
model is wrong: it grows per fan-out target, per (notification × channel × mode), never per delivery
attempt. attempts is incremented in place (deliveries.go:288) and capped at 32 by
deliveries_attempts_ck; retry and requeue UPDATE the same row, and since 00024 each mode carries
its own, so one fact is 2 rows — not 32. A table retried hard for a week does not grow at all.
The design, so the day a trigger fires is an execution: deliveries_fanout_uniq moves off the parent
into a small unpartitioned delivery_keys keyed (notification_id, channel_id, mode), the §C14
pattern again. notifications can only follow, never lead — its FK child moves in the same migration,
and the FK needs the partition key denormalised onto it. Grain monthly on created_at, proposed
and not measured. Price the fifth parent first: oto_partitions_manage is four literal
IF to_regclass(…) blocks (00005_partitions.sql:174-211, re-emitted whole by
00036_retention_defaults.sql:40-89), and a new parameter is a Postgres overload rather than a
replacement, so it drags in the 3-arg DROP, managePartitionsSQL (internal/app/workers.go:314),
Container.managePartitions (:333), am_route_timings_test.go:507’s pronargs = 3, and
test/harness/postgres.go:271’s zero-argument call.
The set is eight; §1’s list is twelve. alert_event_keys is no longer unreaped (Amendment 1) and
silences mirrors Alertmanager’s own set. alert_quality_daily is the case that proves the rule: its
PRIMARY KEY (org_id, day, cluster_key, alertname) (00014_ops.sql:44) already carries the date,
so it is the one table here that could be partitioned without weakening anything — the blocker really
is uniqueness, not membership of this list; it stays whole because it is bounded by org × day ×
distinct alertname and is ADR 0014’s own answer to analytical scale. enrichments is the fourth and
the honest gap: unpartitioned, unreaped, growing with the occurrences it annotates, and carrying
enrichments_subject_uniq UNIQUE (subject_kind, subject_id, enricher) (00010_enrichment.sql:29),
which breaks exactly as the nine above do. It was not in the investigated set; its absence from it is
not a different answer.
The reopen condition, written down and NOT instrumented. The trigger is ADR 0014’s, unchanged and numeric, and it is restated here as a named threshold so it survives being read out of context:
- alert-list p95 above 200 ms with a warm cache,
- a rollup job overrunning its own interval,
- a single org exceeding roughly 10M events per month.
Written down is all they are, and this amendment deliberately leaves them that way. Every metric oto
exports is a counter, a queue gauge, or a write-path histogram — oto_ingest_duration_seconds,
oto_ingest_process_duration_seconds, oto_job_duration_seconds, oto_thread_head_wait_seconds,
oto_clock_skew_seconds. There is no read-path or HTTP handler latency metric of any kind, and
nothing anywhere carries a warm/cold cache dimension, so trigger 1 is unmeasured.
oto_ingest_alerts_total is an unlabelled counter of alerts accepted for processing — neither
alert_events rows nor per-org — so trigger 3 is unmeasured. Trigger 2 is the near miss:
oto_job_duration_seconds{kind="stats.rollup"} emits the duration, but the interval it must be
compared against is a schedule, not a series, so nothing evaluates the condition. That gap is stated
rather than closed on purpose: a threshold with a fabricated dashboard behind it is worse than a
threshold with none. There is no alert on these numbers, no panel, and nothing that will page.
Reopening starts by building the measurement for whichever trigger you believe has fired — the number
you bring is the first new evidence since ADR 0014 was written, so say where it came from.
What pins this. Nothing does, and that is right for a decision that changes no code: a test
asserting eight tables are unpartitioned would pin an accident rather than a choice. What is checkable
is the evidence — the nine constraints above at the file and line given, the four IF to_regclass
blocks in 00005_partitions.sql:174-211, and SPEC §C14. Drop or relocate any one of them and the row
naming it here is what stops being true.
Amendment 4 — the thirty days is chosen, and what it actually buys
Section titled “Amendment 4 — the thirty days is chosen, and what it actually buys”oto replay no longer gates on age, so nothing derives raw_retention_days any more. The gate is
supersededBy (internal/ingestion/service/replay.go): it takes the batch’s touched alert keys, which
of them the batch carries firing for, their current states and the batch’s received_at, and no age
argument at all. A replay is refused on one of two limbs — overtaken, the alert’s latest case has
source_updated_at after received_at, so a later batch already wrote to it; or closed, that case is
terminal while this batch says firing, which would mint a new case and page someone for an incident that
ended. Neither limb is a function of how old the batch is. Ticket 539bfdd and commit 7558177 are the
change; git-bug 747bde8 is the ticket that this amendment answers.
The old rule was already unreachable before that change, which is the part worth keeping. It
described a readable payload whose keys had aged out, and that state cannot occur:
dedupeKeyHorizonOf (internal/app/workers.go) keeps a key for raw_retention_days + 1 days floored
at 30, and the extra day is the DAILY partition grain — so the keys always outlive the bytes they
guard. Once the partition is dropped there are no bytes: s.batches.Get returns NotFound and
ProcessBatch skips (internal/ingestion/service/process.go), which --force cannot override.
The coupling therefore runs the opposite way to the one this ADR’s body states: the key horizon is
derived from the raw window, not the raw window from the key horizon.
The decision: 30 stays, as a chosen number, on four things that are true today
Section titled “The decision: 30 stays, as a chosen number, on four things that are true today”- It is the depth of two operator screens.
GET /api/v1/sources/{id}/rejectionsandGET /api/v1/sources/{id}/failed-batchesread the raw tables and take no time-window parameter, deliberately, because an operator asking “why did my alert never appear?” does not know when it went missing (internal/sources/api/feeds.go). Retention is the window. This is the reason §2’s “psql-only debugging artifact” is struck. - It is the whole window in which a replay can be attempted. Not because a replay stops being safe, but because past the boundary there is nothing to replay and no flag that changes it.
- It costs 51 MB at 1 000 alert firings a day and 510 MB at 10 000 (§3, measured at 1 698 B per 3-alert batch). Ninety days costs 153 MB and 1.5 GB. No number in this range decides anything, so storage does not pick this default and is not offered as if it did.
- Lowering it is irreversible and raising it is free, and 30 already shipped. That asymmetry —
this ADR’s own — is what breaks the tie between 30 and any other affordable number, and it is why
the answer to “we want longer” is
raw_retention_days, not an amendment.
What it is NOT: one calendar month of replay safety. A three-month-old failed batch is replayable if its alerts have not moved; a two-hour-old one is refused if they have. The number bounds how long the bytes exist, and nothing else.
What still holds the 30-day alert_event_keys floor is the second half of
alerts/domain.DedupeKeyRetention’s own comment, which never depended on replay: the reconciler
re-applies transitions, and that table is the only thing that dedupes them, so an operator who drops
the install-wide raw window to a day must not thereby unclaim the keys of cases that are still open.
The two thirties are now independent numbers that happen to be equal, not one number stated twice.
Debt this amendment recorded, and what has since been paid. Nine sites stated the retired derivation as fact, one of them a test. All nine are corrected except the migration, which has already run:
- ✅
internal/app/retention_horizon_test.go— the test kept its assertion and lost its false name:TestTheShippedRawRetentionIsStillDerivedFromTheKeyHorizonis nowTestTheShippedKeyHorizonCoversTheShippedRawWindow, which is what the equality actually proves — keys outlive the bytes they guard. - ✅
internal/platform/tuning/defaults.go(DefaultRawRetention),internal/identity/domain/org.go,internal/identity/domain/settings.go(theWhystring the settings API serves),internal/alerts/domain/event.go(DedupeKeyRetention’s “IT IS THE PRIMARY NUMBER” is now “an independent number”),internal/sources/api/feeds.go,deploy/helm/oto/values.yaml,docs/setup/tuning.md. - ✅
api/openapi/openapi.yaml— theraw_retention_daysdescription, plus the separate falsehood beside it: it told consumers no operation in this contract readsingest_batchesoringest_rejectionswhile/api/v1/sources/{id}/rejectionsand/api/v1/sources/{id}/failed-batchesin the same file do. It now names them. - ⛔ NOT paid, deliberately:
db/migrations/00036_retention_defaults.sql’sCOMMENT ON TABLEstill says “THE 30 IS DERIVED, NOT CHOSEN” and “there is no API that reads this table”. It is an applied migration and is not amended in place; the correction belongs in a new migration that rewrites the comment.
SPEC §D.3/§D.4 and the tuning UI copy were corrected with this amendment; the docs-site mirror needs
a sync-docs run to follow.