Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0056 — A Case the upstream stopped speaking about expires, and says why

Status: Accepted · 2026-10-05 — the owner’s ruling, after asking for a manual “End case” button and being shown SCOPE-BOUNDARY verdict #34. Holds: CONTEXT.md’s door “No human writes a signal’s state” and verdict #34 (no manual resolve), unchanged. Nothing here gives a person a way to end a Case. Extends: the reaper (SPEC §B.4) and alert_cases.resolve_reason. Relates to: 0006 (the reconciler, and its “warn loudly” on send_resolved: false), 0040, 0044 (the authority test §3 passes). Amended: 2026-10-05, same day — Amendment 1 (HA clusters expire on all of their sources; existing sources start with the expiry off; the deletion waits a resolve grace; cluster_id is immutable; the two new expiries ship behind a flag).

Context

Section titled “Context”

The owner wanted to end stale Cases by hand. Investigation found why they are stale. A Case ends only by an upstream resolve or by the reaper, and the reaper misses two common shapes:

  1. No upstream end time. Alertmanager zeroes endsAt on firing webhooks; only a reconcile pass that sees the alert live sets source_ends_at. A Case known only from webhooks is never a reap candidate.
  2. A deleted source. The reaper finds no single live source for the Case and holds it forever.

It also found that nothing shows either: last_observed_at and source_ends_at are in the API and rendered nowhere, and source_health.warnings (where send_resolved: false is recorded) never reaches the UI.

Decision

Section titled “Decision”

1. Staleness is shown

Section titled “1. Staleness is shown”

A Case shows how long since upstream last said anything about it, and, when it cannot expire, why: no upstream end time, or its source is held (not healthy) or gone. A source shows its warnings and how many Cases it holds. Facts only; no state changes.

2. A deleted source’s open Cases expire as source_removed

Section titled “2. A deleted source’s open Cases expire as source_removed”

Deleting a source is an act an operator took on configuration. Its open Cases close with resolve_reason = 'source_removed': the one thing that could have told oto they ended is gone, and oto says exactly that rather than holding them open on a promise nobody can keep.

3. A source may set a max silence; past it, open Cases expire as silent

Section titled “3. A source may set a max silence; past it, open Cases expire as silent”

Each source carries an operator-written max_silence_s (default 24h for a new source, editable; NULL turns it off; an existing source starts off — Amendment 1). An open Case under a healthy source with no upstream observation for longer than that closes with resolve_reason = 'silent'. Alertmanager repeats a firing notification every repeat_interval (4h by default), so a day of silence means upstream stopped speaking about it. The health guard still applies: under an unhealthy source oto cannot tell “resolved” from “Alertmanager is down”, so it holds, and §1 shows that it is holding.

4. Both read as expired

Section titled “4. Both read as expired”

The four alert states are unchanged. A Case closed with timeout, silent or source_removed is expired; only upstream is resolved. The reason is shown wherever the state is, because “ended because the source went silent for a day” is not the same fact as “resolved”.

Consequences

Section titled “Consequences”
  • resolve_reason’s CHECK and its domain enum widen from two values to four.
  • An org with long-firing alerts whose Alertmanager repeat_interval exceeds 24h must raise max_silence_s, or those Cases expire while still firing. The source settings say so next to the field.

Amendment 1 — 2026-10-05: HA clusters, existing sources and the release flag

Section titled “Amendment 1 — 2026-10-05: HA clusters, existing sources and the release flag”

The owner’s rulings R1–R3 and the judged fixes to the reaper, landed in a62ac3f. Each one narrows §2 or §3; none gives a person a way to end a Case.

R1 — an HA cluster’s Cases expire, on all of its sources

Section titled “R1 — an HA cluster’s Cases expire, on all of its sources”

§1–§3 were written as if a cluster had one source, and the reaper acted only under exactly one live source, so every Alertmanager HA pair’s Cases were held forever. Now timeout and silent may end a Case only when its cluster has at least one live source and every live source is healthy (SPEC §B.4). An HA pair is two witnesses: the replica oto cannot see might be the one still carrying the alert, so one unhealthy replica holds every open Case on the cluster, and a held Case names only the replicas the guard could not vouch for, never a healthy sibling.

silent’s threshold is the cluster’s effective max silence: the longest max_silence_s among its live sources, or off when any live source’s is NULL. Off on one replica is off for the cluster, because an operator who turned it off for a source did not consent to its sibling’s number ending that source’s alerts. Every expiry, timeout included, re-reads the cluster’s live set inside its transaction and stands down unless it equals the set the guard proved.

R2 — existing sources start with max silence off

Section titled “R2 — existing sources start with max silence off”

Migration 00094 adds max_silence_s bare and only then sets its default to 86400. Every source that exists when it runs reads NULL (off); a source registered afterwards takes a day. This is what keeps ADR 0044 §3’s line true for this threshold: the reaper acts only on a number an operator wrote. A day is offered — the default on a new source, which an operator registering it can read back and change, and the value the settings field shows when someone turns it on for an existing one — and nothing is backfilled, so no Case that already exists is swept away on the first tick by a threshold nobody chose for its source. The bounds are 1 hour (below Alertmanager’s own default repeat_interval, so a mistyped minute count is refused rather than expiring every live Case) to 30 days.

B5 — source_removed waits a resolve grace after the deletion

Section titled “B5 — source_removed waits a resolve grace after the deletion”

§2’s Cases expire once no live source feeds the cluster and the newest removal is a resolve_grace old, checked in the scan and again in the expiring transaction. A source registered on the cluster inside that grace — the usual shape of “delete it and re-add it with the right URL” — stands the expiry down. Deleting one replica of an HA pair ends nothing: the other still speaks for every Case they shared.

R3 — cluster_id is immutable

Section titled “R3 — cluster_id is immutable”

A source’s cluster is what the reaper’s live-set test is about, so moving a source between clusters would silently make one cluster’s Cases source_removed-eligible and hand another a witness it never had. PATCH /api/v1/sources/{id} refuses cluster_id by name, as it refuses kind; re-home a source by registering a new one and deleting the old.

B2 — the two new expiries ship behind a flag, off

Section titled “B2 — the two new expiries ship behind a flag, off”

silent and source_removed run only when jobs.expire_silent_and_removed (OTO_JOBS_EXPIRE_SILENT_AND_REMOVED) is true; it defaults to false in this release. Off, case.reap is exactly the timeout sweep it was before 00094. timeout is not behind the flag. The rollout is two-phase, expand/contract (CONTEXT.md §6):

  1. Ship the release. Migration 00094 and every reader that can spell silent and source_removed (API, UI, channel cards, webhook consumers) go out together, with the writer dark. Nothing produces either reason yet.
  2. Turn the flag on once every replica runs this release and webhook consumers have been told the resolve_reason set widened (docs/setup/webhook.md). A later release may flip the default.

⚠️ Once the flag has been on, rolling back below this release needs goose down 00094 first. The older release cannot spell either reason; 00094’s Down rewrites them to timeout (still expired) before narrowing the CHECK. Rolling back with the flag never turned on needs nothing: no row carries either reason.

The UI cannot see the flag — the API does not expose it — so every silent and source_removed forecast on screen is worded “if enabled” rather than promising an expiry the reaper may not run.