Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0039 — Group close stays activity-driven

⛔ SUPERSEDED — THE OBJECT OF THIS DECISION IS DELETED. alert_groups is gone, root and branch (git-bug 7570090, migration 00069): there is no generation to close, so nothing is activity-driven or otherwise. group_close_delay_s, whose meaning this ADR was written to keep, left with it (git-bug 7287b28, migration 00071). The conversation is now the Case, and a conversation holds exactly one — so what ends a conversation is the Case ending, and there is no delay in it.

⭐ The reasoning is kept because the shape recurs: this ADR refused to implement a clause verbatim on the grounds that doing so would delete a mechanism the clause never mentioned. That is the same failure mode as reading a “stated purpose” as a “whole purpose”, which git-bug 7570090 went on to pay for twice.

Status: Accepted · 2026-08-18

Amends: 0038 — records what the derived key deliberately did not change. ADR 0005’s thread-ownership claim survives unamended.

Context

Section titled “Context”

git-bug fe73f9a collapsed alert_group_members into alert_cases.group_id. Its Done when went one clause further: group close should become “no live case with this split key” rather than last_activity_at plus a delay. That clause was not implemented, and an omission with no written reason is indistinguishable from an oversight. This ADR is the reason.

The clause is attractive because, after 00051, membership is derived rather than recorded — so “are there live cases?” is now a cheap, always-true question rather than a join against a table that could drift. The temptation is to make close ask exactly that question and nothing else.

Decision

Section titled “Decision”

CloseIdle stays driven by last_activity_at. The rollup’s refusal to close over a live member stays as the safety property. Three grounds:

  1. Taken literally, the clause deletes group_close_delay. A generation whose cases have all ended would close on the next sweep. An alert that resolves and re-fires inside the delay — the ordinary flapping case the delay exists for — would then find its generation gone and open a new one, and a new generation is a new Slack thread that cannot be re-parented. The clause trades a correctness property nobody reported broken for a visible regression in the product’s most visible output.

  2. The safety goal the clause states is already held. CanCloseAt refuses while counts.Live() > 0, over a rollup re-derived inside CloseIdle’s own transaction. What 00051 changed is that this refusal became true rather than nominal: the firing/suppressed counts now derive from alert_cases.ended_at, which the state machine writes on every T5/T6. Before 00051 the same check consulted a recorded membership row that could lag; after it, the check cannot be stale. The ticket’s motivation was satisfied by the part that shipped.

  3. The two rules differ only when the delay is doing its job. “No live case” and “no live case and idle for group_close_delay” agree on every generation except one that has gone quiet within the window — which is precisely the generation the delay exists to hold open.

Consequences

Section titled “Consequences”
  • group_close_delay keeps its meaning, and a generation may sit open with zero live cases for the length of the window. This is the re-fire grace, not a leak.
  • The safety property is now asserted, not assumed: a test pins that a generation holding a live case is refused closure (Held == 1) rather than merely that the sweep ran.
  • fe73f9a is closed as partially implemented, deliberately. The unbuilt clause is recorded here rather than left as a silent gap in the ticket’s Done-when.
  • If the delay is ever shown to hold generations open long enough to matter, the fix is to tune the delay — an org-wide knob that already exists — not to remove it.

Alternatives rejected

Section titled “Alternatives rejected”
considered rejected because
Implement the clause verbatim deletes group_close_delay; every short resolve/re-fire strands a Slack thread
Close on “no live case” but keep the delay as a second condition that is the behaviour already shipped, written twice
Close immediately and re-parent the thread on re-fire Slack has no re-parent; a thread’s identity is its first message
Leave the omission undocumented in the ticket an unexplained gap reads as an oversight and invites the next agent to “fix” it