Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0052 — An Incident is drawn over Cases, and its response is handed off

Status: Accepted · 2026-10-02 — settled in a design session with the owner, who authorised building it the same day; Accepted when the delta list at the end was applied (git-bug ee9f306). §4’s at-most-one-Incident clause is still flagged for the owner’s review, and Accepting the ADR does not rule on it. Supersedes in part: 0013 — its refusal of incident objects and its classification of incidents as PERMANENTLY OUT. 0013’s handoff contract is not superseded; this ADR is built on it. SCOPE-BOUNDARY SS-3 and verdict #33. Amends: 0045 — “N alerts, N conversations” gains one exception. Relates to: 0039 (a grace window returns, for a different entity), 0042 §3 and 0044 (the authority test every grouping here passes), 0053, 0054.

Context

Section titled “Context”

SS-3 refused incident objects partly on the ground that “AlertGroup is already the multi-alert container.” Migration 00069 deleted alert_groups, and ADR 0045 accepted that a storm of N alerts opens N conversations. oto now has no object that spans more than one alert, so the pressure SS-3 described — forty alerts, and people wanting one thing to talk about — is stronger than when SS-3 was written. An Investigator (0053) adds a second pressure: it needs the whole storm to say anything useful, not forty fragments of it.

What SS-3 refused is still refused. The owner ruled out every part of the response — a lead, a status such as “mitigated”, human-set severity, comms, the write-up — on the ground that better platforms already do it. What is admitted is the grouping, and the handoff of the response to the platform that manages it.

Decision

Section titled “Decision”

1. An Incident is a set of one or more Cases, drawn as one story

Section titled “1. An Incident is a set of one or more Cases, drawn as one story”

An Alert has Cases; an Incident spans Cases. The noun is Incident because that is what operators will call it whatever the glossary says (Arc was considered and rejected for exactly that reason).

2. Two authors, and a model is not one of them

Section titled “2. Two authors, and a model is not one of them”
  • A Correlator — matchers over Cases, optionally a count over a window — written by an operator. “Any severity=critical Case” draws one-Case Incidents; “≥5 Cases in one (cluster, namespace) within 10 minutes” draws many-Case ones.
  • A human, recorded as actor metadata.
  • The Investigator only proposes membership, as a Suggestion. Every Incident therefore has an answer to “why is this an Incident?” that is a rule someone wrote or a person who decided.

3. Inside oto, an Incident’s state is read off its signals

Section titled “3. Inside oto, an Incident’s state is read off its signals”

An Incident is active while any member Case is open and quiet otherwise. No human writes it. “Quiet but not fixed” and “fixed but still noisy” are facts about the response and live only in the external tool. The two may disagree, and that is correct: they describe different things.

4. Membership over time

Section titled “4. Membership over time”
  • A later Case matching the Correlator joins while the Incident is active, and within the Correlator’s operator-written quiet_grace after it goes quiet; past that, it draws a new Incident. A flapping alert therefore does not draw ten Incidents overnight.
  • A human-drawn Incident never grows by itself.
  • A human may add or remove Cases on any Incident; a Case a human removed is never re-added by a Correlator.
  • A Case belongs to at most one Incident. When several Correlators match one Case, the first in the operator’s Correlator order draws it; a human may move it to another Incident. Two memberships would mean two conversations to post a Case’s facts into and two external incidents to send them to, with no rule to choose between them. ⚠️ Decided while the owner was away, to unblock implementation; flagged for review.

group_close_delay_s and refire_grace_s left with AlertGroup (00069, 00071) because their entity was wrong — a grouping no operator could configure — not because a grace window is. This one is a number an operator wrote, which is the test 0044 set.

5. Declared outbound, facts only

Section titled “5. Declared outbound, facts only”
  • Declaring is a notification. subject_kinds gains incident; “every Incident goes to incident.io” is one catch-all policy routed to an incident-tool Channel. Retries, idempotency, delivery audit and drills come with it. An org with no such policy has Incidents in oto and sends nothing.
  • The outbound mapping binds an Incident to (destination, external incident id), the way a ChannelThread binds a Conversation to a Slack root, so later facts become updates on the same external incident.
  • The provider is generic (owner’s ruling, 2026-10-02). No vendor-specific incident provider ships. The generic webhook’s incident.id is stable for the Incident’s life and is the de-duplication key a receiver keys on, so oto’s own key is the mapping. If a receiver’s 2xx response carries external_url / external_id, oto records them once as the receipt of a delivery — the way it records a Slack ts — and shows the link; that is not reading the incident back. Severity, where a tool needs one, is the receiver’s mapping from the member alerts’ own labels, never a value oto invents. Recipes for specific tools are documentation, not code.
  • oto sends facts, never commands: drawn, member added or removed, quiet, active again, new Finding, and Remedy transitions (0054). It never sends PagerDuty’s resolve or any close or status change. An org that wants auto-resolve writes it in its own tool, keyed on oto’s metadata. This is 0013’s rule run the other way: the incident tool does not resolve oto’s alert, and oto does not resolve the incident tool’s incident. ⚠️ Narrowed by 0055 §4: oto’s own code and oto’s mapping catalog never send a command, but an operator-written payload mapping on a webhook Connection may turn a fact into one — the operator’s rule, not oto’s.
  • Every fact carries a per-Incident sequence — 1 for drawn, allocated in the transaction that records the fact, reused on its retries — so a receiver orders facts that arrive out of order (owner’s ruling, 2026-10-04; 00093).
  • Nothing is read back. Two-way sync is not considered.

6. An Incident may be a conversation

Section titled “6. An Incident may be a conversation”

Per Correlator, an operator may say its Incidents are conversations: conversation_kind gains incident, and later facts about member Cases post into the Incident’s thread instead of opening their own. Nothing is held back to wait for a Correlator, and nothing already posted moves. This narrows 0045 to “N alerts, N conversations — unless an operator-written Correlator says they are one story.” It passes 0042 §3 where AlertGroup did not: the decision about what shares a message is one the org wrote.

Consequences

Section titled “Consequences”
  • oto’s counts and the incident tool’s counts can differ: oto draws Incidents for analysis and conversation as well as for declaration. A narrowing policy decides which go out.
  • An operator who connects auto-resolve to quiet reopens scenario 1 for themselves. The provider docs must say so, because auto-resolve is PagerDuty’s default.

Delta list (applied 2026-10-02, git-bug ee9f306)

Section titled “Delta list (applied 2026-10-02, git-bug ee9f306)”
  • SCOPE-BOUNDARY: SS-3, verdict #33, and the PERMANENTLY-OUT entry for incidents.
  • SPEC §I.1 / §I.1.1: correlation and incidents rows; subject_kinds, conversation_kind, notifications_subjkind_ck, threads_subjkind_ck, policies_subjkinds_ck.
  • CONTEXT.md: the incident scope ban, §1’s “permanently not an incident manager” (still true of the response — reword, do not delete), and the ⚠️ markers on Incident, Correlator and Conversation.
  • tools/lintvocab: incident_id stays banned on signal rows; the bare word is admitted. Applied as: the rule fires only where the column would sit on alerts, alert_cases, notifications or notification_deliveries, so an Incident’s membership table keys on incident_id legally. The bare word was never in lintvocab’s list — only in the prose bans and AC-49’s grep, both amended.
  • SPEC §D.8’s DDL (subject_kinds, conversation_kind and their four CHECKs) is amended by the migration that widens them, not ahead of it; §I.1 records the widening as part of the module.
  • ADR 0013 and ADR 0045: a pointer to this ADR.