Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary

Status: Accepted · 2026-08-08 Superseded in part by: 0052 — its refusal of incident objects and the PERMANENTLY-OUT classification of incidents. 0052 admits the grouping (an Incident drawn over Cases) and keeps the response out; this ADR’s handoff contract is not superseded, and 0052 is built on it. Superseded in part by: 0054 — FR-1’s refusal of facts about a response effort, for the one noun Remedy. Every other FR-1 refusal stands. Doctrine: docs/design/SCOPE-BOUNDARY.md (the full test, 32 worked verdicts, the audit delta, the slippery-slope map and the handoff contract). This ADR records the decision; that document is what engineers cite when refusing a feature. Resolves: red-team memo §7, open decision 3 — “Is oto alert-first forever, or does it grow incidents and on-call?” Applied by name in: 0036 — AlertOccurrence becomes AlertCase, which is the first time a word adjacent to §A.1’s scope-ban family has been argued through FR-1 rather than admitted because the list did not happen to contain it. Its §2 grants the word with six limits, in the same shape as this ADR’s anti-rota clause; §5.6’s column ban re-points at alert_cases and AC-50’s alternation grows ^case$, case_status and priority to enforce them. Read that ADR before permitting the next borderline noun: the precedent it sets is the argument, not the word.

Context

Section titled “Context”

Alert management and incident management look alike from a distance. Both store timestamped facts about bad things, both notify humans, both draw timelines, both have an “acknowledge” button. Every one of oto’s competitors sits on one side of a line neither of them names: keep.dev’s product opinion is “alerts are raw material, incidents are the unit of work”; incident.io, FireHydrant and Rootly start where a human becomes responsible; PagerDuty starts where a human is woken up.

oto bets the opposite: the signal is the unit of record, and the work happens elsewhere. That is a defensible product opinion, not a gap anyone overlooked — and defending it requires a boundary an engineer can apply in thirty seconds, because the boundary will be tested one small pull request at a time. It is never “let’s build incident management”; it is always “it’s just one nullable column”.

Three specific pressures made this urgent now, before any code exists:

  1. SPEC.md §A.1 bans the word incident, while §I.1 lists a module named incidents as DEFERRED-POST-V1. The spec contradicts itself, and the contradiction is a propped-open door.
  2. escalate_after_s, escalation.check and Reason='escalation' already exist. They are defensible as specified — one reminder, same channel — but “escalation” is PagerDuty’s word for a ladder, and the schema is one array literal away from being one.
  3. oncall and analytics (charter: “MTTR/MTTA rollups”) are classified DEFERRED-POST-V1, which reads as not yet. For these two the answer is not ever, and the classification says otherwise.

R8 already answers a narrow version of this question (no per-person metrics). It needs a general form.

Decision

Section titled “Decision”

oto is alert-first permanently. The boundary is the Flight Recorder Test:

FR-1. Complete the sentence “this row is a fact about ______.” A signal (Alert, Occurrence, Group, RuleSnapshot, Notification, Delivery, Source, or an aggregate over those) → IN. A person, team, rota, responsibility, response effort, ticket or customer-facing statement → OUT. Human actions are IN only as timestamped annotations on a signal’s timeline, carrying an actor as metadata. In one line: actor, never subject.

Three secondary heuristics, applied in order when FR-1 does not decide cleanly:

  • H-1 Obligation. Does it create, route to, or discharge an obligation on a named human? → OUT.
  • H-2 Orphan. Delete every alert; does the artefact still mean something? → OUT. (Config is exempt.)
  • H-3 Write-path. Does it change any system outside oto’s DB and its channels? → OUT of v1, on safety grounds — a separate axis from FR-1, and the only one that is earnable.

FR-1 separates acked_by = alice (a fact about the occurrence) from assigned_to = alice (a fact about Alice). Identical columns; opposite products. It also derives R8 for free.

Consequential rulings (delta list in SCOPE-BOUNDARY §5, applied by another agent):

  • escalation → unacked_reminder throughout: reason enum, notifications_reason_ck, escalate_after_s, the openapi field, the worker and the package. Bound as exactly one stage, forever, same channel_ids — a scalar that must never become an array. escalation joins the §A.1 banned words.
  • mention_on_escalation → mention_on_reminder, restricted to usergroups and !here/!channel. Individual <@U…> mentions are dropped: a static list of individuals oto pings is a rota with the schedule hard-coded. ⚠️ SUPERSEDED — see the amendment below. Individual mentions ARE permitted.
  • Module map: incidents and oncall move from DEFERRED-POST-V1 to PERMANENTLY OUT; the legitimate part of incidents is renamed correlation (machine-derived groupings over signals, no human create endpoint, no human-set state). analytics loses MTTA from its charter entirely.
  • Column ban on alerts / alert_occurrences / alert_groups: no assigned_to, owner_id, watchers, incident_id, ticket_id, sla_due_at, or any present-tense person reference. acked_by is past-tense attribution and is the only person reference permitted on a signal row.
  • No human-writable signal state. There is no POST /alerts/{id}/resolve, ever. state is owned by Alertmanager and mirrored by oto (C2); ack_state is the only state axis a human may write. Currently true by omission — omission is not enforcement, so it becomes an explicit §E.1 principle.
  • stats and alert_quality_daily are KEPT unchanged and gain one guard: no time_to_ack, acked_seconds or any column measuring the interval between a machine event and a human action.
  • AC-49: a lint rule greps internal/, web/src/ and db/migrations/ for the banned vocabulary. Bans that are not mechanically enforced decay in a quarter — the same argument §I.3 makes about layering rules.

The handoff is part of the decision, not an afterthought. oto is an upstream producer of signal facts and never a downstream consumer of incident state. It exposes durable identity (alert_key, group_key — stable across alertmanager.yml edits), the generic oto.notification.v1 webhook envelope, the full read API and SSE with durable resume, stable deep links, and rule provenance at fire time — the one artefact no incident platform has and all of them want at postmortem. Exactly two inbound human verbs exist: acknowledge and comment. An incident tool marking its incident resolved MUST NOT resolve oto’s alert.

PagerDuty and incident.io are Channel providers, not scope changes. A destination for a fact about a signal is IN by FR-1; R5 defers the implementation, it does not exclude it. oto pages nobody; oto tells the thing that pages.

Consequences

Section titled “Consequences”
  • A future engineer can rule on a feature request in thirty seconds and cite a clause number. The refusal is a design property, not a taste argument.
  • Assignment, rotas, paging, incident objects, status pages, postmortems, war rooms, MTTA/SLA reporting, human-set severity, triage verbs and manual resolve are permanently out. So is reading Slack conversation or ticket state back into oto.
  • The buyer is capped: oto can never be the only tool a team purchases. This is stated out loud and turned into a coexistence pitch rather than hidden (memo §5e).
  • Enterprise procurement will ask for MTTA and be told the metric is unrepresentable by construction. That cost lands on sales, not on the engineers who use the product daily.
  • Ephemeral presence (“3 people are viewing this alert”, derived from live SSE connections, never persisted) is the sanctioned answer to “who’s on it” — it delivers the human value with zero stored facts about people.
  • One dissent is recorded rather than suppressed: snooze is missing and that is a defect. It is not deferred, not refused, not discussed — merely absent, which is worse, because absence is not a decision anyone can revisit. Suppressing oto’s own notifications for an alert_key until T passes FR-1, H-1, H-2 and H-3 cleanly; it is nearer to channels.verbosity than to a silence, and it is the manual sibling of the flap and storm damping the spec already ships as visible states. Without it the only quiet button a user has is muting the Slack channel — which also hides the real incident, the failure mode the memo calls fatal. A product whose ethic is “default to quiet” that ships no quiet button is inconsistent with itself. SCOPE-BOUNDARY §8 argues it belongs in v1.

Amendment — 2026-08-08: individual @-mentions on mention_on_reminder are permitted

Section titled “Amendment — 2026-08-08: individual @-mentions on mention_on_reminder are permitted”

⛔ SUPERSEDED IN FULL — 2026-08-20, git-bug bd0fb1d. THE SUBJECT OF THIS AMENDMENT NO LONGER EXISTS. The owner withdrew the unacked reminder outright and ruled that the mention goes with it: “when we are dropping the entire reminder flow whats the point of mention_on_reminder”. There is no mention_on_reminder key, no mention audience, and no mention surface anywhere in oto — internal/channels/providers/slack/config.go:55 says “⛔ THERE IS NO mention_on_reminder HERE, AND ITS REMOVAL IS A BUG FIX”, and SPEC §L.5.1’s schema no longer declares it. Read the section below as history.

⭐ AND IT TIGHTENS THIS ADR RATHER THAN LOOSENING IT. §5’s original delta ruling REFUSED individual mentions on H-1 grounds — oto must not name a responder. The amendment overruled that. The withdrawal restores the stricter position by removing the surface entirely, so H-1 is now satisfied by construction rather than by argument. The five numbered guarantees below all still hold; four of them hold vacuously.

This amends the §5 delta ruling above. The product owner overruled it. The code is correct as shipped; this section exists because the ADR was the stale half of the disagreement.

internal/channels/providers/slack/schema.json permits <@U…> / <@W…> alongside <!subteam^S…>, !here and !channel, capped at maxItems: 10. Do not “fix” the schema or the pattern to match the original ruling. They are the decision.

The reasoning. The original ruling banned individual mentions because “a static list of individuals oto pings is a rota with the schedule hard-coded”. The analogy is seductive and it is wrong on the mechanism. What makes a rota a rota is that it is time-aware and it moves — the whole product is the schedule, the handover and the “who is on now” query. A fixed list of two names in a channel’s config is none of those things. It is the same act as typing @alice into the channel by hand, which is what the person configuring oto will do anyway.

And that is the operative point: refusing @alice is paternalism users route around. The routes are all worse than the thing being refused — a Slack workflow that reposts oto’s messages with a mention, a usergroup created with one member in it and never updated, or a channel muted entirely because the one signal that mattered did not reach anyone. Each of those is less visible to oto, less legible to the next engineer, and more likely to rot than a two-name list in a config field oto validates.

Meanwhile the door the boundary actually guards stays shut. FR-1 is not “oto never names a person”; it is “oto stores no present-tense fact about a person”. mention_on_reminder is a property of the channel, not of any human in it: nobody is assigned, nobody is on call, no state is written to a person, and no row anywhere records that Alice is responsible for anything.

The anti-rota clause, which is the binding part of this amendment. The permission is granted with these limits, and they are what keep it from becoming the thing it was feared to be:

  1. Capped at a small N. maxItems: 10 today. A list that wants to be long is a list that wants to be a rota; the cap is the tell, and it fires before the concept arrives.
  2. Never time-aware. No time_of_day, no days_of_week, no timezone, no second list, no alternation, no “primary/secondary”. These are the actual rota primitives and every one of them remains banned.
  3. Fixed, not derived. The list is written by a human into a channel’s config. It must never be computed from a schedule, an external system, or oto’s own data — the moment oto decides who to mention, oto is doing on-call.
  4. A mention is text, not a route. It renders into one message on one already-routed reminder. It creates no delivery, no retry, no acknowledgement obligation, no second stage — §G.9.1’s one stage, forever, is untouched.
  5. Nothing is stored about the mentioned person. No mentioned_at, no ack attribution derived from the mention, no per-person counter anywhere. R8 is unchanged and unchangeable.

What did not change. escalation is still a banned word — and now with no replacement term at all, since the reminder it was reshaped into was withdrawn too. unacked_reminder_after_s is still a scalar and still one stage. ⛔ There is no unacked_reminder_after_s: migration 00068 deleted it and there are ZERO stages, not one (git-bug bd0fb1d). notification_policies.channel_ids still references channels and nothing else. Multi-stage escalation is still refused, and this amendment is the argument for why it can be: letting a team mention the two people who actually care removes the pressure that would otherwise be spent arguing for a ladder.

Alternatives rejected

Section titled “Alternatives rejected”
  • “Stay alert-first” as a stated intention, without a mechanical test. This is the status quo and it is exactly what fails: intentions do not survive twenty individually reasonable pull requests. The spec already demonstrates the decay — a banned word and a module bearing that name coexist in one binding document.
  • Draw the line at “no on-call and no paging.” Too far out. It permits assignment, incident objects, ticket sync and MTTA reporting, all of which cross the line long before telephony does, and each of which drags paging in behind it.
  • Draw the line at “read-only, no writes anywhere.” Wrong axis. That is H-3, a safety rule with its own expiry date (R3 is earnable). Conflating it with FR-1 would either permanently forbid snooze, which is legitimately in scope, or make the incident-management refusals look negotiable.
  • Grow into the archived Grafana OnCall vacuum (memo §1, the largest named market opening). It is real, and it is refused deliberately: it drags in telephony vendors, 24/7 reliability obligations and compliance that a small self-hosted project cannot meet, and it would put oto head-on against PagerDuty instead of alongside it. The vacuum oto fills is the self-hosted OSS alert-history and routing UI, which is the other half of what OnCall’s archival left behind.