Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so

Status: Accepted — in full, §5’s precedence rank included. The rank was recorded as an implementation decision pending ratification when this ADR was written; the owner ratified it on 2026-08-20 and it is now a ruling like the rest of this document. Date: 2026-08-20 Resolves: git-bug 7570090, stage 6’s remaining half — the replacement that ticket names for hardcoded flap detection. Migrations: 00072_a_policy_binds_a_subject_and_counts_it.sql — notification_policies gains subject_kinds, count_min, count_window_s and four CHECKs; and 00073_a_floor_can_record_its_own_silence.sql — notifications_suppmap_ck widens from six values to seven, plus notif_policy_idx. Amends: SPEC §B.8.2 (the precedence chain, six values → seven), §D.8 (the two tables’ DDL), §H.6 and the NotificationSuppressedReason schema in api/openapi/openapi.yaml. Relates to: 0042 — this is the ADR a reader of 0042 needs in order to be told why a new damper is admissible at all. 0042 is not superseded: §3’s ruling that oto may not withhold on its own judgement is the test this axis had to pass, and §4’s “nothing replaces it at the notification layer” is unchanged, because what arrives here is a policy column, not a detector.

⚠️ §5’S RANK IS AN IMPLEMENTER’S CHOICE AND IS LABELLED AS ONE THROUGHOUT. below_threshold ranks between throttled and verbosity in the shipped chain. The argument for that position — ceiling before floor — is stated in §5 because a chain with no argument is worse than a chain with a reviewable one, but it is reasoning produced by this change and not a ruling recovered from anywhere. No SPEC clause, no earlier ADR and no ticket ranks a ceiling against a floor; the question was not asked before this axis existed. The value’s ADMISSIBILITY (§2–§4) is ticket-authorised and is the decision this document records; its POSITION is a choice made in order to ship, offered here to be overturned cheaply. Overturning it moves one slice literal (suppressorOrder), one SPEC line and one contract description, and breaks nothing stored.

1. The decision

Section titled “1. The decision”

A notification policy may carry a count condition: stay silent until at least count_min of these have happened inside count_window_s. When the count falls short, oto records the notification and sends nothing, with suppressed_reason = 'below_threshold'.

The vocabulary therefore holds seven suppression reasons, not six:

channel_disabled → no_policy → snoozed → throttled → below_threshold → verbosity → duplicate_render

What is counted is fixed by the schema rather than by a comment: the distinct subjects of the one subject_kinds value the policy binds, for that policy, over the sliding window [now - count_window_s, now], including the fact being evaluated and including rows this same floor already suppressed. So five re-fires of one Case count once, and a policy holding its tongue still climbs towards its own threshold.

2. It is throttled’s dual, and that is what makes it cheap

Section titled “2. It is throttled’s dual, and that is what makes it cheap”

The same two policy fields, read with the opposite sense:

says reads
throttled “you have already been told this many times in this window” a ceiling
below_threshold “this has not happened enough times in this window yet” a floor

A policy may carry both, and a single evaluation may be refused by either. Nothing new is measured and nothing new is detected: the evaluator counts rows in notifications, which is why 00073’s only structural addition is an index — notif_policy_idx ON notifications (org_id, policy_id, created_at DESC) — the same shape notif_conversation_idx already has for the throttle. notifications.policy_id has carried an FK since 00011 and no index; a foreign key creates none on the referencing side.

3. Why this is admissible when flapping and storm were not

Section titled “3. Why this is admissible when flapping and storm were not”

This is the test the value had to pass, and it is the only reason this ADR exists rather than a line in a migration header.

ADR 0042 deleted storm, and ADR 0041 Amendment 1 deleted the flapping damper with it (migration 00059 narrowed notifications_suppmap_ck to six for both), on one principle: oto may be quiet for a reason outside its own judgement — nowhere to send, a human asking for less, a provider rate-limiting, nothing changed — and its own opinion that a real firing was not worth mentioning is not on that list. flapping compared against DefaultFlapThreshold = 5 over DefaultFlapWindow = 7200 s: constants welded into Go, which no operator could see, read back or change. That is oto holding an opinion about somebody else’s signal.

below_threshold produces a silence with the same shape — this keeps happening, say nothing yet — and the only thing separating it from the value the codebase spent two ADRs deleting is where the number comes from. It comes from notification_policies.count_min over count_window_s: two columns an operator wrote, can read back through GET /api/v1/notification-policies, and can clear with one PATCH. The silence it produces is the one they asked for, by name, on a policy they can see.

A damper whose threshold is the operator’s is their request. A damper whose threshold is ours is our opinion. That is the line, it is the line ADR 0042 drew, and this axis is on the permitted side of it. Every value in the seven is still either the absence of a destination (channel_disabled, no_policy), a human asking for less (snoozed, verbosity, throttled, below_threshold), or nothing to say (duplicate_render). Not one is a judgement, and nothing may add one back.

4. The count needs a unit, which is why the binding arrives in the same migration

Section titled “4. The count needs a unit, which is why the binding arrives in the same migration”

00072 adds subject_kinds and the count together rather than in two migrations, because a count is a number of somethings and the something has to be named.

  • policies_count_subject_ck CHECK (count_min IS NULL OR cardinality(subject_kinds) = 1). A count condition on an unrestricted binding has no unit; on a two-kind binding it has two, and summing an alert-subject fact (an identity, true across every firing it ever had) with a case-subject fact (one firing episode) produces a number that is about nothing.
  • policies_count_pair_ck CHECK ((count_min IS NULL) = (count_window_s IS NULL)) — symmetric, unlike policies_digest_pair_ck. A digest’s window alone is a complete instruction; neither half of a count condition means anything alone. A threshold over unbounded history is not evaluable, and a window with no threshold counts facts and compares the number against nothing. Both would silently mute or silently un-mute a channel.
  • count_min is 2..10000. Two, because the fact being evaluated is itself inside the window, so a threshold of one clears unconditionally and states no condition — the same argument that keeps zero out of policies_digest_floor_ck.
  • count_window_s is 60..86400 — the throttle’s window bound, not the digest’s, and with no divisor rule, because this window is not tiled against the UTC day.
  • subject_kinds is NOT NULL DEFAULT '{}' and empty means every kind, which is what every row written before 00072 says and is today’s behaviour exactly. The direction matters: the failure mode on this path is a no_policy suppression, not an error anybody sees.

5. The rank — ratified

Section titled “5. The rank — ratified”

below_threshold sits directly below throttled and above verbosity. The reasoning, offered as reasoning:

  • The two are the same two policy columns read with opposite senses, so they belong adjacent, and both belong above verbosity, which is a property of a destination rather than of the policy.
  • The ceiling ranks first of the two because a spent cap is the active fact — oto has been speaking about this conversation and has stopped, against a number the operator has already been hit by — whereas an unmet floor is the ordinary resting state of every policy that carries one: for most of its window a count condition is unmet by design. A resting state that outranked an active damper would mask it on every policy carrying both, which is the same argument that already puts verbosity below throttled.

✅ RATIFIED BY THE OWNER, 2026-08-20. The history is kept deliberately, because it is the part a future reader needs: “ceiling before floor” appeared in no SPEC clause and in no earlier ADR when this change shipped. It was this change’s answer to a question nothing had asked, and it was recorded as a proposal — here, in suppression.go’s suppressorOrder comment, in SPEC §B.8.2 and in migration 00073 — precisely so it could not be mistaken for a ruling recovered from the design record. The owner then made it one. So the rank above is now authority, and this section is where it comes from — not SPEC §B.8.2, which records it, and not suppressorOrder, which implements it.

Reversing it remains cheap by construction, and that stays true after ratification: one slice literal, one SPEC line, one contract description and this section. Nothing stored depends on the order — the chain decides which reason a suppressed row records at write time, and a row already written keeps the reason it was written with. A reversal is therefore a new ADR superseding this section, not a migration.

6. Consequences

Section titled “6. Consequences”
  • notifications_suppmap_ck admits seven values. 00073’s ADD CONSTRAINT cannot fail on existing data — it only widens — and its Down refuses if any row records below_threshold, with the query to inspect them. Deleting a recorded suppression deletes the only evidence oto chose silence, so the rollback makes that a deliberate act rather than a way to get the migration to run.
  • A new operator-facing axis is a new thing to explain. The reason appears in the API contract with its own prose, in SPEC §B.8.2’s chain, and in §D.8’s column comment and CHECK. §N.1/§N.4’s three-places rule applies to the pair count_min / count_window_s exactly as it applies to any bound: the DTO validate tag, the domain constructor and the DDL CHECK, or TestValidatorMatchesDDL fails.
  • It is not flapping under a new name and must not be described as one. The two produce the same shape of quiet by a different author, and the operator-facing copy says whose number it is.
  • subject_kinds overlaps reasons, and the overlap is conceded. Reason.Subject() is total, so as a filter the binding is derivable from a hand-narrowed reason list. It earns its column as the count’s unit, and as a declaration an operator can read without knowing the Reason-to-subject map by heart.

7. What this ADR does not decide

Section titled “7. What this ADR does not decide”
  • The rank. §5, explicitly.
  • Whether a count condition should ever be able to schedule. It may not, ever (SCOPE-BOUNDARY §4.8). subject_kinds is an altitude, not a calendar.
  • Whether oto ever regains an automatic damper. It does not. This axis is admissible because the threshold is the operator’s; a future value whose threshold is oto’s is refused by ADR 0042 §3 and by this document’s §3, and neither needs re-litigating.