Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0023 — Terminal thread states are decided first, and the send is three phases

Status: Accepted · 2026-08-09 · Narrowed once — frozen left the thread state vocabulary on 2026-08-19 (git-bug e5c060b, migration 00066); the decision itself is intact and the gate order is now dead → unsequenced → already-resolved → root → predecessor → proceed. See the callout under Decision. Amends: SPEC §G.7 (which specified the opposite order, and a single-transaction send) Relates to: 0005 (the thread a generation owns), 0008 (update-in-place, which is why a duplicate root is the expensive failure), 0021 (the audit that found both)

Context

Section titled “Context”

Ordering inside one ChannelThread is enforced by a sequence gate: a delivery may send only while holding the thread’s advisory lock and only when thread_seq == last_sent_seq + 1. Two decisions inside that gate turned out to be load-bearing in a way the SPEC had backwards.

1. The switch tested “no root yet” before “the thread is dead.” As written, a delivery whose mode needed a root and whose thread had no provider_thread_id snoozed for two seconds — and the dead check sat below it, unreachable for exactly the deliveries it existed to rescue.

The witness is one Slack error. A root delivery gets channel_not_found; §H.9 marks the thread dead; the thread still has a null provider_thread_id, because nothing ever landed. Every reply, every update and every reminder queued behind it now matches case 1 and snoozes. A River snooze consumes no attempt, by design — an item waiting its turn has not failed — so the attempt ceiling, the one bound that would eventually turn this into a dead-letter, can never fire. The thread waits at 0.5 Hz for a root that nothing left in it will ever post. No message, no dead row, no delivery.skipped event, no metric: the destination is silent and the alert page says pending. For a product whose stated line is “oto’s silence must never be indistinguishable from no alert”, that is the worst reachable state.

Worse, it was not a rare interleaving. It needs no race at all — one terminal error on one root message is the entire reproduction.

2. The provider call sat inside the transaction that advanced the sequence. The theory was that committing the send and last_sent_seq together made ordering atomic. It cannot: a network call and a database write are not committable together, and the attempt inverted the failure it was meant to prevent. A cancelled context or a failed COMMIT after chat.postMessage returned rolled the claim back to pending, un-incremented attempts, and let the job run again — posting a second root into somebody’s channel — while erasing the sending status that is the only thing §G.5’s ambiguity latch keys on. The one crash window it was designed to close was the window it opened, silently, and under ADR 0008’s update-in-place model a duplicate root reads to everyone in the channel as a second incident.

It also held a pooled connection and a thread-wide lock for the duration of a call that a rate-limited destination is entitled to keep waiting for thirty seconds.

Decision

Section titled “Decision”

Terminal states are decided first. The gate’s evaluation order is: dead → frozen → unsequenced → already-resolved → root → predecessor → proceed. A dead thread yields recover_thread, never a snooze; a frozen thread yields abandon. Only then do the waiting conditions get a chance to answer.

⚠️ NARROWED, 2026-08-19 — frozen NO LONGER EXISTS (git-bug e5c060b, migration 00066). Not superseded: nothing in this ADR is falsified. The decision it records — terminal states first — stands and is why dead is still tested before every waiting condition. But frozen was never reachable: Freeze had no production caller, the group-close sweep never called it, and no row could hold the state. It has left threads_state_ck, the two ThreadState vocabularies, the ThreadStore port and the gate, so the order is now dead → unsequenced → already-resolved → root → predecessor → proceed, and abandon/thread_frozen is no longer an emittable verdict on oto_thread_order_decisions_total. The argument above is left as written, because it is the record of what was decided in August 2026 and half of it is still load-bearing.

Two bounds close the same hole from the other side:

  • A head with no root recovers immediately. If an item is at last_sent_seq + 1, needs a root and there is none, every earlier slot is already resolved — so nothing left in the thread can post one. That is not a wait, it is a wedge, and it is answered now rather than in fifteen minutes.
  • MaxWait = 15 minutes, measured from the delivery’s created_at, converts either waiting verdict into recover_thread (root_never_landed, head_of_line_stalled). It exists precisely because a snooze consumes no attempt: without it nothing bounds a wait at all. Fifteen minutes is set against Alertmanager’s repeat_interval floor — an item still stuck past it will be superseded by a fresher notification anyway.

MaxWait bounds the wait, not the wedge, so the contract is completed by a terminal outcome: a recover_thread that sweeps nothing, and finds no delivery owning the head, dead-letters the item (status='dead', error_class='permanent'), advances the head past its slot and appends the event. Answering a fruitless recovery with another snooze is the original bug in a different costume.

The send is three phases, with the provider call between two transactions:
TX 1 lock → read the thread under it → re-derive the mode → decide → claim (status='sending', attempts+1) → render → persist the bytes → COMMIT
— call the provider. No transaction, no pooled connection, no lock held
TX 2 lock again → markSent guarded on status='sending' → move last_sent_seq

The claim is durable before the call, so a crash anywhere afterwards leaves a row that says this may have been sent — which is exactly what §G.5’s 120 s lease and the ambiguous latch consume. Ordering survives the shorter lock because ordering was never the lock’s job: last_sent_seq does not move until TX 2, so everything behind this item still sees itself out of turn. The lock serialises deciding, not sending.

One further rule falls out of the same failure and is stated here because it is what makes the ordering decision meaningful: the mode is re-derived once, before the gate is asked, and the gate and the sender share that single value. The gate reading the row’s stale mode while the sender re-derived its own was a second, independent route to the same silent thread.

Consequences

Section titled “Consequences”
  • Exactly-once is not claimed, and the SPEC now says where the duplicate can appear: a labelled ambiguous re-send of a post_root after a lost claim. Under-delivering a firing alert is worse than a visible, marked duplicate.
  • A send that lands but cannot be recorded — the lease expired mid-call and another worker took the row — records nothing and increments oto_delivery_claim_lost_total. That counter is an alert, not a statistic: every increment is a message in a channel with no sent row behind it.
  • A thread can now reach a terminal outcome that says “oto gave up” on the alert page. That is a visible failure where there used to be an invisible one, and it will be seen by operators.
  • The gate’s verdict vocabulary is closed and is a metric label (oto_thread_order_decisions_total{action,reason}), so every wedge class is countable rather than inferred from silence.
  • TX 2 runs on a non-cancellable context with a 30 s budget. Shutdown is therefore slightly slower than a hard stop, deliberately: losing the row that records a delivered message is how oto forgets a send and repeats it.

Alternatives rejected

Section titled “Alternatives rejected”

Keep the SPEC’s order and rely on the retry ceiling. This was the implicit design, and it does not work for one reason: a snooze consumes no attempt. Making snoozes consume attempts instead would bound the wait — and would spend the retry budget of every message queued behind a busy thread on waiting rather than on sending, killing exactly the deliveries a heavily-used channel is trying hardest to deliver.

Sweep the thread on a timer from lifecycle. A periodic reaper advancing stuck heads would also end the wedge, and it was rejected because it makes liveness a property of a background job rather than of the dispatch path. The dispatcher already holds the lock and has already read the thread; the sweep costs a handful of indexed reads there, and a defect in it fails visibly on the next send instead of quietly at 60 s intervals.

Skip the stalled slot instead of re-enqueuing its delivery. Simplest possible recovery, and wrong: skipping a slot that may still be in flight publishes a reply before the message it replies to, and skipping a possibly-sent root orphans every reply behind it. Recovery refuses to move past an unresolved slot and instead names the delivery that owns the head, which the worker re-enqueues.

Two-phase commit, or an outbox for the provider call. The honest way to make a network call and a commit atomic, and unavailable: Slack has no idempotency key and no prepare phase, and oto never reads Slack back (C9). An outbox moves the same ambiguity one hop without removing it.

Keep the call inside the transaction but shorten the timeout. Narrows the duplicate-root window without closing it, and keeps a pooled connection and a thread-wide lock pinned for the length of a provider call. A smaller version of a failure that should not exist is still that failure.