Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0012 — Pastel product chrome, saturated colour reserved exclusively for state

Status: Accepted · 2026-08-08 · accent choice superseded by ADR 0030 (2026-08-16); the two-tier architecture below is unchanged

Context

Section titled “Context”

The owner named the product oto (音, Japanese for sound — a chime) and requires a light, pastel palette. That is a real and defensible position for an alerting tool: this category’s interfaces are uniformly loud, and loudness at rest is how people learn to ignore a channel.

But pastel and alerting are in genuine conflict, and the conflict cannot be dodged:

  • A pastel red at ~3:1 against a white surface fails WCAG AA (4.5:1) for body text.
  • Even where it technically passes, low-chroma red carries no urgency. A critical state that reads as “gentle pink” is a safety failure in a product whose output is 3am interruptions.
  • ~8 % of men have a red/green colour vision deficiency. Low-saturation state colours collapse toward each other far faster than saturated ones.

Resolving this by darkening everything destroys the calm. Resolving it by keeping everything pastel makes oto unsafe. Neither is acceptable.

Decision

Section titled “Decision”

A strict two-tier colour system.

Tier A — chrome is pastel. Backgrounds, surfaces, borders, navigation, tables, panels, form controls, charts’ gridlines. Low chroma, high lightness (light) / low lightness (dark). Chrome never uses a state hue. The brand accent is a periwinkle (#5B54D6 / #A6A0FF) chosen specifically because it is not adjacent to any state hue. (2026-08-16: the accent is now a beni crimson-pink — see ADR 0030 — for the same reason, not a different one.)

Tier B — saturated colour means state, and nothing else. No decorative accent, no chart series, no hover effect may use a Tier-B hue. Scarcity is what makes it loud: when a saturated colour appears, it means exactly one thing.

Each state ships four tokens rather than one:

Token Role Obligation
-fill pastel tinted surface (row background, badge) —
-border hairline / 3 px status bar ≥ 3:1 vs page background
-text dark (light mode) / pale (dark mode) text on that fill ≥ 4.5:1 vs its own fill
-solid saturated accent: severity dot, status bar, chart mark ≥ 3:1 vs page background

So a critical row is a pastel fill with a saturated 3 px left bar and dark red text — calm at a distance, unmistakable at a glance, legible for everyone.

Supporting rules: colour is never the only channel (every state carries ≥ 2 of colour / icon / text label); severity is carried by the icon, state by the colour; dark mode is the default; no flashing ever, and the single 2 s pulse on unacked-critical is removed under prefers-reduced-motion.

Full token sets with measured contrast ratios for every text-on-surface pair, light and dark, are in SPEC §M.4–M.5, and are asserted by web/src/design/contrast.test.ts plus an axe-core Playwright run in both themes.

The Slack Block Kit palette (§H.2) is a separate, unchanged system. The Grafana OnCall values (#a30200, #daa038, #dddddd, #2eb886) stay exactly as they are.

Consequences

Section titled “Consequences”
  • Every state needs four values instead of one, and every new state costs a contrast measurement. That is the price of being both calm and safe, and it is paid once.
  • Charts get a dedicated neutral/brand ramp and may never plot with state hues, which costs some expressiveness in dashboards.
  • Designers cannot reach for “a nice red” for anything decorative. This will feel restrictive and is the point.
  • The two colour systems (UI and Slack) will look different side by side in a screenshot. That is accepted and documented, so nobody “fixes” it later.

Alternatives rejected

Section titled “Alternatives rejected”
  • Pastel everywhere including state: fails WCAG AA for text on tinted rows and blunts urgency. Unsafe.
  • Saturated everywhere (the category default): loses the brand and, more importantly, loses the signal — when everything is loud, nothing is.
  • Pastel fills with pastel text, relying on the icon for legibility: shifts the entire accessibility burden onto iconography and fails for anyone using a screen magnifier or a low-quality display.
  • Harmonising the Slack palette with the UI palette: would replace the best-tested open-source alert palette in existence with untested pastels, on a substrate (someone else’s Slack theme) oto does not control and cannot measure. Trading correctness for coherence is the wrong trade in this product.