Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0051 — A `raw` template owns the whole message, and a template may carry its replies

Status: Accepted · 2026-10-02 Amends: 0050 — its rule that the colour is never the author’s, for raw only, and its “Owed” items on thread replies and on raw validation Relates to: 0008 (why block_ids and metadata stay oto’s), 0012 (the state palette oto’s own card keeps)

Context

Section titled “Context”

An operator rebuilt the resolved card as a raw template and found most of the message was not theirs to write. The bar stayed green on a resolved digest alert because oto paints state. The [RESOLVED] … on cluster line above the card was oto’s. {{ links.group }} came out as a private-use handle and {{ case.started_at }} as a marked timestamp, because only the card parser and the text speller resolved those. {{ actions }} meant nothing in JSON. A hand-written button failed outbound validation, and that failure killed the delivery — the one outcome ADR 0050 calls the most important property of the feature. A summary with a " in it broke the JSON, and the error told the author to use a | json filter that has never existed.

The owner’s ruling: oto should let an author change every element of the message, and raw is what that is for. The author also wanted the thread replies, which no template reached.

Decision

Section titled “Decision”

raw owns the message

Section titled “raw owns the message”

A raw body may be a bare array of blocks, {"blocks": [...]}, or the message itself:

{ "text": "…the push notification…", "color": "#a30200", "blocks": [ … ] }
Element card / text raw
Blocks the author’s the author’s
Bar colour oto’s (state) the author’s; one Slack cannot parse is oto’s
Top-level text the document’s plain words the author’s; absent is oto’s
Where the buttons go {{ actions }} a {"type": "oto_actions"} block
Links, times, emphasis resolved resolved — spelled per string, mrkdwn text objects get Slack’s
block_id, metadata, button action_ids and values oto’s oto’s

The last row is the line, and it is drawn at what nobody sees and everything depends on. A block_id is read back off an interaction; message metadata traces a click to its delivery; an action_id an author could rewrite is an alert nobody can acknowledge from Slack.

The colour rule ADR 0050 stated — “a template that could paint a firing card green could lie” — is withdrawn for raw, deliberately. It is still true. It is also the author’s lie to tell in a format whose whole promise is that they wrote the message, and the operator who asked for it wanted the bar to mean severity, which oto’s palette cannot say. card and text keep oto’s colour.

Values are escaped for Slack and for JSON at binding, and the author’s text is not

Section titled “Values are escaped for Slack and for JSON at binding, and the author’s text is not”

A raw template is Block Kit typed by hand, so nothing downstream can tell the author’s <https://grafana|Grafana> from a label that says <!channel>. The value is defused where it is still a value: & < > become Slack’s entities, then the string is escaped as JSON-string content. A quote or a newline in an annotation no longer breaks the document, and no | json filter is needed. | upper and | lower step around both kinds of escape, so changing a value’s case cannot change what it means.

A stray }} is ordinary JSON and is no longer refused in raw. Every other delimiter mistake still is.

A template that Slack would refuse costs the template, not the alert

Section titled “A template that Slack would refuse costs the template, not the alert”

render/slack runs Validate on a templated payload before choosing it. A failure falls back to oto’s own message, as every other template failure already did. This closes 0050’s “Owed” item on raw validation and restores its unconditional promise.

A template may carry its thread replies

Section titled “A template may carry its thread replies”

notification_templates.reply_source (migration 00082) is a second body, in the template’s own format, rendered for thread replies with reason bound. NULL is oto’s own replies — every template written before it. A body that renders nothing for a reason keeps oto’s reply for that reason, so restyling one reply is one {% if reason == 'all_resolved' %} around the body.

It lives on the template, not beside it, and the policy still names one template. A card and its replies are one voice, edited in one dialog (two tabs), versioned together — version bumps when reply_source changes — and attributed to one revision on the delivery row. The alternative, a second picker on the policy, would let a card and its replies drift apart and was declined.

Save-time validation renders a reply body against one fixture per reply reason (template.ReplyFixtures) as well as the ordinary corpus. Rendering nothing is not reported, and the missing-{{ actions }} warning does not apply. The preview returns reply_renderings, one row per reason, where a spelling with neither text nor error reads as “oto’s own reply”.

Consequences

Section titled “Consequences”
  • oto’s own card, replies and both golden corpora are byte-for-byte unchanged. Everything here is reachable only through a template.
  • A raw author can now make a resolved card red. That is the author’s call, made in the one format that says it is.
  • Digests still take no template. The webhook renderer still ignores reply_source, because it renders no thread replies from a template today.
  • An author-written button that isn’t oto_actions must still pass Slack’s rules and oto’s action_id pattern; one that doesn’t sends oto’s own message rather than a broken one.

Owed

Section titled “Owed”
  • Digests take no template.
  • A Dialect registry (carried over from 0050).