Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0055 — An incident-tool integration is data or a bridge, never code inside oto

Status: Accepted · 2026-10-02 — the owner’s ruling in a design session. Revised the same day (§2): the vendor mapping first landed as a json NotificationTemplate and was moved onto the webhook Connection; the reason is kept in §6. Amends: 0052 §5 — “oto never sends a resolve” narrows to “oto’s own code and oto’s catalog never send one” (§4). Extends: 0047 — a webhook Connection may carry a payload mapping. Leaves untouched: 0050 and 0051 — templates stay wording. Relates to: 0054 (commands refused in oto’s process — the same reason applies here), 0044 (the authority test §4 rests on).

Context

Section titled “Context”

The owner ruled that oto ships no vendor-specific incident provider (0052 §5, amended), and asked for a plugin ecosystem built on the events oto already gives out. Those events exist: the versioned oto.notification.v1 envelope on the generic webhook, with retries, idempotency, delivery audit and drills, and a stable incident.id on every Incident fact. What does not exist is a way to reach a vendor whose API wants a different JSON shape, without writing a service.

Decision

Section titled “Decision”

1. The event contract is the plugin API

Section titled “1. The event contract is the plugin API”

oto.notification.v1 is the surface every integration builds on. It already signs each body (X-Oto-Signature, HMAC with the Connection’s secret); it gains what an API built on by strangers needs on top: secret rotation, a signed timestamp against replay, and a written compatibility promise — additive changes only within v1; a removal or a change of meaning is v2, sent side by side for a stated period.

As built (git-bug 2765f74; the owner’s rulings of 2026-10-02):

  • The signing secret has its own slot on the Connection (migration 00088, signing_credential_id), beside the authentication credential rather than instead of it, so a receiver that needs a bearer token — incident.io’s — can also verify a signature.
  • Rotation overlaps for 24 hours. A new secret signs at once; the one it replaces keeps signing beside it for 24 hours, so X-Oto-Signature carries two v1= entries in that window and never more than two.
  • The signed base is v1: + X-Oto-Timestamp + : + body, HMAC-SHA256, sent as X-Oto-Signature: v1=<hex>. The timestamp is per attempt and is what lets a receiver refuse a replay.
  • This is a one-time breaking change, accepted by the owner. Until it, the header was sha256=<hex> over the body alone, and it was never documented. A receiver written against it stops verifying; the setup guide and the release that ships this say so. From here the format moves only under the promise below: a new scheme is a new entry beside v1=.
  • The stated period is 180 days — the owner’s ruling. When v2 ships, v1 and v2 are both sent for 180 days, and the end date is announced before v1 stops; the same holds for a new signature scheme.

2. A plugin is data: a payload mapping on the webhook Connection

Section titled “2. A plugin is data: a payload mapping on the webhook Connection”

A webhook Connection may carry a payload mapping: a document that renders the request body and headers from the envelope for each fact, and may name where in a 2xx response the external incident’s URL and id are found (a path expression), feeding the outbound mapping of 0052 §5. A Connection without one sends the plain oto.notification.v1 envelope, exactly as before.

It is destination setup, configured with the Connection in Settings → Connections, and it is held to that surface’s stakes, not a template’s:

  • Checked before it is saved. It is rendered against a sample envelope for every fact type, and a test send is offered; a mapping that does not render cannot be saved.
  • A failure is a failed delivery. A mapping that fails at send time marks the delivery failed, visible and retryable in the delivery audit. It never falls back to the plain envelope, which the vendor could not parse — that would turn a broken mapping into a missing incident.
  • Every interpolated value is JSON-escaped, with no opt-out.
  • A mapping never holds a secret. A vendor key that must travel in the body (PagerDuty’s routing_key) is a sealed credential on the Connection, which the mapping refers to by name and oto fills in at send time; a mapping file is safe to share in the catalog because it carries none.

A catalog of community mappings is a folder of files — contributed, reviewed and imported onto a Connection; no code ships with one.

A value a tool requires and the envelope cannot always supply — PagerDuty’s severity, for an Incident whose labels give none — is the operator’s to choose at import, not the catalog’s (owner ruling, 2026-10-02). The catalog file declares it as a choice over the values the tool accepts and writes a placeholder where it goes; the import asks, and writes the literal pick into the Connection’s copy. A copy that still holds a placeholder cannot be saved, and the catalog check refuses a file that hard-codes the fallback instead.

A mapping cannot decline a fact, so which facts reach a tool is the notification policy’s business. A tool that opens a new incident for an event on the key of one a human resolved — PagerDuty and incident.io both do — would be paged again by any later fact; the starters’ setup routes them only drawn and active_again (owner ruling, 2026-10-02: fixed by policy, with no new schema).

3. Anything that needs logic is a bridge, outside oto

Section titled “3. Anything that needs logic is a bridge, outside oto”

OAuth flows, several calls per fact, a vendor’s REST API for notes: a small service the operator runs, which receives oto’s webhook. It runs out of oto’s process, in any language. oto documents the pattern; it does not host bridges.

4. An operator’s mapping may turn a fact into a command; oto’s never does

Section titled “4. An operator’s mapping may turn a fact into a command; oto’s never does”

An operator may write a mapping that turns quiet into a vendor’s resolve. That is a rule the operator wrote, keyed on a fact oto stated — 0044’s test, passed — and it is the operator’s, not oto’s. oto’s own code sends no command, and no mapping in oto’s catalog turns a fact into a resolve, close or status change; a catalog review refuses one. The docs say plainly that quiet is not fixed, and that a mapping which resolves on quiet ends a response the moment the signals stop.

5. Refused: code plugins inside oto

Section titled “5. Refused: code plugins inside oto”

Go plugins, and WASM runtimes (wazero, Extism) running operator- or community-supplied code in oto’s process, are refused. oto’s process holds every org’s channel credentials and its database; 0054 refused running commands there for the same reason. Data (§2) and bridges (§3) cover the need without it.

6. Why the mapping is not a template

Section titled “6. Why the mapping is not a template”

The first draft made the mapping a fourth NotificationTemplate format, json. The owner refused it: a template is wording — how a message reads — and 0050 makes it safe to edit by falling back to oto’s built-in card on any failure. A vendor mapping decides what the incident tool does. On the template surface, a wording edit could switch “resolve on quiet” on, and the harmless fallback would become a silent missed incident. Where a fact goes and in what wire format is the destination’s business, so the mapping lives with the destination.

Consequences

Section titled “Consequences”
  • The vendor matrix is the catalog’s problem, not the code’s: adding a tool is a pull request of one file.
  • 0052 §5’s sentence “oto never sends a resolve” is narrowed by §4 and must be re-read with it.
  • Templates keep three formats; nothing about them changes.