Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0047 — A channel answers to a connection, and Slack gets two scopes back

Status: Accepted · 2026-08-21 Relates to: 0018 (the manifest this amends), 0008, 0034 (where a channel is created)

Context

Section titled “Context”

A channels row was two things wearing one name: an org-wide credential — “the bot token for Slack workspace T123” — and one destination — “#sre-alerts”. Every Slack channel carried its own sealed bot token, so five channels in one workspace meant the same token pasted five times into five separate settings-screen dialogs, and there was nowhere to set the workspace up once and point several channels at it. The settings screen that created channels was also, necessarily, the screen where the org-wide setup happened — an admin surface and a per-destination one, sharing a form.

Two smaller defects rode along with the bigger one. First, channels.config.team_id and channels.config.conversation_id were both required, but nothing resolved one from the other: an operator typed a Slack channel ID by hand, copied out of Slack’s own “Copy link” menu, with no verification that it named the channel they thought it did. Second, moving channel creation out of admin Settings and onto the Notification Policy screen — where an operator is already naming a destination for a routing rule — had nowhere to put the org-wide credential if channels kept carrying their own.

Decision

Section titled “Decision”

A channel_connections row is the org-wide setup — a Slack workspace’s bot token and team id, or a webhook receiver family’s shared credential. A channels row is one destination, and references a connection by id (channels.connection_id, NOT NULL). Several channels share one connection.

  • Settings → Channels becomes Settings → Connections. An admin sets up a Slack workspace or a webhook receiver family once. Nothing about one destination — a specific #channel, a specific URL — is configured there any more.
  • Channel creation moves to the Notification Policy screen. That is where an operator is already naming a destination for a routing rule; requiring a trip to admin Settings for every new #channel was the cost this decision removes. Creating a channel there means picking an existing connection — admin setup stays admin setup, only the destination itself is created inline.
  • channel_id and channel_name resolve each other, live, for Slack. Type a name, get the id back read-only; type an id, get the name back read-only. This is genuinely new behaviour, and it costs something specific — see below.
  • Both providers get a Connection, not just Slack. A webhook connection may carry none, basic, or bearer (the same three kinds a channel used to carry directly, now shared across every URL under one connection), or a new kind, webhook_signing_secret: an HMAC-SHA256 secret the webhook provider signs the outbound JSON body with (X-Oto-Signature: sha256=<hex>), so a receiver can verify a payload actually came from oto. This is the direction ADR 0018’s Slack signing secret already covers for interactivity; the webhook provider had nothing equivalent for its own outbound sends.
  • Migration is breaking, no backfill. Every existing channels and channel_credentials row is dropped (migration 00075). There is no way to group existing rows into connections without guessing which shared a workspace, and this is pre-release.

This reopens a closed scope decision, on purpose

Section titled “This reopens a closed scope decision, on purpose”

ADR 0018 fixed the Slack API interface at three methods — chat.postMessage, chat.update, auth.test — and the manifest states, at length, that channels:read/groups:read and conversations.info were requested, then removed after the first live run showed they served a Channel.Probe and a Channel.ResolveConversation that nothing called. The manifest’s own words: “A scope an app cannot use is a scope an admin approved for nothing.”

Name↔id resolution needs exactly the method that reasoning removed. conversations.info answers id→name; there is no Slack method that answers name→id at all except walking conversations.list, which needs the same two scopes. Restoring them is not an oversight repeating itself — the method now has a real caller, Provider.ResolveConversation (internal/channels/providers/slack/resolve.go), invoked from a settings-time HTTP endpoint (POST /api/v1/channel-connections/{id}/slack/resolve) that a human’s typing triggers. The distinction ADR 0018 drew — “oto never reads Slack to reconstruct its own state” (C9, ADR 0008) — is unchanged: this is metadata lookup at configuration time, once, when a channel is named, not a read on the delivery path and not a probe nothing calls.

The cost is real and is not hidden: every self-hosted install that upgrades past this migration adds two scopes to its Slack app and re-pastes deploy/slack/manifest.yaml, which a reviewer at that company reads again. channels:read/groups:read remain workspace-wide metadata reads — they see every channel’s name and archive state, not just ones oto is configured against — and that breadth is the whole reason they were cut once already. The trade this ADR makes is: a bidirectional name/id control on the settings screen, permanently, is worth two metadata scopes, once, per install.

Consequences

Section titled “Consequences”
  • A shared credential is set up once. Five Slack channels in one workspace now share one bot token, entered once. Rotating it rotates every channel under it in one PATCH.
  • The scope-removal narrative in providers/slack/channel.go, provider.go, and the manifest had to be rewritten, not appended to. Those files stated “three methods, one scope” as settled fact in several places; leaving the old prose next to the new methods would have the code disagree with its own comments about what it does. All three are corrected in this change.
  • A connection still referenced by a channel cannot be deleted (ON DELETE RESTRICT on channels.connection_id, surfaced as a 409 naming the channels, the same shape ADR 0034’s channel/policy 409 already established one hop over).
  • The cross-table invariant “a channel’s type must equal its connection’s type” has no CHECK constraint — two tables, and Postgres cannot express a CHECK across them. channels/api enforces it at create and update, the same way checkRenderer already catches a cross-provider renderer channels_rend_ck cannot see.
  • A webhook connection’s shared credential is one row, one kind. A connection needing both a bearer token (to authenticate INTO the receiver) and a signing secret (to let the receiver verify payloads FROM oto) cannot have both today — channel_connections.credential_id is a single FK. Nothing in v1 needs that combination; a connection that does would need a second credential slot, deliberately deferred rather than built for a case that does not exist yet.

Alternatives rejected

Section titled “Alternatives rejected”

Keep the credential on the channel, add a “copy from another channel” convenience in the UI. Copies a value into a second row rather than sharing one; rotating the token still means finding and re-editing every channel that copied it. Rejected: it treats the symptom (retyping) and leaves the actual defect (N independent credentials for one workspace) in place.

Resolve only id→name (keep channels:read/groups:read off, use conversations.info alone via some other credential-free path). There is no credential-free path to conversations.info — it is a scoped Slack API call regardless of direction — so this saves nothing over the two-scope restoration and delivers only half the control: an operator who has the name but not the id, which is the common case when setting up a new channel, gets nothing.

Leave channel creation in Settings, alongside connections. Rejected per the redesign’s own premise: if channel creation stays in admin Settings, every new destination is an admin-settings round trip even though the operator defining a notification policy is the one who knows which #channel they want. Moving it to the Notification Policy screen is what makes the connection/channel split pay for itself in fewer clicks, not just cleaner data.