Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

oto_slack_unknown_action_total
Type counter
Labels none, deliberately (see below)
Registered in internal/channels/service/metrics.go
Alertable yes — any increment. Both causes are oto’s fault
Rule OtoSlackUnknownAction in deploy/prometheus/oto-rules.yaml

SPEC §H.8 spells this metric slack_unknown_action_total, without the prefix. The code adds the oto_ prefix on purpose — the SPEC line is an omission, not a second rule, and an unprefixed metric would be the only one in oto’s namespace a scrape config could not select with {__name__=~"oto_.*"}.

What it counts

Section titled “What it counts”

Verified Slack interactions naming an action_id oto has no branch for. The signature was valid, the payload was genuine, a human really did press the button — and oto had nowhere to route it.

The known ids are the four that write — oto.ack, oto.unack, oto.snooze, oto.unsnooze — plus oto.more, plus anything under the oto.noop. prefix (the URL buttons, which Slack delivers an interaction for and oto must acknowledge with nothing to do).

⭐ oto.more was in NO set at all and every press of it counted here. It is the links overflow — the :package: More menu holding Timeline, Prometheus, Alertmanager, Rule history and Show all labels — and the renderer has always emitted it (render/slack/root.go, pinned by five goldens and named in SPEC §H.8). Matching neither the writing ids nor oto.noop., it fell to the router’s default arm and incremented this counter on the one element of the card that was working exactly as designed. It is now admitted to the closed set, deliberately rather than renamed to oto.noop.more: cards already sitting in Slack carry the literal oto.more, so a rename would turn every one of those menus into a silent no-op — the defect relocated, not fixed — and the overflow is not link-only anyway (see below). If you are reading this runbook because of a historical spike, oto.more is the first thing to rule out.

⭐ oto.more IS THE ONE ID WHOSE ARM IS INERT FOR SOME OPTIONS AND NOT OTHERS, AND THE VALUE IS WHAT TELLS THEM APART. Its four URL options (Timeline, Prometheus, Alertmanager, Rule history) carry a url, so Slack navigates them itself: oto acks and does nothing else, and a press of one enqueues no job. Show all labels carries a value — labels|<case id> — and is the one option that asks oto to render something, so a press of it enqueues a slack.interaction job and is answered with an ephemeral message listing the alert’s labels (applyShowLabels in internal/channels/service/interactions.go). Neither half touches this counter.

The reply is an ephemeral, deliberately not a modal (git-bug 60e6e10, owner’s ruling). SPEC §H.8 designates that value for views.open; a modal needs a trigger_id spent inside Slack’s three-second synchronous window, and this work runs on the queue, so the trigger_id would already be dead. response_url has no such clock. It was inert for every option until 60e6e10 — the value was minted, routed, and answered with silence, against §B.8.6’s “buttons are never no-ops” — and that press never incremented this counter either, correctly: it was known-and-routed, and conflating “one option does nothing” with “oto could not route this” would have put a permanent floor under a series whose whole meaning is “oto is broken”.

⚠️ A Show all labels press that answers “oto cannot list labels from Slack in this deployment yet” is a WIRING gap, not an unknown action. The labels port (channels/service.Labels) is optional and nil-tolerant: an unwired deployment answers the press with that sentence rather than with silence. It is honest and it is not this metric’s subject — zero here is correct while that sentence is what users see. Wire the adapter in internal/app/container.go beside Cases and Snoozes; the compile-time proof belongs in internal/app/assertions.go with its neighbours, because a nil port fails at no injection site.

Why it exists, and why it is a counter rather than a log line

Section titled “Why it exists, and why it is a counter rather than a log line”

An unknown action is answered 200, and must be: Slack disables an app’s event subscriptions when more than 95 % of deliveries fail inside a 60-minute window. So the endpoint deliberately reports success for a press it could not serve, and this counter is the only evidence anywhere that it happened. Without it the path is back to its original defect — an authentic press, a tick shown to the user, and no record at all.

It carries no labels on purpose. The obvious label is action_id, whose values come from a payload oto does not author; an unknown id is by definition outside oto’s closed set, so labelling by it hands an unbounded key space to Prometheus. The id is on the WARN log beside the increment, which is where high-cardinality detail belongs.

What a non-zero value means

Section titled “What a non-zero value means”

Exactly one of two things, and both are bugs in oto:

  1. A card rendered an action_id the consumer does not switch on — the renderer and the router disagree.
  2. An action_id was renamed, which retroactively breaks every card already sitting in Slack. Cards are durable; a rename is a wire-contract break for every message already posted, forever.

The user-visible effect is worse than an error: the person sees the interaction accepted and believes they acknowledged the alert. They did not.

What to check

Section titled “What to check”
  1. The WARN log beside the increment — it carries action_id, slack_team_id, slack_channel_id. The action_id string is the whole diagnosis.
  2. Compare it with the constant block in internal/channels/service/interactions.go (ActionAcknowledge, ActionUnacknowledge, ActionSnooze, ActionUnsnooze, ActionOverflow, ActionNoopPrefix) and with what internal/channels/render/slack/ actually emits. The block is the closed set; anything the renderer emits that is not in it is this metric’s whole subject.
  3. Did it start at a deploy? GET /api/v1/version. A rename lands with a release.
  4. Whether the increment came from the HTTP path or the slack.interaction worker — both record it, because from an operator’s chair it is the same fact.

What to do

Section titled “What to do”
  • Renamed id: restore the old id and keep serving it. Cards already in Slack cannot be re-rendered, so the router must accept both the old and the new id indefinitely — the rename is not complete when the code ships, it is complete when the last card carrying the old id has aged out.
  • Renderer/router mismatch: add the branch, or stop rendering the button. A card carrying an Acknowledge nobody consumes is worse than a card with no button at all.
  • Tell the affected people their acknowledgement did not land — oto answered 200 and they have no way to know.