Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

oto_render_invalid_total
Type counter
Labels provider — slack, webhook · renderer — the renderer id from the channel row · mode — post_root, update_root, thread_reply, broadcast_reply
Registered in internal/notification/service/metrics.go; incremented in internal/notification/service/dispatch.go
Alertable YES — page on it. The code says: this counter is an alert, not a statistic, and it is the only one this failure has

What it counts

Section titled “What it counts”

Deliveries that died because oto could not build a legal payload for them: the renderer returned an error, or the payload its own §L.6 outbound checks (V0–V18) refused.

Why every single increment matters

Section titled “Why every single increment matters”

Every one is an oto bug, never the destination’s. Nobody was told, and no provider was ever called — the message did not fail to send, it failed to exist. The delivery is marked dead with error_class='config_invalid' in the same transaction, and the offending bytes are kept in notification_deliveries.rendered so the card can be reconstructed exactly.

⚠️ Why this counter exists at all, when oto_jobs_dead_total looks like it should cover it

Section titled “⚠️ Why this counter exists at all, when oto_jobs_dead_total looks like it should cover it”

It does not cover it, and for a long time both this repository’s SPEC and three of its own code comments said it did. The chain was checked and it is broken in two places:

  1. The job succeeds. dispatch.go marks the delivery dead in the same transaction and then reports SUCCESS, correctly — the job was asked to resolve a delivery and it did. So the River dead-letter is never reached and oto_jobs_dead_total stays flat while cards die.
  2. Propagating the error would not fix it. A render error has no case in jobs.Classify and falls through to retryable, so River would wake a job whose delivery is already resolved; the second pass exits at Status.Resolved() and succeeds. One extra wake-up, no dead-letter, no information.

And even where oto_jobs_dead_total does fire, it cannot tell “oto built an illegal card” from “the destination’s token was revoked”. One is fixed by shipping a new oto and one by a customer editing their channel config; they want different people woken, so they need different counters.

⛔ There is no check label

Section titled “⛔ There is no check label”

V0–V18 is a bounded set and would make a legal label, and it is still refused. The check that rejected a payload is a Slack concept, dispatch.go’s own header forbids provider-specific code there, and render/slack/validate.go refuses the same label from the other side. The check name reaches an operator through the log line and through the dead delivery’s stored error, sitting next to the bytes that failed — which is more useful than a series, because you can read the card.

What to check

Section titled “What to check”
  1. The ERROR log record, which carries the check name inside the error string: 
    slack render invalid (V14): top-level text is 412 chars, limit 300
    delivery_id= channel_id= thread_id= provider= renderer= mode=
  2. notification_deliveries for delivery_id: status='dead', error_class='config_invalid', and rendered holding the payload that was refused. GET /api/v1/deliveries/{id} serves it.
  3. Which mode is failing. A failure only on update_root and not on post_root points at the §H.4 strikethrough path or at a card that has grown past a budget as its trail lengthened.
  4. Whether it correlates with one renderer. Both renderers validate their own output, and only one of them is likely to be at fault.

What to do

Section titled “What to do”
  • Look at the stored payload first. It is the whole bug, kept deliberately for this reason, and the check name tells you which rule it broke.
  • A customer NotificationTemplate is the FIRST thing to check (ADR 0050), and this is a reversal from the retired Wording design. A Wording emitted text into a block Go built, so structural validation was unreachable by construction. A template owns the blocks: a card renders through oto’s own IR compiler and a raw template supplies Block Kit directly. Both are caught here rather than by Slack, and both fall back to oto’s built-in card — so an increment means an alert went out in the DEFAULT voice, not that one was lost. Read notification_deliveries.template_id on the affected rows first. If an increment coincides with a new template, the bug is in the sink or in a budget, not in the template — and that is worth knowing precisely because it looks like the template.
  • A sudden onset after a deploy is a rendering regression: the golden files (internal/channels/render/*/testdata/*.golden.json) run Validate over every fixture in CI, so a failure here is a shape that no fixture covers. Add the fixture.
  • The delivery is terminal and will not retry itself. Once the bug is fixed, POST /api/v1/deliveries/{id}/retry is what re-sends it.