Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose

Status: Accepted · 2026-08-08 Relates to: 0014 (this is the rule that protects it), 0009 (snapshot-at-fire-time semantics), 0013 (oto is a flight recorder, not an observability store) Resolves: the open question “is oto k8s-enriched or only k8s-shaped?” Amended by: 0054 §5 — a ToolServer may expose write Tools; oto still holds no cluster credential.

Context

Section titled “Context”

All Kubernetes API enrichment was deferred: it needs cluster RBAC, it is Robusta’s core competence, and it is a large sink of effort. That left oto k8s-shaped — it groups by namespace and cluster and installs by Helm — but unable to read a cluster.

The product owner’s proposal is to close that gap through MCP servers and through Kubernetes events oto has already pulled, rather than by holding cluster credentials.

This is a better answer than the one that was deferred, but it contains a trap, and the trap is the reason this ADR exists.

Decision

Section titled “Decision”

Cluster context is fetched on demand at enrichment time, through an MCP client, from a server the operator runs. oto never holds cluster RBAC; the trust boundary sits at the MCP server. This is materially easier to sell self-hosted than “give this thing cluster-reader”.

Two constraints fall out of the existing enrichment design and are binding:

  1. MCP enrichers run in the async phase only, never inside the pre-notification budget. An external server can be slow or down, and enrichment must never delay or block a notification.
  2. The result is a snapshot recorded at that time, not a live view. It is stored as an enrichments row with provenance, exactly like a rule snapshot. The UI must render it as “this is what the cluster looked like when we asked”, never as live state.

⛔ The prohibition. oto does not subscribe to, ingest, or store the Kubernetes Events stream.

This is the rule that keeps 0014 true. Every argument for Postgres-only rests on oto’s data being human-scale. Kubernetes Events are machine-scale — a busy cluster emits thousands per minute — and ingesting them would, in one decision, force a time-series store, invert the storage architecture, and turn oto into an observability product. It would also breach 0013: a k8s Event stream is not a fact about a signal we were told about, it is a general-purpose telemetry feed.

What we do instead: at enrichment time, fetch only the events relevant to this alert — same namespace and object, within a bounded window — and store that bounded set as the enrichment payload.

Consequences

Section titled “Consequences”
  • Cluster enrichment requires the operator to run an MCP server. That is a real onboarding step and it must be honestly documented, not hidden. Alerts remain fully useful without it.
  • Enrichment quality depends on a third party’s availability. Partial failure is already normal in the pipeline, so a missing enrichment degrades a card rather than failing a notification.
  • Because results are snapshots, an alert investigated a week later shows what was true at fire time. This is a feature — it is the same property that makes rule snapshots the differentiator — but the UI must timestamp it explicitly or users will read it as current.
  • We are choosing not to compete with Robusta on depth of cluster enrichment. That is a real competitive concession and should be stated plainly rather than papered over.

Alternatives rejected

Section titled “Alternatives rejected”

An in-cluster agent that ships events to oto. This is Robusta’s model and it gives the richest context. Rejected for v1: it makes oto a distributed system with an agent to version, deploy and secure, and it is the path that leads directly to the firehose problem above.

Direct Kubernetes API access with a cluster-reader ServiceAccount. Simpler than MCP and needs no extra server. Rejected because it puts cluster credentials inside oto, which is the single biggest objection a security reviewer will raise against a self-hosted alerting tool.

Storing the event firehose and querying it at enrichment time. Rejected above; it is the decision that would quietly invalidate the storage architecture.