Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open

Status: Accepted · 2026-08-08 Relates to: 0013 (policies route signals, never people), 0014 (indexed queries are what make Postgres-only work)

Context

Section titled “Context”

keep.dev exposes CEL for searching alerts, and the question is whether oto should follow.

Keep’s reason is sound for Keep: they ingest from roughly a hundred heterogeneous providers, so an alert is effectively a schema-less JSON document, and a general expression language over arbitrary fields is the natural fit. oto’s model is Prometheus-shaped with a well-defined label set. The stronger schema is precisely what makes indexed SQL viable, and it removes most of Keep’s motivation.

There is a real argument on the other side and it should not be dismissed: Kubernetes users now know CEL, from CRD x-kubernetes-validations and ValidatingAdmissionPolicy. cel-go is well maintained, sandboxed and non-Turing-complete with bounded evaluation. “Users won’t know it” is a weak objection for our specific audience.

The strong objection is different. CEL does not translate to SQL. Evaluated in the application, it requires fetching rows to test them — a scan, defeating keyset pagination and every index the schema was built around. The real-world answer is a CEL→SQL translator, and only a subset of CEL survives translation. That ships a language whose published semantics do not match what is actually supported, and users discover the boundary at unpredictable places. That gap is a permanent support tax, and it is what makes this look cheaper than it is.

Decision

Section titled “Decision”

Search uses Alertmanager-style label matchers plus structured field filters. {namespace="payments", severity=~"critical|warning"} is the native idiom of our audience, translates directly to indexed SQL and GIN containment, and covers the overwhelming majority of real queries. Combined with saved views, it is the v1 search surface.

Filters are represented internally as an AST, not as a bag of query parameters. This is the cheap decision that keeps the door open: matchers compile to the AST today, and if CEL earns its place later it compiles to the same AST. The translatable subset then becomes an explicit, testable capability list rather than an emergent surprise.

If CEL is ever added, anything untranslatable is rejected at parse time with a precise message — never silently degraded to a scan. A query that cannot use an index must fail loudly, because the alternative is a filter that works in staging and times out during an incident.

On the notification-policy path, be more conservative still. An expression language on the routing path is a foot-gun: a policy that errors at evaluation time means an alert is not routed. Matchers are almost certainly sufficient. If richer predicates are ever added there, three properties are non-negotiable — evaluation must be total (never errors), failure must be fail-open (notify anyway; never silently drop), and it must be exercised through the existing dry-run preview. The scope boundary continues to bind: predicates over signals only, never people, teams, schedules or times of day.

The case that would justify going further is predicates over derived state — “only notify if this has fired more than three times in 24h”. That is better served by a small set of named conditions than by a general language.

Consequences

Section titled “Consequences”
  • Users cannot express arbitrary boolean logic over alert fields in v1. Matchers plus structured filters plus free-text search is the ceiling, and some genuinely reasonable query will not be expressible. Accepted.
  • Every filter is index-backed, so search latency stays predictable as the event table grows — which is one of the load-bearing assumptions of 0014.
  • The AST is extra indirection for a v1 that has only one front end compiling to it. That cost is deliberate and small.
  • Saved views become the answer to “I need this complicated query often”, which is usually the real request behind asking for an expression language.

Alternatives rejected

Section titled “Alternatives rejected”

CEL as the primary search surface, with a CEL→SQL translator. Rejected: the supported subset diverges from the documented language, and the failure mode is a scan during an incident. Reconsider only behind the AST, with the capability list defined first.

Application-side CEL evaluation over fetched rows. Rejected outright — it discards pagination and indexing, and its cost grows with exactly the data volume that matters.

A bespoke query language. Rejected: all of CEL’s translation cost, none of its familiarity, plus a grammar to maintain.