Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0014 — Postgres is the only store; no TSDB or column store in v1

Status: Accepted · 2026-08-08 Relates to: 0001 (Postgres as sole datastore), 0003 (the event stream), 0016 (the rule that protects this decision) Resolves: red-team memo objection 5 — “Postgres is right for the alert entity and wrong for the event stream. Same word, two workloads.”

Context

Section titled “Context”

The red team’s objection is well aimed: alerts are OLTP and the event stream is time-series, and conflating them bites somewhere around 50–100M rows. The natural reflex is to reach for a second engine — ClickHouse, TimescaleDB, DuckDB over Parquet — before the pain arrives.

The reflex is wrong here, for one reason that is easy to miss: oto’s data is human-scale, not machine-scale. Metrics are millions of samples per second, which is precisely why Prometheus exists and why it sits upstream of us. An alert is rare by construction. A fleet large enough to be interesting produces on the order of 1–2M events per day at the pessimistic end, and if an installation genuinely produces more, that is a broken alerting configuration — which is a thing oto is supposed to report on, not a scale target to engineer for.

Every hot query is scoped (org_id, alert_id | group_id) plus a time range, which prunes to a single monthly partition. The only genuinely analytical workload — firing frequency, firing-duration percentiles, noise reports — is served by precomputed daily rollups (alert_quality_daily) written by a background job.

Decision

Section titled “Decision”

Postgres is the only datastore. The event tables are RANGE-partitioned by time; analytical questions are answered from rollup tables, not from scans of the event stream.

We do not add a time-series database, a column store, or an embedded analytical engine in v1.

The escape hatch is designed in rather than promised: the event tables are append-only and partitioned, so a cold partition can be DETACH PARTITION’d and exported to Parquet for DuckDB or ClickHouse without touching the write path or the live schema. That makes an analytical store a bolt-on, not a migration — which is exactly what justifies deferring it.

Revisit when any of these is observed, and not before:

  • alert-list p95 above 200 ms with a warm cache,
  • a rollup job overrunning its own interval,
  • a single org exceeding roughly 10M events per month.

“Observed” is doing real work in that sentence, and today nothing observes it. No metric oto exports answers the first or the third, and the second is only half emitted. These three are also the reopen condition for the other decision derived from this one — that the never-reaped tables in 0024 stay unpartitioned, because on each of them the constraint a partition key would have to join is the product invariant itself. What is measured, what is not, and the notification_deliveries design that is ready for the day a trigger does fire, are in 0024 Amendment 3.

Consequences

Section titled “Consequences”
  • One dependency. helm install plus a Postgres URL remains the whole deployment story, which the red team identified as a real asset against Robusta (whose UI is enterprise-gated) and Keep.
  • Transactional integrity is preserved end to end: the outbox property in 0001 depends on the job queue and the state change sharing one transaction, which a second datastore would break.
  • Rollups are eventually consistent by minutes. Analytics screens must state their as-of time rather than implying live numbers.
  • We accept that a very large single-tenant installation will hit the revisit triggers before a small one does, and that the first response will be tuning partitions and rollups, not a new engine.

Alternatives rejected

Section titled “Alternatives rejected”

TimescaleDB. Keeps one wire protocol and adds hypertables and continuous aggregates. Rejected because it is an extension many managed Postgres offerings do not permit, which would silently constrain where oto can be self-hosted — a high price for a scale we do not have.

ClickHouse or a column store from day one. Genuinely better at the analytical shape, and genuinely a second datastore to operate, back up, migrate and keep transactionally consistent with Postgres. Premature at our volume, and reachable later via the Parquet path.

Embedded DuckDB over exported Parquet, now. Attractive because it adds no service. Rejected as solving a problem we do not yet have, while adding a second query dialect and an export-sync path to keep correct. This remains the preferred option if a revisit trigger fires.