Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0021 — Correctness and testing strategy

Status: Accepted · 2026-08-09 Relates to: 0001 (River in Postgres — the outbox is a real collaborator, never a mock), 0006 (the reconciler is the only way into suppressed), 0007 (the 202 promise), 0011 (where each bound is enforced)

Context

Section titled “Context”

oto is a flight recorder. Its wedge is not throughput and not features — it is that the record is complete and true: a 202 is a promise the payload is durable, expired is never dressed up as resolved, and oto’s silence is never indistinguishable from “no alert”. That argues for proving those specific paths rigorously.

Against that, heavy upfront testing slows iteration and couples tests to implementation. The philosophy is classical, lean, grow-from-bugs.

There is a third pressure, and it is the one that actually bit. oto was written against a 4,852-line specification and barely executed until late. That produced a defect class which grow-from-bugs structurally cannot catch: a feature that never worked has nothing to regress. A setting the operator sets that no branch reads (OTO_INGEST_*, channels.config.transport). A response field the contract promises that no line populates (delivery_summary on four detail responses). A service fully implemented, fully unit-tested, and never constructed (NewInteractionService). Fifteen-plus instances were found by running the product for an afternoon. The response at the time was tools/lintreach — a 1,375-line whole-module go/types analyzer that reads for the same shapes — which now ships with a 328-entry baseline of accepted debt and cannot see the adjacent class (a field that is written and read, with the wrong value).

Today the suite is 42 test files against 84,476 lines of production Go. Six of fourteen domains have none: grouping, rules, enrichment, streaming, stats, silences. test/harness — the package whose own doc comment claims to own “real Postgres via testcontainers, fake Slack, fake Alertmanager, fake Prometheus and object builders” — is empty, and its testcontainers bootstrap is instead copy-pasted into three unrelated files.

Decision

Section titled “Decision”

Classical (Detroit-school), lean, regression-driven, with a feature-acceptance floor.

1. Real collaborators in-process; mock only true externals

Section titled “1. Real collaborators in-process; mock only true externals”

Exercise the real service, repository and domain objects together. A test double is permitted only at an adapter boundary that leaves the process:

Real, always Faked at the port
Postgres (testcontainers) Slack Web API + interaction callbacks
River / platform/jobs (it is Postgres) Alertmanager v2 and Prometheus v1 HTTP
Every internal/*/service, repository, domain Enricher endpoints (HTTP, MCP)
The chi router and httpx stack Outbound generic webhooks
platform/clock via FakeClock —

There is no Redis, no second datastore, and no message broker to stub (0001, 0015). Mocking a repository is not permitted — a mocked DB lies about SQL semantics, and every invariant that matters here (dedup on the identity key, per-thread sequence gating on advisory locks, the ON CONFLICT that is the idempotency mechanism) lives in the SQL.

test/harness owns the one bootstrap: container, migrations, per-test schema isolation, the fakes above, and object builders. The three hand-rolled TestMain copies collapse into it.

2. Lean in v1; grow the suite from real bugs

Section titled “2. Lean in v1; grow the suite from real bugs”

Enough integration coverage per module to catch glaring breakage, then every fix lands with the regression test that would have caught it. Don’t fight test infra; keep velocity. Renderers stay as they are — pure functions with checked-in testdata/*.golden.json — because that shape is already right.

3. Wedge exception — the record must be complete and true

Section titled “3. Wedge exception — the record must be complete and true”

Four targeted integration tests guard the paths that must not break, against real Postgres:

  1. The 202 is durable. Kill the process between the webhook’s commit and batch processing; on restart the raw batch is present and processed exactly once. A 2xx we cannot honour is the one failure that deletes an alert forever (0007).
  2. Never fabricate a resolution. With source_health.status != 'healthy', the reaper is blocked and no occurrence transitions to expired.
  3. Clamp, never reject. A backward-skewed upstream startsAt clamps ended_at, flags clamped: true, records the skew, and does not abort the ingest transaction.
  4. Silence is distinguishable from success. Every delivery for an alert dies; the alert detail response still reports delivery_summary with non-zero dead and a last_error_class.

This is not a deterministic-simulation framework. The hexagonal ports keep that door open at low marginal cost if the wedge ever demands it.

4. Feature-acceptance rule — the floor that grow-from-bugs cannot reach

Section titled “4. Feature-acceptance rule — the floor that grow-from-bugs cannot reach”

Every ingestion, alerts, notification, channels or sources feature lands with at least one functional test that executes the feature end to end through the real container and router, fakes at external ports only. “Compiles, lints, and the JSON Schema validates” is not acceptance.

Concretely, at land time:

  • A new config knob lands with a test that asserts a branch reads it — two runs differing only in that knob must produce different observable behaviour. This is the OTO_INGEST_* class.
  • A new response field lands with a test asserting it is populated with a value the test can predict, not merely present. This is the delivery_summary class.
  • A new service lands reachable from internal/app/container.go, proven by a test that reaches it through an HTTP route or a registered River worker. This is the NewInteractionService class.

The SolidJS UI is exempt and stays grow-from-bugs, covered by the Playwright suite in web/e2e/. A settings form is UI; the setting behind it is not, and falls under the config-knob rule above.

Consequences

Section titled “Consequences”
  • Fast iteration; the test surface tracks real risk rather than line count.
  • The four wedge paths get deliberate coverage that no amount of unit testing would have produced.
  • tools/lintreach stays, as a second line rather than the only one. It catches the shape §4 cannot: a declaration added between feature landings, in a module nobody is currently testing. §4 changes what it is for — it stops being the mechanism that substitutes for running the product and becomes a ratchet. The condition on keeping it is that the 328-entry baseline only ever shrinks: a gate with a permanent amnesty list is a gate nobody reads. Regenerating the baseline to absorb a new finding is not permitted; the finding is fixed, or the declaration is deleted. tools/lintvocab stays on the same terms for a different job — defending the scope boundary (0013).
  • Six untested domains are a backlog, not a blocker. Order by wedge proximity: grouping (it owns the thread), rules (the differentiator), streaming, enrichment, silences, stats.
  • CI wall-clock rises with testcontainers. Accepted: one container per package, schema-per-test.

Alternatives rejected

Section titled “Alternatives rejected”

Retire lintreach once §4 is in force. Considered and rejected. The argument for retiring it is real — two mechanisms for one property, a 328-entry amnesty list, and a permanent false-positive maintenance cost (whole-struct conversions, embedded promotion, three reflection-based decoders modelled by hand). The argument against is stronger: §4 is a discipline and lintreach is a mechanism, and the failure mode we are guarding against is precisely a discipline lapsing. It stays, under the shrink-only condition above. Revisit if the baseline stalls for two milestones — a ratchet that never turns is just an amnesty.

Mock repositories for speed. Rejected. The invariants are in the SQL. A green suite over mocked repositories would have been fully compatible with every one of the fifteen defects.

Full DST harness in v1. Proves the wedge hardest, but the harness cost lands before the product has users. The ports keep it available later.

e2e-first, defer the unit and integration tiers. Fastest to write, weakest assurance for a product whose entire claim is that its record is trustworthy.