Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0011 — Validation is seven layers with two opposite trust models

Status: Accepted · 2026-08-08

Context

Section titled “Context”

“Validate the input” is one sentence covering at least seven genuinely different problems in oto, and the two most important of them want opposite behaviour.

An API request body is written by an authenticated user who can read the error and fix it. Rejecting it loudly with a precise field path is the correct, kind response.

An Alertmanager webhook body is written by software oto does not control, over a protocol whose published documentation is incomplete (routeLabels is on the wire and absent from the docs), and whose retry classifier permanently deletes any notification answered with a 4xx. Applying API-style strictness there — DisallowUnknownFields, reject-on-first-violation, 400 on a bad label — silently destroys alerts during the exact window when the customer’s cluster is on fire.

A single shared “validation layer” is therefore not a simplification; it is a defect.

Separately: bounds get written down three times (a struct tag, a domain constructor, a DDL CHECK) and drift. When they drift, a 422 becomes a 500 — the user is told “internal error” for something they could have fixed.

Decision

Section titled “Decision”

Seven layers, each with a named library, a named trust model and a named failure shape (SPEC §L).

  1. API DTOs — go-playground/validator/v10, invoked only through httpx.Bind[T] so no handler can forget. RegisterTagNameFunc makes every reported path a JSON name. DisallowUnknownFields is ON. Result: 422 + RFC 9457 violations[] of {field, code, message}, with a closed tag→code map.
  2. Untrusted inbound — hand-written bounds (B1–B17, every value literal). Lenient decode, hard bounds, per-alert rejection into ingest_rejections, and still 202. The only permitted 4xx are 401, 413 and undecodable-400, all genuinely permanent. Redaction runs before the raw persist.
  3. Domain invariants — value objects with unexported fields and New…() (T, error) constructors; a single total Transition function is the only way an occurrence changes state. There is no optional Validate() anywhere. Illegal states are unrepresentable.
  4. Provider config — santhosh-tekuri/jsonschema/v6, draft 2020-12, compiled from embed.FS at boot. The same bytes validate server-side and render the settings form, so there is one source of truth and the UI needs no code when a provider is added.
  5. Outbound render — 18 explicit Block Kit checks before any Slack call. A violation is a dead delivery with the offending payload persisted, never a truncated send.
  6. Persistence — CHECK/NOT NULL/UNIQUE/FK as the last line of defence. A CHECK violation reaching HTTP is a 500 and an alert, because it means layers 1–3 have a hole.
  7. Frontend — valibot (tree-shakeable, ~10× smaller than zod), used for forms and for parsing API responses. Responses parse looseObject (additive server changes must not break a deployed UI); forms parse strictObject.

Drift is closed by four CI gates: Go DTOs → OpenAPI, server → OpenAPI (schemathesis), OpenAPI → TS types, OpenAPI → valibot — plus TestValidatorMatchesDDL, which asserts every custom-rule regex is byte-identical to its DDL CHECK.

The repository layer never validates a business rule (that is the service’s job; duplicating it produces two subtly different rulebooks) but does reject malformed row models and is the single place SQLSTATEs become errs.Kind.

Consequences

Section titled “Consequences”
  • A bound now lives in three files, and changing one without the others fails CI. That friction is the feature: it is what stops a 422 silently becoming a 500.
  • Error responses are uniform enough that the SolidJS form layer maps violations[].field onto controls mechanically, with no per-endpoint code.
  • The ingest path has a property test (TestIngestNever4xx) and a fuzz target, because “we returned 400 once” is a data-loss incident, not a bug report.
  • Seven layers is more machinery than a small service needs. It is justified here because oto’s entire premise is “never lies”, and because half the input is hostile by construction.

Alternatives rejected

Section titled “Alternatives rejected”
  • One shared validation layer: would apply user-input strictness to Alertmanager payloads and delete alerts. Non-viable.
  • ozzo-validation / hand-rolled: nicer for conditional rules, but far less ecosystem support and no RegisterTagNameFunc equivalent, which is what makes JSON field paths free.
  • zod on the frontend: ubiquitous, but 3–4× the bundle for identical ergonomics on a dashboard where the alert table is already the bundle budget.
  • Generating Go DTOs from OpenAPI (oapi-codegen): would remove gate G1, but reintroduces the codegen coordination dependency between independent implementation agents that C19/C20 removed. A drift test is cheaper than a drift generator here.
  • Validating only at the edge: the classic mistake. It makes every internal caller a trusted caller, and workers (reconciler, reaper, notifier) are internal callers that write domain state.