Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0022 — Length-prefixed identity pre-images

Status: Accepted · 2026-08-09 · amended 2026-08-20 (Amendment 1) Amends: 0004 (which specified the 0x00 framing), SPEC §C.0–§C.7 Relates to: 0021 (the tests that found this)

Context

Section titled “Context”

Every identity oto computes — alert_key, group_key, rule_fingerprint, batch_dedup_key, notification.idempotency_key — is a SHA-256 over a pre-image built by concatenating fields. Until now the fields were separated by bytes: 0x01 and 0x02 inside §C.1’s label serialisation, 0x00 between the outer fields of every other key.

Separator framing is injective only if no field can contain the separator. Three fields can:

  • label values — arbitrary UTF-8 from the operator’s monitoring system, and a JSON unicode escape decodes straight through the ingest path;
  • receiver — free text from alertmanager.yml;
  • expr — arbitrary PromQL;
  • Alertmanager’s own groupKey — which SPEC §C.4 explicitly documents as unescaped and unbounded.

Nothing constrained them. NewLabels validated label names against a charset and bounded values only by length. So two different field splits produced one digest:

{alertname:"X", b:"1", c:"2"}      ->  alertname 01 X 02 b 01 1 02 c 01 2 02
{alertname:"X", b:"1\x02c\x012"}   ->  alertname 01 X 02 b 01 1 02 c 01 2 02

A three-label set and a two-label set, byte-identical. They become one Alert: one row, one timeline, one Slack thread, with one alert’s episodes written into another’s history. For a product whose only claim is that its record is true, that is the worst available failure.

The outer framing was no better, and its witness is subtler — it does not even require a field to contain a NUL in the obvious way:

("a", {b: ""})                            ->  61 00 | 00 00 00 01 62 00 00 00 00
("a\0\0\0\0\x01b\0\0\0", {})              ->  61 00 00 00 00 01 62 00 00 00 | 00

because canon({b:""}) ends in a zero-length prefix, which absorbs the terminator the receiver would otherwise have contributed.

The test that was supposed to prove this safe, TestGroupKey_ReceiverAndLabelsAreSeparateFields, was off by one byte — its forgery produced a pre-image one trailing NUL longer than the honest one, so it passed under the old framing too and proved nothing.

Decision

Section titled “Decision”

Every field in every §C pre-image is length-prefixed:

field(x) := uint32be(len(x)) || x        -- len() counts BYTES

Four bytes, big-endian, fixed width, applied uniformly. Nothing is escaped and no byte is reserved. Each key writes N framed fields and then one final field raw, because a remainder needs no prefix to be found; where that tail is itself a §C.1 serialisation it is self-delimiting in turn. The one place a canonical blob is not last (§C.6’s rule_labels) is framed like any other field.

This is injective because the encoding is uniquely decodable — read four bytes as n, take exactly n bytes, repeat — which gives it an explicit left inverse.

Separately, a label value may not contain 0x00. That is a storability bound, not sanitisation: Postgres text cannot hold U+0000 at all, so such a value fails at INSERT however it is serialised. Nothing else is stripped, replaced or normalised.

Labels.Fingerprint (§C.3) is deliberately untouched. It reproduces prometheus/common’s algorithm rather than oto’s framing, and it is the join key for /api/v2/alerts reconciliation — changing it would silently break every Alertmanager match.

Amendment 1 — §C.7 may append ONE fixed-width field after its tail (2026-08-20)

Section titled “Amendment 1 — §C.7 may append ONE fixed-width field after its tail (2026-08-20)”

notification.idempotency_key now takes an optional trailing occasion — field(uuid), written only when the key names one — which makes it the single §C key that writes a field after the raw tail. SPEC §C.7.1 has the product argument (a re-snooze was minting a byte-identical key and its announcement was being dropped); this amendment records what it does to this ADR’s framing rule, because that rule is what makes the appended field safe rather than a second forgeable boundary.

The decision below says each key writes N framed fields and then one final field raw, “because a remainder needs no prefix to be found”. Appending after that remainder means the remainder is no longer the end, so the boundary between them has to come from somewhere. It comes from width:

  • the occasion is a uuid, hashed as its 16 raw bytes, so its field is a fixed 20 bytes whose first byte is the 0x00 of uint32be(16);
  • the tail it follows is strconv.Itoa, which emits only 0x30–0x39;
  • so the first 0x00 after the reason field is the split, always, and the encoding stays uniquely decodable — read digits to the first non-digit, and what remains is either nothing or exactly one framed 16-byte field.

The two bounds this rests on are binding. An occasion of variable width could open with a digit (a length ≥ 48·2²⁴ does) and forge the boundary, and a tail that was not decimal would break it from the other side. No second field may be appended after a tail, and no appended field may be variable-width. Anything richer than one uuid belongs in front of the tail, which re-keys the whole key and is a different decision from this one.

uuid.Nil writes no bytes, not sixteen zero ones, which is the property that made this additive: every key that names no occasion is hashed over byte-identical input and nothing stored moved — the opposite of this ADR’s own headline consequence below, and the reason this amendment needed no release-gated re-key.

Consequences

Section titled “Consequences”
  • Every alert_key, group_key, rule_fingerprint, batch_dedup_key and idempotency_key changes. This is why it landed pre-release: there is no history to re-key. After release it would have cost a migration that splits every timeline in two, which for a flight recorder is the most expensive migration that exists.
  • The same fix had to be applied in four places — alerts/domain, ingestion/domain, rules/domain, notification/domain — because §C.6 and §C.7 each have two implementations. That duplication is now cross-checked by tests and tracked as debt; it is a recurring cost, not a one-off.
  • Pre-images grow by 8 bytes per label instead of 2, and by 4 bytes per outer field. At most 384 bytes against the 16 KiB label-set cap. Irrelevant to a hash.
  • A label value carrying a NUL is now rejected into ingest_rejections rather than stored. It currently reports undecodable, which is lossy; a dedicated reason is tracked separately.
  • Anyone recomputing an alert_key outside Go can reimplement fixed-width prefixes trivially. That was the argument for fixed width over a varint, along with SerialisedSize staying exact arithmetic.

Alternatives rejected

Section titled “Alternatives rejected”

Escape the separators inside values. Fully correct, and rejected on product grounds: escaping means oto editing an operator’s bytes in order to store them. oto is a flight recorder; the record is what upstream said. Length prefixing achieves the same injectivity while leaving every byte untouched.

Reject values containing the separators. The cheap fix, and the right one after release — it changes no existing key. Rejected here only because the project is unreleased, so the correct fix is free. It remains the fallback if a future format change is ever needed post-release.

Varint / LEB128 length prefixes. Saves three bytes per field on short values. Rejected: fixed width is trivially reimplementable by an external consumer, and it keeps SerialisedSize exact rather than a length-of-length computation.

Hash each field separately and combine the digests. Also injective, and a larger change to every call site for no benefit over prefixes.