oto — orientation for implementers

Read this first. Then read docs/design/SPEC.md §A–§C and the section for your module. SPEC.md is binding; this file is the map.

---

1. What oto is

The alert history layer your Prometheus stack does not have — self-hosted, with a UI, without adopting an AIOps platform or a paid SaaS.

For every alert that has ever fired, oto shows: when it first appeared, every episode since, what the rule said at that moment, who was told, on which channel, in which thread, who acknowledged it, and how it ended — as one continuous, replayable timeline.

Why this exists. Alertmanager holds no history (it is a silence console). Grafana’s history covers only Grafana-managed rules. Robusta’s UI is SaaS-or-enterprise. Keep is incident-first and generic. Grafana OnCall OSS was archived 2026-03-24, leaving a documented vacuum for a self-hostable OSS Slack-first alert layer over an existing Prometheus/Alertmanager stack.

What oto is not — permanently: an incident manager, an on-call/paging system, a workflow engine, a rule editor, or anything that tracks who owes work. oto draws Incidents — a set of Cases told as one story (ADR 0052) — but it does not manage one: the response (status, lead, severity, comms, write-up) lives in the incident tool the Incident is declared to, and oto only sends that tool facts. oto is a flight recorder. It records the aircraft. It does not fly the plane, roster the crew, decide who is in command, or write the accident report — and it is trusted precisely because it does none of those things.

1b. THE LINE — read this before adding any feature

docs/design/SCOPE-BOUNDARY.md and ADR 0013 are binding. Cite them by clause number.

FR-1, the Flight Recorder Test

Complete this sentence about the row your feature writes: “This row is a fact about ______.”

• A signal — Alert, AlertCase, RuleSnapshot, Notification, Delivery, AlertSource, or an aggregate over those → IN.
• A person, team, rota, responsibility, response effort, ticket, or customer-facing statement → OUT.

Human actions are IN only as timestamped annotations on a signal’s timeline, carrying an actor as metadata. In one line: ACTOR, NEVER SUBJECT.

case.acked_by = alice is a fact about the case — it was acknowledged, by whom. case.assigned_to = alice is a fact about Alice — she owes work. Identical columns; opposite products. That is the whole line.

Secondary heuristics, in order, when FR-1 does not decide cleanly:

• H-1 Obligation — does it create, route to, or discharge an obligation on a named human? → OUT.
• H-2 Orphan — delete every alert; does the artefact still mean something? → OUT. (Config exempt.)
• H-3 Write-path — does it change any system outside oto’s DB and its channels? → OUT of v1, on safety grounds. A separate axis from FR-1, and the only one that is earnable.

FR-1 refusals are permanent. H-3 refusals (e.g. R3, silence-write) are earnable. Do not conflate them; they have different expiry dates.

⚠️ One H-3 refusal has been earned, and one FR-1 noun admitted with it (ADR 0054, accepted 2026-10-02): a cluster write is IN as a Remedy — proposed by an Investigator, executed by the operator’s ToolServer only after human approval, every transition a fact — which is SCOPE-BOUNDARY verdict #23 earned on SS-4’s terms (an audit trail and a confirmation UX). Remedy is the only response-effort noun FR-1 now admits. Silence-write (R3) is still unearned.

The four doors that must stay shut

Door	Clause
No person-reference column on a signal row. No assigned_to, owner_id, watchers, incident_id, ticket_id, sla_due_at. acked_by is past-tense attribution and is the only exception.	SPEC §D.4.0
No unprompted reminder, at all. The one reminder stage was withdrawn (git-bug bd0fb1d); oto sends nothing nobody asked for. This SUPERSEDES “one stage, forever” and is strictly stronger — there is no first stage for a second to follow. Re-adding any unprompted reminder needs an ADR arguing against FR-1 by name. escalation remains a banned word.	SPEC §G.9.1
No human writes a signal’s state. There is no POST /alerts/{id}/resolve, ever — nor close, merge, dismiss, reopen. ack_state is the only state axis a human may write.	SPEC §E.1.1
notification_policies routes to destinations, never to people. No user_ids, schedule_id, time_of_day.	SPEC §D.8

Permanently out of scope, with hand-offs: SPEC §I.1.1.

---

2. The three commitments

1. Append-only truth. Every observation and action is an immutable AlertEvent; current state is a projection. If you would ever want to UPDATE it, it is not an Event.
2. Ingestion is decoupled from everything. The webhook path does two things in one transaction — durably record the raw payload, enqueue work. Slack, Prometheus and enrichment are never on it.
3. Interfaces before implementations. Channel, Renderer, Enricher, AlertmanagerClient are ports with registries. Slack and the generic webhook are one-of-N implementations.

---

3. Domain language (use these exact words)

Term	Means
Alert	The identity of a label set within (org, cluster). Created on first sight, survives resolution forever. oto’s answer to Sentry’s Issue.
AlertCase	One contiguous firing episode of an Alert, (alert_id, seq). What you ack; whose FIRING DURATION is measured. Never “MTTR” — banned (§A.1). Strictly terminal: open → closed, once. A re-fire opens the next seq, never revives this one (ADR 0040). Two numbers, and they are not interchangeable: seq is the firing ordinal within ONE alert, number is the case’s name within the ORG — unique, monotonic, what a human quotes, and what /cases leads each row with (migration 00081). Forty alerts that have each fired once carry forty numbers and forty seq of 1.
AlertEvent	One immutable thing that happened at one instant. The timeline. Append-only.
Incident	A set of one or more Cases drawn together as one story — an Alert has Cases, an Incident spans Cases. A machine (by a stated rule) or a human may draw one. Inside oto it is a fact about signals: it is active while any of its Cases is open and quiet otherwise — read off its Cases, never set by a hand. Its response — status, lead, severity, comms, write-up — is managed in an external tool it is declared to, and oto holds only the outbound mapping to that tool’s object; nothing about the response is read back (ADR 0052). Avoid: arc, group, cluster, episode, storm, outage, correlation (that names the relationship, not the container).
Correlator	An operator-written definition that draws Incidents — matchers over Cases, optionally with a count over a window — so that why a Case is in an Incident always has an answer someone can read back. Only a Correlator or a human draws an Incident; a model may propose membership, never decide it (ADR 0052). Avoid: rule (that is the Prometheus alerting rule), incident rule, policy (that routes notifications).
RuleSnapshot	A content-addressed capture of a Prometheus alerting rule at a point in time. The differentiator.
AlertSource	One configured Alertmanager (+ optional Prometheus). HA replicas share a Cluster.
Cluster	Identity/failure domain. cluster_key participates in alert identity.
Channel	A configured destination instance (“Slack workspace T123, #sre-alerts”). Not a channel type.
Payload mapping	An optional document on a webhook Connection that turns the oto.notification.v1 envelope into the request a particular incident tool expects, and says where its response names the external incident. It is destination setup, not wording: a failure is a failed delivery, never a fallback. Without one, the plain envelope is sent. (ADR 0055) Avoid: template (that is wording, ADR 0050), plugin (a mapping is data, not code), provider.
Notification	The channel-agnostic intent to communicate one fact. Idempotent.
NotificationDelivery	One materialisation of a Notification on one Channel. Owns retry state and provider ids.
Conversation	What a channel_threads row is about, as the pair (conversation_kind, conversation_id): a Case, a digest, or an Incident whose Correlator says its Incidents are conversations (ADR 0052 §6). A Case conversation holds exactly one Case — a new Case always means a new thread, unless it joins an Incident that is a conversation, in which case later facts about it share the Incident’s thread; nothing already posted moves, and nothing is held back to wait for a Correlator. A digest’s conversation is keyed by the policy that asked for it, one per policy per channel. It is what decides which facts share a message, and that decision belongs to the notification layer and to what an operator wrote, never to a stored grouping oto derived on its own (git-bug 7570090, migration 00069). The Incident kind is the operator-written exception to ADR 0045’s “N alerts, N conversations”, not a return of AlertGroup.
ChannelThread	The binding of a Conversation to (slack_channel_id, root_ts).
Enrichment	One typed, provenanced result from one named, versioned Enricher.
Investigator	A named, versioned configuration of a model-driven investigation: which model, which prompt, which Tools it may call, and its budgets. The AI-for-SRE category word “agent” may describe it in marketing, never name it. (ADR 0053, accepted 2026-10-02.) Avoid: agent (that is vmagent in a VictoriaMetrics stack, and the in-cluster daemon ADR 0016 rejected), bot, assistant.
ToolServer	One configured MCP server an Investigator may read through. It is where trust stops: oto holds no cluster credential, the ToolServer’s operator does — oto keeps only the ToolServer’s own access token, sealed. Its access is declared read or write: only a read ToolServer’s Tools may be on an Investigator’s allowlist, and a write one is never offered to a model (it is where a Remedy’s write Tool is bound). Reached over HTTP only — oto starts no process. (ADR 0053, accepted 2026-10-02; git-bug 2e9a086.) Avoid: connection (that is a channel’s org-wide setup, ADR 0047).
Tool	One capability a ToolServer exposes that an Investigator is allowed to call, named <toolserver>__<tool> on an allowlist and to a model. oto’s own history — prior Findings, a Case’s timeline, the rule as it stood at fire time — is offered as built-in Tools on the same footing. A Tool an Investigator holds while investigating is read-only; a write Tool is called only to execute an approved Remedy. (ADR 0053, accepted 2026-10-02.) Avoid: action (that is the Slack button row).
Investigation	One run of one Investigator against one subject — a Case, an Incident, a digest or a policy — frozen once it ends. A subject may have many over time; the latest one’s Finding is the one shown. (ADR 0053, accepted 2026-10-02.) Avoid: analysis, triage (banned), probe (a source health check).
Step	One immutable entry in an Investigation’s transcript: a model turn, a Tool call, or a Tool’s result. If you would ever UPDATE it, it is not a Step. (ADR 0053, accepted 2026-10-02.) Avoid: event (unqualified is banned), observation (that is ingestion’s).
Finding	What an Investigation concluded about its subject — a summary, a classification, and the Steps it rests on. A snapshot of what was seen at that time, never a live view, and never an input to whether a notification is sent. It may cite other Cases; citing is not drawing an Incident. (ADR 0053, accepted 2026-10-02.) Avoid: verdict, evidence (both are drills’), insight, recommendation.
Remedy	A change to a cluster that an Investigator proposes and oto executes only once a human approves it — any write Tool or command the ToolServer’s permissions allow. A Remedy names the write Tool that would carry it out, or says that no configured Tool can — and then it cannot be approved. It is proposed, then approved and executed or failed, or declined, or expired, each by a named actor. Only a user granted approval on that Remedy’s ToolServer may approve, and a Remedy judged risky needs two different such users. Every transition is sent to the declared Incident as a fact. Responsibility rests with the approvers and with the permissions the ToolServer was granted. (ADR 0054, accepted 2026-10-02, which supersedes FR-1’s “response effort” refusal for this one noun. No Remedy ships before the one authz approval grant its §4 names.) Avoid: action (the Slack button row), fix, remediation, runbook, workflow.
Classification	The class a Finding places its subject in, chosen from a closed set the operator wrote — or unclassified, which is always admissible and is the right answer under doubt. oto ships no classes of its own. A Finding keeps the class it was given even if the set later changes. (ADR 0053, accepted 2026-10-02.) Avoid: severity, priority (both banned as human-set axes), category, label (that is a Prometheus label), noise (oto’s opinion of someone else’s signal).
Suggestion	A change a Finding proposes and only a human can apply: to oto’s own configuration (a policy’s count condition), or to an Incident’s membership. It is applied or it lapses; there is no reject verb. Never a change to anything outside oto. (ADR 0053, accepted 2026-10-02.) Avoid: proposal, recommendation, action, fix, remediation.
Silence	A read-only mirror of an Alertmanager silence.
Stanza	One named, ordered unit of a rendered message: title, body, fields, members, trail, rule, actions, footer (SPEC §H.7’s block budget). ⚠️ IT IS AN INTERNAL NOUN AGAIN. It named a customer-facing override slot under ADR 0037; ADR 0050 withdrew that, and a Stanza is now only how the Slack renderer talks about blocks Go builds. Nothing an operator writes names one. It is not a block — that is Slack Block Kit’s own type name, and it cannot name the same unit on the webhook renderer.
NotificationTemplate	One whole notification message an operator wrote — a document you can read top to bottom, not a set of overrides (ADR 0050). Three formats: card is Markdown plus :::fields and {{ actions }}, parsed to oto’s own document IR and compiled per provider, and the only PORTABLE structured one; text is one flat string; raw is literal Slack Block Kit JSON, pinned to Slack. It carries no when clause — notification_policies.template_id names it, because the policy already has the matchers. Every interpolated value is markdown-escaped unconditionally in card, with no opt-out, so a label cannot become syntax. Any failure falls back to oto’s built-in card, which is what makes it impossible for a template to mark a delivery dead. It is not a “Wording” — that was the retired per-Stanza override.

Ambiguity bans: unqualified event; issue; “notification” meaning a Slack message (that is a Delivery); “group” meaning a UI grouping (that is a view); “alert” meaning a case or a Slack message.

⛔ AlertGroup IS A RETIRED NOUN AND MUST NOT COME BACK (git-bug 7570090, migration 00069). It named one generation of a notification grouping, keyed by (org, cluster, alertname, namespace-or-∅) from the alert’s own labels (ADR 0038), and its glossary row said what it was for in the same breath as it named the object: it “decides which facts share a message” and “owns exactly one Slack thread”. Deciding which facts share a message is the definition of a notification policy, and the row held no fact an operator supplied and none upstream sent — it held a decision oto made, identically for every org forever. alert_groups is deleted, alert_cases.group_id with it, and the conversation is the Case. Three consequences worth stating because each is a place the noun tries to return: the axes above are no longer any entity’s identity (see SPEC §C.4); alert_group is gone from the subject_kind and conversation_kind vocabularies, and a value still spelling it is REJECTED, not canonicalised — ConversationKind.SubjectKind() (internal/notification/domain/notification.go) is total over the two kinds that exist and returns the empty SubjectKind for anything else, which threads_subjkind_ck refuses at the write. That is a loud failure rather than a thread silently opened under the wrong key, and it is what the tests assert: rejection, not a quiet rewrite into case. ⛔ Do not add a canonicalising arm. Mapping the old spelling onto case would be widening the kind to mean “generation or case”, which is the one thing the retired constant’s own comment warned against — a generation could hold many Cases and a conversation may not, so the cardinality changed with the spelling. Migration 00069 answered the old spelling by narrowing the CHECKs (threads_subjkind_ck, notifications_subjkind_ck, notifications_convkind_ck), not by giving the decoder an arm. And the plurality Reasons new_alerts and some_resolved left with it, because arithmetic over a set of one has no answer to give.

Scope bans — these MUST NOT appear in a Go identifier, a table or column name, a JSON field, an API path or UI copy. AC-49 greps for them in CI:

escalation · on-call/oncall · rota · schedule · assignee/assigned_to · owner_id · responder · triage · postmortem · war room · SLA · MTTA · MTTR · severity override · close (of an alert) · merge · dismiss · watcher/subscriber

Say firing duration, not MTTR. ⚠️ incident was on this list until ADR 0052 made an Incident an oto noun. What it guarded is still shut, and by the right instrument: incident_id stays banned on a signal row (alerts, alert_cases, notifications, notification_deliveries — tools/lintvocab enforces exactly that), and the incident’s response is PERMANENTLY OUT below. ⛔ escalation has no replacement term and this line used to offer one: it said “say unacked reminder”, and the one reminder stage escalation had been reshaped into was itself withdrawn (git-bug bd0fb1d) — see the No unprompted reminder, at all row above. The word stays banned with nothing to say instead, which is the point: there is no oto concept in that shape.

The four states and THREE orthogonal axes

firing · suppressed · resolved · expired, plus two orthogonal axes: ack_state (unacked|acked) and snooze (§B.8). An acked alert is still firing. A snoozed alert is still firing and must still be rendered as firing — colouring it calm would be a lie.

The four words are the ALERT’s. They live on alerts.state. alert_cases.state holds open | closed and nothing else (ADR 0040): an episode’s only fact about itself is whether it is still running. The four-way reading of a Case is derived and total — open + no suppression_reason is firing, open + one is suppressed, closed + resolve_reason='upstream' is resolved, closed + 'timeout', 'silent' or 'source_removed' (ADR 0056) is expired. Say an episode is open or closed, and an alert is firing, suppressed, resolved or expired; the two vocabularies are not interchangeable and swapping them is how the column acquired four values in the first place.

Snooze suppresses oto’s own notifications for one alert_key until T. It is stored only in oto (alert_snoozes), auto-expires (5 min…30 days, never indefinite), is attributed and visible, and touches nothing in the cluster. It is nearer to channels.verbosity than to a silence. snoozed is NOT a suppression_reason and NOT a state — alert_cases.suppression_reason mirrors Alertmanager’s four reasons and nothing else. Snooze records itself as notifications.suppressed_reason='snoozed'. A snoozed and silenced alert shows both facts.

Rules you must not get wrong:

• expired ≠ resolved. resolved requires an explicit upstream status="resolved". expired means we stopped hearing about it. Never fabricate a resolution.
• suppressed is invisible to webhooks. Alertmanager’s MuteStage drops muted alerts before the webhook, so only the reconciler can enter suppressed (T3). But either the reconciler or ingest can leave it (T4): a webhook arrival is positive proof of non-suppression, because a suppressed alert would never have been sent. The asymmetry is deliberate.
• A closed case never reopens. A re-fire always opens the next seq, unacknowledged, whatever the clock says (T7; there is no T8 — ADR 0040). An acknowledgement is a receipt for one firing, and the second firing is not the one that was signed for. refire_grace used to decide it and is now deleted outright — the setting, its bounds, its derived default and its two API surfaces (git-bug 7287b28, migration 00071). ADR 0040 removed the transition; the knob outlived it by a release, validating a number nothing read.
• ended_at is clamped to started_at. A backward-skewed upstream clock must never abort an ingest transaction. Clamp, flag clamped: true, measure the skew — never reject.
• Losing sight of an alert is not the alert resolving. The reaper is blocked unless the case’s cluster has a live source and every live source’s source_health.status = 'healthy' — for timeout and silent alike, so one unhealthy HA replica holds the whole cluster (ADR 0056 Amendment 1). source_removed is the one expiry it does not ask, because there is no source left to be blind to: it fires only when no live source has fed the case’s cluster for a resolve grace, so deleting one HA replica ends nothing. silent and source_removed ship behind jobs.expire_silent_and_removed, off by default (ADR 0056).

---

4. Module map

Module	Mark	Responsibility
platform	CORE	Config, logging, telemetry, two DB pools + tx, httpx, authn, jobs, secrets, ratelimit, errs, clock, id, and tuning — the one home of the shipped §D.1 defaults (§5.2c). Not a domain.
identity	CORE	Orgs, users, sessions, PATs and ingest tokens → Principal + TenantScope.
sources	CORE	Alertmanager/Prometheus registry, credentials, health; owns the AM v2 + Prom v1 clients.
ingestion	CORE	Durably accept raw batches and normalise them to Observations. Nothing else — and not the reconciler: SPEC §I.2’s tree draws ingestion/worker/reconcile_source, but it is implemented in sources (internal/sources/service/reconcile.go, which says so), because every collaborator it needs is owned there.
alerts	CORE	Identity/dedup, the case state machine, the append-only timeline. The heart.
rules	CORE	Fetch, content-address, version and diff rule definitions at fire time.
enrichment	CORE	The Enricher port, the budgeted pipeline, caching, provenanced results.
notification	CORE	Policy matching, idempotent intents, fan-out, thread ordering, throttle/damping.
channels	CORE	Channel/Provider/Renderer ports + Slack and generic-webhook impls.
streaming	CORE	ui_events, LISTEN/NOTIFY bridge, SSE hub with resume.
silences	PERIPHERAL	Read-only mirror of AM silences. No write path.
stats	PERIPHERAL	Alert-hygiene accounting. Never per-person.
drill	PERIPHERAL	Synthetic end-to-end delivery drills. It imports no other module: it reaches five of them — alerts, ingestion, notification, channels, rules — by writing their table names into SQL (alerts, alert_cases, alert_events, ingest_batches, ingest_rejections, notifications, notification_deliveries, notification_policies, channels, channel_threads, rule_snapshots). Those eleven, plus its own delivery_drills, are DECLARED in test/arch/sqltables_test.go with their owner and how far the drill may go against each. The reads stay SQL on purpose — a port satisfied by the owning module’s service would have the drill ask the code under test whether the code under test worked.
⛔ grouping	DELETED	It owned durable groups, generations, derived membership and group lifecycle — 20 non-test files, 5 243 LOC — and it is gone (git-bug 7570090, migration 00069). ⭐ The measurement is the lesson: deleting it produced four build errors, all in internal/app. notification never imported it and alerts carried only a uuid. The module coupling was thin and the concept coupling was broad, which is why the change touched 187 files and almost none of them were the module. The conversation is now the Case (see the glossary), so nothing replaced it.
app	WIRING	The composition root. Constructs every concrete, satisfies every port, registers the workers and routes. THE one place allowed to know every module, and deliberately outside every cross-domain rule. Not a domain.
incidents	ADR 0052	Incidents drawn over Cases by a Correlator or a human; active/quiet read off member Cases; declared outbound as a notification, facts only. Replaces the deferred correlation row.
investigator	ADR 0053, 0054	Investigators, Investigations, Steps and Findings over oto’s own history and operator-configured ToolServers, asynchronously and off the notification path; Remedies once the authz grant exists. Models are reached through one port and one OpenAI-compatible adapter; ToolServers through the official MCP client; oto’s own loop records Steps and enforces budgets. ⛔ “anything AI” was DEFERRED-POST-V1 in the row below until 2026-10-02, when ADRs 0053 and 0054 were accepted.
k8scontext, changefeed, views, audit (config changes only), authz (except the one Remedy approval grant ADR 0054 §4 needs — remedy_approver_grants, read through identity, written only by oto grant / oto revoke from the host shell, never by a route), extra channel providers	DEFERRED-POST-V1	Do not build. Do not stub beyond the ports that already exist. ⛔ anything AI — moved to investigator above.
Incident response (status, lead, severity, comms, write-up), oncall, assignment, multi-stage escalation, paging, status pages, postmortems, SLA/MTTA, manual resolve/merge/close, watchers	PERMANENTLY OUT	There is no version of oto containing these. Adding one needs an ADR arguing against FR-1 by name. See SPEC §I.1.1 for the hand-offs.

Dependency direction

⚠️ Four different mechanisms cross a module boundary and only one of them is an import. An earlier version of this section drew all of them as one kind of arrow, which made it wrong about direction in one place and wrong about being enforced nearly everywhere. They are drawn apart now.

1. Compile-time imports — the real DAG, and the only edges anything checks.

enrichment ──► rules ──► alerts
                          ▲   ▲
              silences ───┘   └─── sources

⛔ alerts imports no other module. Everything it needs from enrichment, notification and streaming is a port it declares itself in alerts/service/deps.go. This is what lets oto run with notifications entirely disabled — which is how the first correctness tests run — and it is why alerts is the sink of this graph, not its source.

⛔ alerts ◄── grouping WAS DRAWN HERE AND BOTH THE ARROW AND ITS MODULE ARE GONE, and the paragraph it carried is kept because the lesson outlived the edge. It read: “alerts ──► grouping was drawn here for a long time and it is BACKWARDS — grouping/service and grouping/api import alerts/service; alerts imports neither. A contributor who trusted the old arrow would have written the import that this arrangement exists to prevent.” A drawn edge pointing the wrong way is worse than no edge, because it reads as permission.

⚠️ A COMMENT THAT DESCRIBES DELETED MACHINERY IN THE PRESENT TENSE IS A DEFECT, AND THIS IS THE STANDING INSTRUCTION RATHER THAN A LIST. internal/app/adapters.go’s alertObserver was the worst case — it called itself “the one place that may know both alerts and grouping” and described resolving an alert_groups generation as something it does — and it is now written in the past tense. That is the shape to copy: say what the seam used to do, name what deleted it (git-bug 7570090, migration 00069), and keep the argument for the seam when the argument outlived the work. There is no grouping to know and no group to resolve, so a present-tense sentence about either is simply false.

⛔ DO NOT READ THAT AS “THE LAST ONE IS FIXED”, AND NO COUNT IS RECORDED HERE ON PURPOSE. 7570090 deleted an entity that was named in prose across the tree, so present-tense references to grouping, alert_groups, a group id, a member set or an /alert-groups/* route survive in every file nobody has re-read yet. They are found by reading, not by grep — the same words are correct in a past-tense sentence — and a number written down here would be stale the moment somebody fixes one more site, which is how “the last one” got written in the first place. Fix what you find, in the file you are already in.

Two cross-module imports are not module dependencies and are drawn nowhere above:

• RULE K — every module may import alerts/domain, the shared domain kernel (§5.2b). ingestion, rules, silences, sources and notification all do.
• RULE V — notification may import channels/domain, because §F.2 has notification/service build channels/domain.NotificationView and hand it to a Renderer whose concrete it never names. channels/service is injected.

2. Ports — the consumer declares the interface, internal/app/container.go satisfies it. No import exists in either direction, and nothing enforces the arrow:

Consumer declares	Satisfied from	Wired at
ingestion/service.AlertObserver	alerts	app.alertObserver (adapters.go)
alerts/service.EnrichmentReader	enrichment	container.go
alerts/service.NotificationReader	notification	container.go
alerts/service.StreamAppender	streaming	adapters.go
notification/service.ChannelRegistry	channels	container.go
rules/service.RuleLookup	sources/service.ResolveRule	adapters.go
silences/service.SilenceSource, silences/api.SourceBaseURLs	sources	app/silencesource.go
alerts/service.CaseOpenings	the outbox (incidents.correlate) — never incidents itself	app.caseOpenings (adapters.go)
incidents/service.Announcer	the outbox (notify.incident, and investigations.incident for a draw or a membership change) — never notification or investigator itself	app.incidentAnnouncers (adapters.go)
sources/api.CaseCounts — open and held Cases per source (ADR 0056 §1)	alerts	app.sourceCases (adapters.go)

3. River job enqueues — a STRING in internal/platform/jobs/kinds.go, not a call. The producer never names the consumer, so there is nothing to enforce at all:

Producer	Kind	Handled by
alerts, ingestion, enrichment	notify.evaluate	notification
alerts, enrichment	enrich.run	enrichment
alerts (through CaseOpenings)	incidents.correlate — on lifecycle, never notify	incidents (the Correlators)
incidents (through Announcer)	investigations.incident — on lifecycle, never notify or investigate; not on quiet (ADR 0053 §4)	investigator
investigator (through FindingDeclarer, RemedyDeclarer)	notify.incident with Reason finding or remedy_* — an Incident’s new Finding or a Remedy transition, declared, never a decision about delivery. Numbered like every Incident fact: the adapter takes the Incident’s next sequence from incidents/service.NextFactSequence in the same transaction (migration 00093)	notification

4. Table names in SQL — no Go edge whatsoever. drill reads five other modules’ tables by name (see its row above); notification/repository/snapshot.go joins alert_sources to learn a source’s kind so it can decide whether an Alertmanager silence URL is one oto can vouch for. alerts/repository/case.go reads sources’ alert_sources and source_health by name too: the reaper’s guarded scans pre-filter on “every live source on the cluster has a healthy row” (liveSourcesHealthySQL, ADR 0056 Amendment 1), and the case-sources reads join alert_sources for the live set and its max silence. ⚠️ The pre-filter is not the verdict — the §B.4 guard is still asked through the SourceHealth port and that answer decides — but a rename of either table or of source_health.status’s 'healthy' breaks the reaper at runtime, not at build time. No compiler, no depguard rule and no import graph can see any of these — test/arch/arch_test.go says so itself: “COMPILE-TIME EDGES ONLY.”

test/arch/sqltables_test.go is the gate that reads the SQL instead, and it covers drill only. Every table internal/drill/** names is declared there with its owning module and its permitted access: a fourteenth table fails CI the way a new import does, a declared table nothing names any more fails as stale, a SELECT that grows into a DELETE fails even though the table was already declared, and a table its owner renames fails on the OWNER’s side rather than at runtime on the drill path. It also holds dispose.go’s two stated invariants — every DELETE scoped by an id and not merely by org_id and a predicate, and AND synthetic still on alerts — which were argued in a comment on a file nothing in the build system knew was special.

⚠️ notification‘s alert_sources join, alerts’ alert_sources/source_health reads and stats’ ten borrowed tables are not declared anywhere. For those a rename still breaks at runtime, and gating them means writing their claims.

⛔ notification ──► silences used to be drawn and is not a relationship: notification neither imports silences, nor declares a port onto it, nor enqueues to it. The silence links it renders point at Alertmanager’s own console, from the source kind read in (4). The real silence edges point the other way — silences ──► alerts (import) and silences ──► sources (port).

What actually enforces this

• test/arch/arch_test.go is the only gate on direction. It reads the real import graph, fails on any cross-module edge diagram (1) does not draw, fails on a declared edge the code no longer has, and refuses an allow-list containing a cycle — so the cheap fix of adding the offending line back does not work either.
• .golangci.yml’s <module>-must-not-reach-into-other-domains rules are symmetric. Each one re-allows every other module’s /service, so they enforce layering (you may reach only <other>/service, never its api/repository/domain) and say nothing about which way an edge points. The only directional depguard rule is dependency-direction-alerts-and-ingestion — alerts and ingestion may not import notification or channels (SPEC §I.1). platform-must-not-import-domains keeps platform out of the graph entirely.
• test/arch/sqltables_test.go is the only gate on TABLE OWNERSHIP, and only over drill. It reads string literals, not imports, so it is the one check in this repo that can see a module reaching across a boundary without an edge to reach it with.
• All three skip _test.go. Test files do cross module lines on purpose, and a fixture that seeds another module’s table to set a test up is not the module reaching across a boundary.
• Mechanisms 2 and 3 are enforced by nothing, and mechanism 4 only for drill. That is the price of the decoupling, and it is the reason they are listed rather than drawn as arrows.

---

5. Layering rules (mechanically enforced — do not work around them)

internal/<domain>/
  api/          HTTP handlers, DTOs, mappers, routes
  service/      business logic + PORT INTERFACES it declares itself
  repository/   SQL, row structs, mappers
  domain/       pure Go: entities, invariants, state machine. NO I/O IMPORTS AT ALL.

1. api MUST NOT import repository. repository MUST NOT import api.
2. domain imports no pgx, database/sql, net/http, os, time.Now (use platform/clock), slack-go or client-go. encoding/json IS permitted — it does no I/O. json:"…" struct tags in domain are forbidden; tags are what would quietly turn a domain type into a DTO. 2b. internal/alerts/domain is the SHARED DOMAIN KERNEL and is the only sanctioned cross-domain domain import: it owns LabelSet, AlertKey, GroupKey, RuleFingerprint, IdempotencyKey, ClusterKey, SlackTS, Severity, State, AckState. It must import no other domain package. pkg/alertkey does not exist — pkg/ is reserved for otoclient. It is the only kernel but not the only exception: depguard’s RULE V also lets notification import channels/domain, because §F.2 has notification/service BUILD channels/domain.NotificationView (§4, mechanism 1). That grant is one module, one package, view types only. There are exactly two such imports in the tree and a third needs an ADR. 2c. internal/platform/tuning is the ONE home of the shipped §D.1 tuning defaults. It is constants and nothing else — no types, no behaviour, no import but time. identity/domain still OWNS the tenant’s tuning (the keys, the bounds, the provenance, the Settings struct); what moved is only the shipped NUMBER, because alerts/domain, alerts/service and platform/config all need it and rule 4 forbids every one of them importing identity. They used to keep copies with a ⚠️ comment each, ADR 0026 moved three defaults at once and two copies were missed, and the miss is silent — a stale fallback is what an org gets when its settings row fails to load. Every declaration outside this package is now a REFERENCE to it, which the compiler keeps honest; a default only one package reads stays where it lives. This is not a general dumping ground: platform may import no domain (rule 7), so nothing with a domain type in it can come here.
3. service imports domain and its own port interfaces. Concretes are injected by internal/app/container.go.
4. Cross-domain calls are service → service, through an interface declared by the consumer. Never repository → repository.
5. Three model sets, and the compiler can tell them apart: api.XxxDTO (json + validate tags) ← domain.Xxx (pure) ← repository.xxxRow (unexported). No DTO may embed a domain type or a row type.
6. Every repository method is (ctx, db.TenantScope, …). TenantScope has an unexported field and can only be built from an authenticated principal. There is no repository method without one.
7. Every composite index starts with org_id.
8. List methods take db.Keyset and return db.Cursor. There is no OFFSET in this codebase.
9. Transactions travel in ctx (db.FromContext). There are no WithTx(tx) method variants.

---

5b. Validation — seven layers, two opposite trust models (SPEC §L)

#	Layer	Library	Failure
1	API DTOs	go-playground/validator/v10 via httpx.Bind[T] only	422 + violations[]
2	Untrusted inbound (webhook, reconciler)	hand-written bounds B1–B19	ingest_rejections row, still 202
3	Domain invariants	value objects + New…() (T, error)	typed error; a bug
4	Provider config	santhosh-tekuri/jsonschema/v6	422 mapped from schema errors
5	Outbound render	18 Block Kit checks (V1–V18)	dead delivery, payload persisted
6	Persistence	CHECK / NOT NULL / UNIQUE / FK	500 + alert (means 1–3 have a hole)
7	Frontend	valibot	inline field error / dev-time throw

The rule that matters most: trusted user input and untrusted upstream payloads have opposite policies. API bodies use DisallowUnknownFields and reject loudly. Alertmanager bodies decode leniently (routeLabels is undocumented; Grafana sends a superset), reject per-alert into ingest_rejections, and still return 202 — a 4xx deletes the alert forever.

Other binding rules:

• No handler calls json.NewDecoder or validate.Struct directly. httpx.Bind is the only door.
• Violation field paths are JSON names, /-separated (matchers/0/name), never Go names.
• A bound (length, count, enum, regex) lives in three places — DTO tag, domain constructor, DDL CHECK — and they must be identical. TestValidatorMatchesDDL enforces it.
• No optional Validate() in domain. If you can construct it, it is valid.
• One JSON Schema per provider serves both server validation and the settings form.
• Never send a Slack payload that has not passed render/slack.Validate.
• The repository never validates a business rule (service’s job) but does reject malformed row models and is the single place SQLSTATEs become errs.Kind.
• Error taxonomy: validation (422) · conflict (409) · precondition (412, wrong entity state) · upstream-failure (502/504) · unavailable (503, our backpressure). A unique violation on an oto-computed key is not an error — it is the idempotency mechanism.

5c. Brand and UI (SPEC §M)

oto (音) is Japanese for sound — a chime. Make one clear sound at the right moment. Calm by default, unmistakable when it matters.

Two tiers, strictly separated:

• Tier A — chrome is pastel. Surfaces, borders, nav, tables, forms, gridlines. Never a state hue. Brand accent is periwinkle (#5B54D6 / #A6A0FF).
• Tier B — saturated colour means alert state and nothing else. No decorative accent, no chart series, no hover. Scarcity is what makes it loud.

Each state ships four tokens: -fill (pastel surface), -border (≥3:1), -text (≥4.5:1 on its fill), -solid (saturated, ≥3:1). A critical row = pastel fill + saturated 3 px left bar + dark red text. Tokens and measured contrast ratios: SPEC §M.4–M.5.

• Colour is never the only channel — every state carries ≥2 of {colour, icon, text label}.
• Severity → icon. State → colour. Dark mode is the default. No flashing, ever.
• Colour is not the only axis with a vocabulary any more (§M.8, ADR 0029). Type is six steps — text-micro 10 / text-meta 11 / text-body 12 / text-item 13 / text-title 14 / text-page 18 — and radius is three: rounded-chip 3 (inline things), rounded-control 4 (things you operate), rounded-surface 6 (things that hold controls). Both were read off 342 existing bracket literals rather than drawn, which is why they are that short. A size written at a call site is a violation, not a shortcut — bracket, Tailwind’s own text-sm/rounded-md ladder, or a raw declaration in a stylesheet alike; web/src/design/scales.test.ts is the gate.
• The Slack palette (§H.2, Grafana OnCall hexes) is a SEPARATE, UNCHANGED system — do not harmonise it. Different substrate, different contrast contract, and those values are the best-tested alert palette that exists. A renderer must not read a --oto-* token; a stylesheet must not contain a Slack hex. Both halves are gates — test/design/boundary_test.go, plus TestSlackPaletteUnchanged, which pins the six hexes to §H.2 itself rather than to palette.go.

6. Conventions

Go

• Module github.com/thulasiram/oto. Go 1.24. Postgres 16+. IDs are UUIDv7 from platform/id.New().
• All timestamps TIMESTAMPTZ UTC. Every AlertEvent carries two: occurred_at (upstream claim, displayed) and recorded_at (oto clock, ordered by). Never conflate them.
• Errors: stdlib errors + platform/errs; mapped to RFC 9457 problem+json in exactly one place.
• Enums are TEXT + CHECK in SQL, closed value-object types in Go. No Postgres ENUM types.
• Constraint and index names are a runtime contract — §L.9 returns them as errs.Error.Code on 23505/23503. <table>_<purpose>_ck|_uniq|_idx. No ck_/ix_/uq_ prefixes. Every constraint is named; the realised schema has 201 named CHECKs across 30 tables.
• Severity is strict in the domain, lenient at the boundary. NewSeverity (strict, for API and config) vs SeverityFromLabel (total, never fails, for upstream labels). alerts.severity stores the raw label value so users can filter on their own vocabulary (sev1, P1, page).
• errs owns Kind/Error/ProblemDTO/HTTPStatus() with no I/O imports so domain can import it; httpx.WriteProblem does the writing.
• Hand-written SQL over pgx/v5. No sqlc, no ORM. squirrel only for the alert-list filter builder.
• Migrations are goose .sql in db/migrations/, expand/contract only. Never a destructive migration in one release. Assume N and N+1 run simultaneously.
• Every job payload carries "v": <int>. A worker meeting an unknown version parks it.
• slack-go/slack is pinned to an exact version, floor v0.23.1 (earlier versions accept an empty signing secret and therefore forged requests).

Testing

• Real Postgres via testcontainers-go. Mocked DBs lie about SQL semantics.
• Renderers are pure functions with checked-in testdata/*.golden.json + a CI Block Kit validator.
• Real upstream payloads live in test/fixtures/ and are replayed deterministically.
• Write test/load/burst_test.go (a 5 000-alert batch) before the feature that handles it.

Ingest path — the rules that must never be broken

• 202 on accept. 503 + Retry-After for anything transient. NEVER 429. NEVER 4xx. 4xx means Alertmanager permanently deletes the alert.
• No outbound network call on the ingest path. p99 < 250 ms.
• Two connection pools: ingest (25 %, 2 s statement timeout, 500 ms acquire) and general. UI queries must never starve ingestion.
• A 2xx is a promise. Never return 2xx for a payload we failed to persist.

Slack

• Title is a section, never a header (header is plain_text — no bold, no links).
• Exactly one attachment wraps all blocks. Colour = STATE. Emoji = severity.
• chat.update in place is primary; thread replies are the exception. repeat interval elapsed → update only, never post.
• ts is TEXT, never a float. The durable handle is (channel_id, ts), and channel_id comes from the API response.
• Top-level text is a complete sentence — it is the push notification, the search snippet and the only thing screen readers read.
• Button value is an opaque UUID. Never a payload, never trusted.
• Every URL button still delivers an interaction payload you must ack (oto.noop.*).
• oto never reads Slack back. Our DB is the memory of Slack.

Product and ethics (requirements, not aspirations)

• Default to quiet, but never silent. Grouping and flap damping ON by default; snooze is the manual sibling. All of them are visible UI states, never silent suppression — silence destroys trust. ⛔ Storm collapse was removed for failing exactly that test (ADR 0042): oto never decides a real firing was not worth mentioning.
• Never claim resolved when we mean expired.
• Delivery failure must be visible per alert. oto’s silence must never be indistinguishable from “no alert”.
• Ack identity IS stored (operationally necessary). No per-person response-time metrics, leaderboards or aggregates. A feature you do not build cannot be misused.
• Label redaction runs before the raw payload is persisted. Never log full payloads at info level. Retention: raw 30 days, events 13 months, configurable — and it deletes the narrative, never the record (ADR 0024). alerts, alert_cases, rule_snapshots, notifications, notification_deliveries and channel_threads have no reaper. What dies at 13 months is the timeline, including human comments, which live nowhere else.
• Surface alert hygiene. The best alert is the one that no longer exists.

---

7. Where to start reading

You are building	Read
Migrations	SPEC §D in full, §C for the key formats
ingestion	SPEC §C.5, §G.1–G.4, §G.8; ADRs 0006, 0007
alerts	SPEC §A, §B, §C.2–C.3, §C.8, §D.4; ADRs 0003, 0004
rules	SPEC §C.6, §D.6, §F.4; ADR 0009
notification + channels	SPEC §F.1–F.2, §G.5–G.7, §H in full; ADRs 0008, 0023
streaming	SPEC §E.4, §D.10; ADR 0010
HTTP API	SPEC §E in full, §L.1–L.2, §L.9, §J
The SolidJS UI	SPEC §E, §L.8, §M in full; ADR 0012
Anything at all	SPEC §L (validation) and SPEC §I.1.1 + SCOPE-BOUNDARY (the line); ADRs 0011, 0013
Work already queued	SPEC §P — pending code amendments, 22 numbered items

docs/design/domain-research.md is verified ground truth about Alertmanager and Slack. When your instinct disagrees with it, your instinct is wrong. docs/design/architect-proposal.md is superseded and is kept only for rationale; where it conflicts with SPEC.md, SPEC.md wins.

8. If the SPEC is ambiguous

Raise it. Do not choose. An undocumented choice made by one implementer is a compile error or a silent behavioural divergence for the next four. Amendments go through docs/adr/ plus an edit to the affected SPEC section in the same commit.