Menu

Overview

Introduction Concepts Architecture Contributing Releasing

Design

oto — System Design Proposal Domain Research: Prometheus Alertmanager + Slack oto — Red-Team Memo oto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1

ADRs

0001 — PostgreSQL is the sole datastore; River is the job queue 0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets 0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities 0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted 0005 — A durable group key, and the AlertGroup generation owns the Slack thread 0006 — The Alertmanager API v2 reconciler is mandatory, not optional 0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment 0009 — Rule definitions are snapshotted and versioned at fire time 0010 — Server-Sent Events with durable resume, not WebSocket 0011 — Validation is seven layers with two opposite trust models 0012 — Pastel product chrome, saturated colour reserved exclusively for state 0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary 0014 — Postgres is the only store; no TSDB or column store in v1 0015 — No stream processor; River in Postgres is the pipeline 0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose 0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open 0018 — Slack distribution: a customer-built internal app, credentials in config 0019 — MIT licence 0020 — Broadcast the transitions that must be seen 0021 — Correctness and testing strategy 0022 — Length-prefixed identity pre-images 0023 — Terminal thread states are decided first, and the send is three phases 0024 — Retention defaults, what each boundary destroys, and cold storage 0025 — The alert list resolves rule snapshots in a batch, not per row 0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus 0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers 0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else 0029 — Type and radius scales, read off the product rather than drawn for it 0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni 0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged 0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched 0033 — The alert filters are a toolbar above the table, not a panel in the rail 0034 — Notifications is a destination, and the record it keeps is readable 0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused 0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name 0037 — A Wording is Liquid, and the structure around it stays oto's 0038 — The group key is derived from the alert's own labels 0039 — Group close stays activity-driven 0040 — A Case is `open` or `closed`, and it is never reopened 0041 — The Alert/Case allocation, and the one rule that decides it 0042 — Storm damping is removed, because the defence was built before its object 0043 — The Slack action row renders five elements, and the fifth is the snooze pair 0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so 0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost 0046 — Corners round again: ADR 0031's radius override is reversed 0047 — A channel answers to a connection, and Slack gets two scopes back 0048 — A Wording is spelled by its channel, and the marks in between are oto's 0049 — A Wording is selected by its own clause, not by a route 0050 — A NotificationTemplate is one whole message, and its policy says which one 0051 — A `raw` template owns the whole message, and a template may carry its replies 0052 — An Incident is drawn over Cases, and its response is handed off 0053 — An Investigator reads, proposes, and never decides delivery 0054 — A Remedy earns the write path 0055 — An incident-tool integration is data or a bridge, never code inside oto 0056 — A Case the upstream stopped speaking about expires, and says why

Setup

Configuration reference Reaching an incident tool: the envelope, a mapping, or a bridge Investigators Remedies Connecting oto to Slack The live Slack verification run Tuning oto Connecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`

Runbooks

oto runbooks Alert search — optional partial/substring matching on alert names oto_clock_skew_seconds oto_delivery_claim_lost_total oto_ingest_accepted_total oto_ingest_alerts_total oto_ingest_duplicate_total oto_ingest_duration_seconds oto_ingest_fingerprint_mismatch_total oto_ingest_process_duration_seconds oto_ingest_rejected_total oto_ingest_shed_total oto_job_duration_seconds oto_job_queue_depth oto_jobs_dead_total oto_jobs_enqueued_total oto_jobs_failed_total oto_jobs_panics_total oto_jobs_snoozed_total oto_jobs_started_total oto_jobs_succeeded_total oto_jobs_unknown_version_total oto_render_invalid_total oto_slack_unknown_action_total oto_stream_connections oto_stream_events_coalesced_total oto_stream_events_dropped_total oto_stream_events_fetched_total oto_stream_events_published_total oto_stream_fetch_errors_total oto_stream_listener_reconnects_total oto_stream_notify_malformed_total oto_stream_notify_received_total oto_stream_poll_recovered_total oto_stream_resync_total oto_thread_gap_recovered_total oto_thread_head_wait_seconds oto_thread_order_decisions_total

oto

The alert history layer your Prometheus stack does not have — self-hosted, with a UI, without adopting an AIOps platform or a paid SaaS.

Read the introduction Architecture

Where to start

IntroductionWhat oto is, what it is not, and how to run the whole stack locally.

ArchitectureThe module map, layering rules, and domain language — CONTEXT.md.

ConceptsEvery noun the system has, what each is keyed by, and how they contain one another.

DesignThe binding SPEC, the scope boundary, and the research behind them.

ADRsEvery architectural decision, in the order it was made.

SetupConnecting Slack and tuning flap/storm thresholds for your own rules.

RunbooksOne page per metric oto registers: what it counts, what to do about it.