Skip to content
oto
Menu
Overview
IntroductionConceptsArchitectureContributingReleasing
Design
oto — System Design ProposalDomain Research: Prometheus Alertmanager + Slackoto — Red-Team Memooto — SCOPE BOUNDARY (the doctrine)oto — BINDING SPECIFICATION v1
ADRs
0001 — PostgreSQL is the sole datastore; River is the job queue0002 — `internal/<domain>/{api,service,repository,domain}` with three model sets0003 — Alert, AlertOccurrence and AlertEvent are three distinct entities0004 — oto owns alert identity; Alertmanager's fingerprint is recomputed, not trusted0005 — A durable group key, and the AlertGroup generation owns the Slack thread0006 — The Alertmanager API v2 reconciler is mandatory, not optional0007 — Webhook response contract: 202 or 503. Never 4xx, never 429.0008 — Slack: `chat.update` in place is primary; threads are secondary; one colour attachment0009 — Rule definitions are snapshotted and versioned at fire time0010 — Server-Sent Events with durable resume, not WebSocket0011 — Validation is seven layers with two opposite trust models0012 — Pastel product chrome, saturated colour reserved exclusively for state0013 — oto is alert-first: the Flight Recorder Test defines the scope boundary0014 — Postgres is the only store; no TSDB or column store in v10015 — No stream processor; River in Postgres is the pipeline0016 — Cluster context arrives via MCP, on demand; never the Kubernetes event firehose0017 — Alertmanager-style matchers, not CEL; and a filter AST that keeps the door open0018 — Slack distribution: a customer-built internal app, credentials in config0019 — MIT licence0020 — Broadcast the transitions that must be seen0021 — Correctness and testing strategy0022 — Length-prefixed identity pre-images0023 — Terminal thread states are decided first, and the send is three phases0024 — Retention defaults, what each boundary destroys, and cold storage0025 — The alert list resolves rule snapshots in a batch, not per row0026 — `refire_grace` and the flap thresholds, derived from a real rule corpus0027 — Delivery drills: a synthetic alert through the real pipeline, and the mark that keeps it out of the numbers0028 — Non-urgency motion: one-shot events, honest activity loops, and nothing else0029 — Type and radius scales, read off the product rather than drawn for it0030 — Tier A repainted as Washi & Ink / Konshi Sutra; accent moves to beni0031 — Sharp corners and higher-contrast grid lines within Washi & Ink; colours unchanged0032 — Dark-theme accent moves from beni to plum-blossom blue-and-white; light untouched0033 — The alert filters are a toolbar above the table, not a panel in the rail0034 — Notifications is a destination, and the record it keeps is readable0035 — Decorative ink as a masked wash: one primitive, four motifs, and the placements it is refused0036 — AlertOccurrence becomes AlertCase, and `case` is argued against FR-1 by name0037 — A Wording is Liquid, and the structure around it stays oto's0038 — The group key is derived from the alert's own labels0039 — Group close stays activity-driven0040 — A Case is `open` or `closed`, and it is never reopened0041 — The Alert/Case allocation, and the one rule that decides it0042 — Storm damping is removed, because the defence was built before its object0043 — The Slack action row renders five elements, and the fifth is the snooze pair0044 — A count condition is a silence the operator asked for, and `below_threshold` is how oto says so0045 — A Case is a conversation, and one Slack thread per alert is the accepted cost0046 — Corners round again: ADR 0031's radius override is reversed0047 — A channel answers to a connection, and Slack gets two scopes back0048 — A Wording is spelled by its channel, and the marks in between are oto's0049 — A Wording is selected by its own clause, not by a route0050 — A NotificationTemplate is one whole message, and its policy says which one0051 — A `raw` template owns the whole message, and a template may carry its replies0052 — An Incident is drawn over Cases, and its response is handed off0053 — An Investigator reads, proposes, and never decides delivery0054 — A Remedy earns the write path0055 — An incident-tool integration is data or a bridge, never code inside oto0056 — A Case the upstream stopped speaking about expires, and says why
Setup
Configuration referenceReaching an incident tool: the envelope, a mapping, or a bridgeInvestigatorsRemediesConnecting oto to SlackThe live Slack verification runTuning otoConnecting oto to VictoriaMetrics (vmalert)The generic webhook and `oto.notification.v1`
Runbooks
oto runbooksAlert search — optional partial/substring matching on alert namesoto_clock_skew_secondsoto_delivery_claim_lost_totaloto_ingest_accepted_totaloto_ingest_alerts_totaloto_ingest_duplicate_totaloto_ingest_duration_secondsoto_ingest_fingerprint_mismatch_totaloto_ingest_process_duration_secondsoto_ingest_rejected_totaloto_ingest_shed_totaloto_job_duration_secondsoto_job_queue_depthoto_jobs_dead_totaloto_jobs_enqueued_totaloto_jobs_failed_totaloto_jobs_panics_totaloto_jobs_snoozed_totaloto_jobs_started_totaloto_jobs_succeeded_totaloto_jobs_unknown_version_totaloto_render_invalid_totaloto_slack_unknown_action_totaloto_stream_connectionsoto_stream_events_coalesced_totaloto_stream_events_dropped_totaloto_stream_events_fetched_totaloto_stream_events_published_totaloto_stream_fetch_errors_totaloto_stream_listener_reconnects_totaloto_stream_notify_malformed_totaloto_stream_notify_received_totaloto_stream_poll_recovered_totaloto_stream_resync_totaloto_thread_gap_recovered_totaloto_thread_head_wait_secondsoto_thread_order_decisions_total

0018 — Slack distribution: a customer-built internal app, credentials in config

Status: Accepted · 2026-08-08 Relates to: 0008 (chat.update is the primary verb, and oto never reads Slack back), 0013 (oto tells the thing that pages; it does not page) Artefacts: deploy/slack/manifest.yaml, docs/setup/slack.md

Context

Section titled “Context”

oto is self-hosted. Every install runs in someone else’s cluster, behind someone else’s firewall, against someone else’s Slack workspace. There is no oto-operated service anywhere in the picture — no control plane, no hosted tier, no shared hostname. That single fact decides the Slack integration, and it decides it more firmly than it first appears.

A Slack integration can be distributed three ways: as a Marketplace app installed by OAuth, as an unlisted app distributed by OAuth outside the Marketplace, or as an internal app the customer creates in their own workspace. The first two require the app to have a stable identity — a client ID, a client secret, a redirect URL, and an events/interactivity request URL. Every one of those is a property of the operator of the app, and oto has no operator. There is no host to redirect to.

Two further facts, both verified, narrow this further:

  1. Socket Mode removes the public-ingress requirement entirely, and Socket Mode is not permitted for Slack Marketplace apps. The overwhelming majority of self-hosted installs have no inbound HTTPS. Choosing a distribution model that forbids Socket Mode is choosing to make onboarding an ingress-and-certificate ticket.
  2. The May 2025 rate-limit change for non-Marketplace apps cut conversations.history and conversations.replies to 1 request/minute with 15-object pages — but explicitly exempted “internal customer-built apps” (clarification, 2025-06-03). It is the loudest recent argument against unlisted OAuth distribution, and it does not touch the model chosen here.

The real cost of the internal-app model is that the customer builds the app themselves, which means ticking scopes by hand. That is the error-prone step, and its failures are the worst kind: silent at setup, then a missing_scope at 3am.

Decision

Section titled “Decision”

Slack integration is a customer-built internal app. The customer creates the app in their own workspace, installs it there, and pastes the resulting credentials into their own oto config.

  • No OAuth. No client ID, no client secret, no redirect URL, no oauth.v2.access exchange, no installation store, no per-workspace token table keyed by an install event. None of that code exists and none of it should be written.
  • No token refresh. token_rotation_enabled: false. Rotation machinery exists to serve OAuth-distributed apps; an internal app’s bot token does not expire.
  • Three credentials, entered by a human: the bot token (xoxb-, per channel, sealed in channel_credentials with AES-256-GCM against the platform/secrets keyring), the app-level token (xapp-, process config, Socket Mode only), and the signing secret (process config, HTTP mode only).
  • Setup is driven by a checked-in manifest, deploy/slack/manifest.yaml, pasted into Slack’s “Create an app from manifest” flow. The manifest is the mitigation for the one real cost of this model, and it is treated as a first-class artefact: every scope it requests carries a comment naming the Slack API method that requires it and the documentation URL that proves it, and the scopes deliberately not requested are listed with reasons. A reviewer at the customer’s company reads that file before approving the install.
  • The scope list is derived from the code, not from convention. The complete set of Slack methods oto can call is fixed by a four-method interface in internal/channels/providers/slack/channel.go: chat.postMessage, chat.update, auth.test, conversations.info. That yields chat:write, channels:read and groups:read. Anything not implied by those four methods is refused, including chat:write.public — oto must be invited to a channel by a human rather than being able to post into any public channel in the workspace.
  • Socket Mode is the default transport; HTTP is a configuration flag for installs that already have ingress. Both run behind one handler.

Consequences

Section titled “Consequences”
  • No OAuth machinery is needed at all. This is the largest consequence and it is a reduction: a whole subsystem — installation records, state parameters, redirect handling, token refresh, revocation webhooks, the multi-tenant token store keyed by install — simply does not exist. It is also a reduction in attack surface. There is no redirect URL to attack and no client secret to leak.
  • Socket Mode stays viable, and that is worth more than it looks. oto runs inside a cluster. An integration that required a public inbound URL would turn “install oto” into a conversation with the network team about an ingress rule and a TLS certificate, and it is the single largest onboarding cost the product could have chosen to take on. Instead oto dials out. It also removes an entire class of failure: no signature-verification path in the default deployment, and no Slack-side auto-disable of event subscriptions when a deploy makes the endpoint briefly unavailable.
  • The customer bears the app-creation cost. They must create an app, install it, generate an app-level token by hand (app-level token scopes cannot be declared in a manifest — the manifest schema covers only bot and user OAuth scopes), copy three values, and invite the bot to each channel. This is a real cost and it is not hidden. The manifest removes the error-prone part of it; the rest is documented step by step in docs/setup/slack.md.
  • The 2025 read-throttle change is moot twice over. Internal customer-built apps were explicitly exempted, and oto never calls the throttled methods: conversations.history and conversations.replies are absent from the provider’s API interface by design, because oto’s own database is the memory of Slack (ADR 0008). Even if the exemption were withdrawn tomorrow, nothing in oto would change.
  • oto cannot be listed on the Slack Marketplace, and cannot offer a one-click “Add to Slack” button. Accepted without regret: a Marketplace listing is a distribution channel for a hosted product, and oto is not one.
  • A support conversation about scopes is now a conversation about a file in the repository, which is a much better conversation than one about a screenshot of a checkbox list.
  • Enterprise Grid org-wide deployment is possible but untested. org_deploy_enabled: false in the manifest. Socket Mode apps can be distributed org-wide through enterprise deployment options; nothing here forecloses it, and nothing here validates it either.

Alternatives rejected

Section titled “Alternatives rejected”

Slack Marketplace listing with OAuth distribution. Near-incoherent for a self-hosted product. OAuth requires a redirect URL and an interactivity request URL, and for a self-hosted install both would have to be the customer’s own hostname — one per install. That is not a distributed app; it is a per-customer app wearing an OAuth costume, with all of OAuth’s machinery and none of its benefit. It would additionally forbid Socket Mode (a Marketplace requirement), forcing every install to expose public HTTPS ingress, and it would put oto’s release cadence behind Slack’s app-review queue. Rejected on all four counts.

Unlisted OAuth distribution (a “distributed” app outside the Marketplace). Same redirect-URL incoherence as above, with the additional penalty that it lands squarely in the population the May 2025 rate-limit change targeted — “commercially distributed apps that are not approved for the Slack Marketplace”. oto happens not to call the throttled methods, so the penalty would be nominal today, but adopting a model whose limits are being actively tightened, in exchange for nothing, is a bad trade.

Incoming webhooks. The tempting one, because it is the simplest possible setup: one URL, no scopes, no token. Rejected because it cannot do the product. An incoming webhook can only post. It cannot chat.update, which is ADR 0008’s primary verb and the entire reason oto is quieter than stock Alertmanager — every state change would become a new message, which is precisely the noise oto exists to eliminate. It cannot thread properly (no ts is returned, so there is no handle to thread off or to edit). And it carries no interactivity, so there is no Acknowledge button, and therefore no ack attribution and no slack_identities. Choosing webhooks would not be a simpler implementation of oto; it would be a different, worse product.

Ship a hosted oto that owns the Slack app. Out of scope. It would resolve the redirect-URL problem by making oto a SaaS, which is a business decision, not an integration one, and one nobody has made.