Finding:routeLabels is emitted on the wire but is absent from the published webhook docs. It is populated from the route’s labels: config (a 0.33-era feature; see A1.1). Do not rely on the docs alone — decode leniently.
Added by “[FEATURE] Add the reason for notifying in dedup stage. #4971” in 0.32.0 (2026-04-08). The webhook docs omitted it until it was fixed in 0.32.3 / 0.33.1 (2026-07-04) — “[BUGFIX] doc: fix missing notification_reason field in webhook documentation (#5329)”.
This is a big deal for us.notification_reason tells us why Alertmanager re-sent a group. "repeat interval elapsed" means “nothing changed, this is a nag” → we should update the existing Slack message, not post a new one. "first notification" → post new. "new alerts added" / "some alerts resolved" → update + optionally thread a delta. Only available on AM ≥ 0.32.0; treat empty string as unknown and fall back to comparing fingerprint sets.
Default max_alerts: 0 = unlimited. Truncation takes the first N in slice order, which is not a meaningful priority order. If the operator sets max_alerts, truncatedAlerts > 0 and we have permanently lost those alert bodies — we can only render “+N more”. Recommend documenting that operators leave max_alerts: 0 and let us paginate in Slack.
webhook_config supports a payload: map of Go templates that replaces the entire default body. The docs carry an explicit “USE AT YOUR OWN RISK … THE ALERTMANAGER TEAM WILL NOT PROVIDE ANY SUPPORT” warning. Our ingest should assume the standard shape but fail soft.
For each: hash(name), hash(0xFF), hash(value), hash(0xFF).
Hash function is FNV-1a 64-bit (hashNew/hashAdd; the file comment states “For our purposes, FNV-1A 64-bit is used”).
Render as exactly 16 lowercase hex chars, zero-padded (%016x).
It is a pure function of the full label set — including alertname, severity, instance, and every other label. It is fully reproducible in our own Go code by importing github.com/prometheus/common/model and calling LabelSet.Fingerprint().String(). We do not need to trust the incoming value; we can recompute and verify.
Added to template data by “[CHANGE] Add Fingerprint to template data. #1945” in Alertmanager 0.19.0 (2019-09-03) (CHANGELOG).
Earlier, “[ENHANCEMENT] Expose the alert’s ‘fingerprint’ (unique identifier) through API (#786)” exposed it on the API only.
Since it is a non-pointer string field with no omitempty, on ≥ 0.19.0 the key is always present in the JSON. On < 0.19.0 it is absent entirely.
Design call: require AM ≥ 0.19.0 (released 2019 — safe), but recompute the fingerprint ourselves from labels anyway so we are immune to the field being missing, and so it stays a derived, not supplied, key.
It embeds the route path. Any edit to alertmanager.yml that changes the matcher set of the matched route (or inserts/removes an ancestor route) changes routeKey, and therefore changes groupKey for alerts that are otherwise identical. Config reloads are routine.
It embeds the rendered group label set, so it changes if group_by changes.
It is unbounded in length — a deep route tree with many matchers plus a wide group_by produces long strings. Alertmanager itself hashes it for internal storage: GroupId: notify.Key(ag.GroupKey()).Hash() (dispatch.go:980), which is a hint that the raw string is not meant as a compact identity.
It is not escaped, so a label value containing : or } makes naive parsing wrong. Treat it as an opaque blob; do not parse it.
Recommended durable key for oto: a hash over (receiver, sorted groupLabels) — stable across route-config edits — with groupKey stored alongside as observability metadata. Fall back to the alert fingerprint set for per-alert identity. This is what lets us keep the same Slack thread across an Alertmanager config reload.
Note the singular/plural asymmetry: /silences (list, create) vs /silence/{id} (get, expire). Easy to get wrong.
Note active/expired/pending on GET /silences are newer state filters (the older API only had filter). UNVERIFIED: exact release that added them.
All four fields are required (arrays are present but may be empty). Note the API has a single suppressed state; why it is suppressed is in the three arrays.
alertGroup: { labels, routeLabels, receiver, alerts[] } — all required. Confirms routeLabels is a first-class concept in current versions.
silence (postable body): matchers (required), startsAt, endsAt, createdBy, comment all required; annotations optional (silence annotations were added in 0.32.0, “[FEATURE] Add silence annotations. #4965”).
if err := api.silences.Expire(ctx, sid); err !=nil { ... }
}
DELETE /silence/{id} does not delete — it expires. It sets endsAt to now; the silence remains queryable with state: "expired". Returns 200, or 404 if unknown.
POST /silences is create-or-update.postableSilence = silence + optional id. Supplying an existing id updates that silence; omitting it creates a new one. Response is {"silenceID": "<string>"} — the operation is described in the spec as “Create / update silence response”.
Server-side validation (400 on failure): startsAt must be strictly before endsAt; endsAt must not be in the past.
POST can return 404 if the supplied id does not exist.
Design call: our “Silence” button should POST a new silence and persist the returned silenceID on the Slack message so the “Unsilence” button can DELETE it. Because expire is idempotent-ish and 404s on unknown IDs, treat 404 on unsilence as success.
“[FEATURE] Add support for multiple matcher set silences. #4957” landed in 0.32.0. UNVERIFIED: how this is represented in the v2 wire model (the matchers field in openapi.yaml is still a flat matchers array) — likely gated behind a feature flag or a newer spec revision. Verify against the target AM version before relying on it.
// If no end time is defined, set a timeout after which an alert
// is marked resolved if it is not updated.
if alert.EndsAt.IsZero() {
alert.Timeout =true
alert.EndsAt = now.Add(resolveTimeout)
}
global.resolve_timeout default is 5m (config/config.go: ResolveTimeout: model.Duration(5 * time.Minute)).
Key insight: when alerts come from Prometheus, EndsAt is always set, so resolve_timeout is never used. It only matters for alerts pushed directly to POST /api/v2/alerts by third parties. A lot of documentation gets this wrong.
Authoritative. Alertmanager decided this alert is resolved.
endsAt in the past
Consistent with resolved.
endsAt in the future (≈ now + 4m)
Still firing; this is the lease, not a prediction.
endsAt == "0001-01-01T00:00:00Z"
Zero value — no end time known.
top-level status == "resolved"
All alerts in the group are resolved.
notification_reason == "all alerts resolved"
Explicit resolve notification.
notification_reason == "some alerts resolved"
Partial resolve within the group.
Rule for oto: trust per-alert status, never infer resolution from endsAt arithmetic. Separately run a staleness sweeper: if we have not seen a firing notification for a group in > repeat_interval + slack, mark it stale/unknown in the UI rather than resolved — because a silenced or inhibited alert also stops arriving (see A6), and so does a dead Alertmanager.
send_resolved defaults to true for webhooks. If an operator sets it false, we will never see resolve notifications, and we must not leave Slack cards firing forever. Detect this by querying GET /api/v2/status (which returns the effective config) and warn loudly, or reconcile via GET /api/v2/alerts.
Notably, the retry stage still logs resolved alerts to the nflog even when send_resolved: false:
notify/retry_stage.go
// If we shouldn't send notifications for resolved alerts, but there are only
// resolved alerts, report them all as successfully notified (we still want the
// notification log to log them for the next run of DedupStage).
A6. Inhibition and silence — what a webhook receiver sees
A silenced alert looks identical to a resolved-and-gone alert from the webhook’s perspective: silence. We get nothing. No “suppressed” status, no notification, no field. The webhook alerts[].status enum is onlyfiring | resolved — there is no suppressed value on the wire.
If a whole group becomes silenced, we simply stop receiving notifications for it. We will not get a status: resolved.
Therefore, if the user silences an alert from our Slack card, we must update the card ourselves — Alertmanager will not tell us. Persist “we created silence X for group Y” locally.
To detect suppression we created ourselves or someone created in the Alertmanager UI, we must poll GET /api/v2/alerts?silenced=true&inhibited=true and read status.state == "suppressed" plus status.silencedBy[] / inhibitedBy[] / mutedBy[]. This is the only way to distinguish “silenced” from “resolved” from “Alertmanager is down”.
Observability: the current metric is alertmanager_notifications_suppressed_total{reason=...} — verified in notify/metrics.go:
Namespace: "alertmanager",
Name: "notifications_suppressed_total",
Help: "The total number of notifications suppressed for being silenced, inhibited, outside of active time intervals or within muted time intervals.",
Beware of stale docs and blog posts: it was introduced (#3565, 0.27.0 era) as alertmanager_alerts_supressed_total — different noun and a typo (one p). It has since been renamed. If our docs or dashboards reference this metric, use the current name and note the rename for operators on older versions.
Architecture consequence: a webhook-only design cannot render silence state. oto needs the API v2 reconciler loop as a first-class component, not an optional extra.
A7. Fetching the originating alert rule definition
Naive approach: alertname label → rule.name. That mostly works, but:
Pitfall 1 — alertname is not unique. Nothing stops two rule groups, or two files, from defining HighErrorRate. rule_name[] can return several rules. Disambiguate by checking that the rule’s static labels are a subset of the alert’s labels, and that annotations keys match. Even then it can be ambiguous.
Pitfall 2 — alertname can be overwritten. An Alertmanager route or a Prometheus relabeling step can rewrite alertname. Also, alert_relabel_configs in prometheus.yml runs before alerts are sent to Alertmanager and can mutate any label, including alertname.
Pitfall 3 — the rule’s labels are templated. Rule labels: values may contain {{ $labels.x }} templates. The /api/v1/rules response returns the raw, untemplated rule definition, but the alert instance carries the rendered values. So a naive “rule labels ⊆ alert labels” subset check fails whenever templating is used. Compare only on non-templated (no {{) label values.
Pitfall 4 — externalLabels are added on the way out. Prometheus adds global.external_labels to alerts sent to Alertmanager. Those labels will be on the alert but never on the rule. Subset checks must be one-directional: rule labels ⊆ alert labels, never the reverse.
Pitfall 5 — multiple Prometheus servers. In a federated / sharded setup, several Prometheis feed one Alertmanager. /api/v1/rules on the wrong server returns nothing. Use generatorURL to identify the origin server: it is externalURL + "/graph?g0.expr=...", built in SendAlerts (GeneratorURL: externalURL + strutil.TableLinkForExpression(expr)). Parse its scheme+host to pick which Prometheus to query, and parse g0.expr to recover the PromQL expression directly, with zero API calls.
Best approach for oto (recommended):
Primary: parse generatorURL. g0.expr is URL-encoded PromQL — decode it and you have the expression as evaluated, no rule lookup, no ambiguity, and it works even for multi-Prometheus setups. This is by far the most robust path for showing “the query”.
Enrichment: for for: / keep_firing_for: / raw rule labels+annotations, call GET /api/v1/rules?type=alert&rule_name[]=<alertname>&exclude_alerts=true on the Prometheus identified in step 1, and cache aggressively (rule definitions change rarely; cache for minutes, invalidate on config reload).
Handle N>1 matches by scoring on non-templated label subset + annotation key overlap, and surface ambiguity rather than guessing.
Pitfall 6 — duration and keepFiringFor are float seconds, not Go duration strings and not milliseconds. 600 means for: 10m.
Pitfall 7 — pagination is not consistent. The docs state: “there are no guarantees regarding the consistency of the response if the rule groups are being modified during the pagination process.” Prefer rule_name[] filtering over paging through everything.
A8. Clustering — can the same alert be delivered more than once?
Prometheus fans out to every Alertmanager. The README is explicit:
“Important: Do not load balance traffic between Prometheus and its Alertmanagers, but instead point Prometheus to a list of all Alertmanagers. The Alertmanager implementation expects all alerts to be sent to all Alertmanagers to ensure high availability.”
So every peer independently has the alert and independently wants to notify.
ClusterWaitStage staggers peers by position. Each peer waits position * --cluster.peer-timeout (default 15s) before notifying:
The notification log (nflog) is gossiped. After peer 0 notifies, it records the notification and gossips that record. When peer 1’s wait expires, DedupStage sees peer 0’s nflog entry and suppresses.
ClusterGossipSettleStage blocks the pipeline until gossip has settled after startup (--cluster.settle-timeout, default = --cluster.pushpull-interval = 1m), so a freshly-started peer does not re-notify everything.
The dedup is time-based and gossip-based, both best-effort. If peer 0’s notification takes longer than 15s, or the gossip does not converge within 15s (network partition, GC pause, --cluster.gossip-interval default 200ms but pushpull is 1m), peer 1 will notify too.
A network partition guarantees duplicates by design — that is the whole point of the HA model. It prefers duplicate notifications over lost notifications.
Retries produce duplicates. If our endpoint returns 200 but the response is lost, or we time out after having processed the request, Alertmanager retries (see A9).
Restarts. nflog is snapshotted to disk, but a crash between notifying and snapshotting loses the record.
The webhook handler must be idempotent. Recommended dedup key:
sha256(groupKey_normalized || receiver || sorted(fingerprint:status for each alert) || notification_reason)
with a short-TTL (say 60s, ≥ n_peers * peer_timeout = 45s for a 3-node cluster) seen-set. Reject/no-op duplicates and still return 200. Do not dedupe purely on groupKey — legitimate repeat notifications share it.
Also: because the webhook can be retried, treat “post to Slack” as the non-idempotent step and guard it behind the dedup check, not the other way round.
s := fmt.Sprintf("unexpected status code %v", statusCode)
...
return retry, errors.New(s)
}
Status
Behaviour
2xx
Success. Notification recorded in nflog. Body ignored.
5xx
Failure, retried.
3xx
Failure, not retried (unless followed by the HTTP client).
4xx
Failure, not retried — permanently dropped.
RetryCodes
Per-integration extra retriable codes. The webhook integration sets none — no default RetryCodes for webhooks, so 429 from our endpoint is not retried.
Critical: returning 429 or 4xx means the notification is silently and permanently lost. If oto is overloaded or rate-limited by Slack, we must return 503, not 429, to get a retry. Never return 4xx for a transient condition. Never return 2xx for a payload we failed to persist.
Also: the response body is ignored on 2xx. There is no way to signal anything back to Alertmanager. Ack fast, process async — but only after durably persisting the payload, because a 2xx is a promise.
// backoff/v5's ExponentialBackOff never returns Stop from NextBackOff, so
// the ticker retries indefinitely until the context is canceled.
b := backoff.NewExponentialBackOff()
tick := backoff.NewTicker(b)
Uses github.com/cenkalti/backoff/v5 with defaultExponentialBackOff settings (v5 defaults: initial interval 500ms, multiplier 1.5, randomization factor 0.5, max interval 60s — UNVERIFIED against the exact pinned v5 version in go.mod).
It retries indefinitely. There is no max-attempts cap. The loop terminates only when the context is cancelled.
The retry deadline (this is what actually bounds retries)
Total retry budget = max(group_interval, 10s) + (peer_position × cluster.peer-timeout).
With the common defaults (group_interval: 5m, single node): 5 minutes of retries. On a 3-node cluster the third peer gets 5m + 30s.
When the deadline expires the failure is recorded as contextDeadlineExceeded on alertmanager_notifications_failed_total{reason=...} and the notification is dropped — the group will be retried from scratch only at the next group_interval tick (and DedupStage may then suppress it, since nothing changed… unless repeat_interval has elapsed).
The docs warn: “NOTE: This will have no effect if set higher than the group_interval.” — because the outer context deadline dominates.
Design call: oto’s webhook handler must respond well under 10 seconds (the floor of the retry budget) even when group_interval is small. Target p99 < 1s: validate → dedup-check → durably enqueue → 200. All Slack I/O happens asynchronously.
Failure-reason metric labels (useful for operators debugging oto)
alertmanager_notifications_failed_total{reason=...}. Values include clientError, serverError, contextCanceled, contextDeadlineExceeded, and — new in main / unreleased at time of writing — authError (HTTP 401/403) and rateLimited (HTTP 429) split out of the generic clientError:
“[CHANGE] notify: The reason label on alertmanager_notifications_failed_total now distinguishes authError (HTTP 401/403) and rateLimited (HTTP 429) from the generic clientError.” — CHANGELOG, main / (unreleased)
A10. Route timing defaults (the cadence we must design around)
Delay before the first notification for a brand-new group. So there is an inherent ~30s floor on alert→Slack latency that we cannot improve. Set expectations accordingly.
group_interval
5m
Minimum gap before sending an update for a group that has changed. Our card can therefore go stale by up to 5 minutes. Also = our retry budget (§A9).
repeat_interval
4h
Gap before re-sending an unchanged group. This is what produces notification_reason: "repeat interval elapsed".
Implication: with defaults, an unacknowledged critical alert only re-pings every 4 hours. That is far too slow for on-call escalation, and it is why every serious tool implements its own escalation timer rather than relying on repeat_interval. oto should track “unacked for > N minutes” itself and escalate (thread reply with reply_broadcast, or a page) on its own clock, independent of Alertmanager.
Also from the docs: “If an alert is resolved before group_wait has elapsed, no notification will be sent” — so short-lived flapping alerts are invisible to us entirely. We cannot report on them, and should not claim to.
A11. Bonus: Grafana Alerting sends a compatible superset
Many teams run Grafana Unified Alerting instead of (or alongside) Alertmanager. Its webhook contact point emits the same version: "4" envelope with the standard fields (receiver, status, alerts, groupLabels, commonLabels, commonAnnotations, externalURL, version, groupKey, truncatedAlerts, per-alert fingerprint), plus Grafana-specific extras:
Extra field
Scope
Use
orgId
top-level
Grafana org — useful as a tenant discriminator
title, message
top-level
template-rendered strings
state
top-level
"alerting" or "ok" — note: different vocabulary from Alertmanager’s firing/resolved
silenceURL
per-alert
direct deep link to silence the rule — we can render this as a button with zero API work
dashboardURL
per-alert
present when the Dashboard UID annotation exists
panelURL
per-alert
present when the Panel ID annotation exists
imageURL
per-alert
screenshot of the triggering panel — a Block Kit image block straight into the card
UNVERIFIED: whether valueString / values are present in the current schema (they appear in older Grafana docs).
Design call: decode into a struct with the Alertmanager fields required and the Grafana fields optional. Costs almost nothing and roughly doubles the addressable install base. imageURL in particular is a large visual win that pure-Alertmanager users cannot get. Map state: "alerting"|"ok" onto our internal firing/resolved.
Its fields: type: "alert", text (plain_text or mrkdwn, max 200 chars), level ∈ default | info | warning | error | success, block_id.
It is genuinely useful for the “Confirm silence” or “Escalate” modal we open from a button, but it cannot be used in the channel alert card. Do not design around it.
Assessment of the newer block types for an alert card
“Alert blocks are currently only supported in modals.”text max 200 chars.
Card
⚠️ Too small
Fields: hero_image, icon/slack_icon, title (150), subtitle (150), body max 200 chars, subtext (200), actions max 3 buttons. “At least one of hero_image, title, actions, or body is required.” A 200-char body cannot hold an alert description. Possible fit for a compact multi-alert digest, not the main card. — card-block
Context actions
➖ Wrong purpose
Holds “feedback buttons and icon buttons”, max 5 items. Aimed at 👍/👎 feedback affordances, not alert actions. — context-actions-block
Table
✅ Viable for label dumps
Max 100 rows, 20 cells/row, 20 column settings, 10,000 chars across all cells. Docs describe apps that “programmatically publish messages that include a table.” Good for an expanded “all labels” or “all instances” view. — table-block
Data table / Data visualization / Carousel / Container / Plan / Task card
⚠️ UNVERIFIED surface support
The blocks index does not state per-surface availability, and several of these are clearly aimed at newer AI/agent surfaces. Verify each individually before use.
Recommendation: build the v1 card from the classic, universally-supported set — header, section (with fields and accessory), context, actions, divider, rich_text — wrapped in a color attachment. That combination is guaranteed to render everywhere including mobile and older clients. Treat table as a v2 enhancement for the expanded view.
The section-block docs also note: “block_id should be unique for each message and each iteration of a message. If a message is updated, use a new block_id.” — relevant to our update flow (we re-render cards on state change).
Status: legacy, but NOT deprecated, and still the only way to get a color bar.
Slack’s own page is now filed under /legacy/:
“This feature is a legacy part of messaging functionality for Slack apps. We recommend you stick with layout blocks.”
— Legacy secondary message attachments
And the migration guide warns content in attachments “may be wrapped, truncated, or hidden behind a ‘show more’ option by Slack clients” and that attachments “may change in the future in ways that reduce their visibility or utility.”
But — the decisive quote:
“The color parameter currently does not have a block alternative, so if you want to retain the color bar, you can use the blocks parameter within an attachment.”
— Attachments to blocks migration guide
✅ Recommended 2025/2026 pattern for severity color
color accepts good (green), warning (yellow), danger (red), or a hex like #439FE0.
Exactly one attachment wrapping all blocks. The legacy docs say “never ever (ever!) send more than 20 attachments”; chat.postMessage hard-caps at 100 (too_many_attachments).
No 2025/2026 sunset for attachments was found.UNVERIFIED: any announced removal date — none exists as of research date.
Alternative if you want to avoid attachments entirely: convey severity with a leading emoji in the header (🔴 / 🟠 / 🟡 / 🟢) plus a context line. This is what several tools do (see B8). The pragmatic answer is do both — emoji for scannability, color bar for the peripheral-vision cue.
A realistic alert card is ~10–15 blocks. The 50-block ceiling is the real constraint when a group contains many alerts. With ~3 blocks per alert instance, we can render ~12 instances before hitting the wall. Plan: render the top N (default 5–10), then a + N more button that opens a modal (100 blocks) or posts a thread reply. Also watch msg_blocks_too_long / invalid_blocks errors.
thread_ts: “Provide a thread_ts value for the posted message to act as a reply to a parent message.” Critically: “Avoid using a reply’s ts value; use its parent instead.”
reply_broadcast (bool, default false): with thread_ts, “indicates whether reply should be made visible to everyone in the channel or conversation.” Slack advises using it sparingly. It produces a message with subtype thread_broadcast.
Response: ok, channel (the channel ID actually posted to — may differ from what you passed if you passed a name), ts, and message (the full server-parsed message object).
ts is only unique per channel. The durable handle is the pair (channel_id, ts) — always store both. Use the channel value returned by the API, not the one you sent.
ts is a string like "1548261231.000200". Store it as a string, never as a float — the fractional part is a 6-digit sequence counter, and float round-tripping will corrupt it. This is the single most common Slack integration bug.
Distinguishing parent from reply: a message is a parent when thread_ts == ts; a reply when they differ. Presence of thread_ts at all means it is in a thread. — Retrieving messages
Store (alert_group_key → channel_id, parent_ts, last_rendered_hash) in our DB, keyed by our own durable group key (see A3), not Alertmanager’s groupKey.
Slack documents the delete side: when a message with threaded replies is deleted, “those replies will be left in the conversation, with a placeholder informing viewers about the deleted message.” And “a parent message object will retain a thread_ts value, even if all its replies have been deleted.” — Modifying messages
What chat.postMessage does with a thread_ts pointing at a deleted parent is UNVERIFIED — Slack does not document it, and notably thread_not_found is NOT in the chat.postMessage error table. Threading-related errors that are listed: cannot_reply_to_message, restricted_action_thread_locked.
Design call: treat any non-ok response to a threaded post as “parent is gone.” Clear the stored parent_ts, post a fresh top-level card, and store the new ts. Same for chat.update returning message_not_found.
Passing textwithoutblocksdeletes the previous blocks.
To keep blocks while updating, omit text.
Omit attachments to retain the previous attachments — but since our card lives inside an attachment (for the color bar), a state change from firing→resolved means resending the whole attachments array with the new color. The color bar can be updated.
No API-level time limit on editing is documented anywhere in the method reference or the Modifying Messages guide.
The error edit_window_closed exists and means “The message cannot be edited due to the team message edit settings” — i.e. a workspace admin setting, not a platform-wide TTL.
UNVERIFIED in the strict sense: whether bot messages are updatable literally indefinitely. The documented constraint is workspace edit settings, not message age.
Design call: we can update an alert card for the whole lifetime of an incident (hours/days). But handle edit_window_closed and message_not_found as terminal: fall back to posting a new message (optionally threaded under the old one) and re-point our stored ts.
“generally allows posting one message per second per channel”, plus a workspace-wide limit of “several hundred messages per minute” with generous burst behaviour
chat.update
Tier 3 (50+/min)
conversations.history
Tier 3 (50+/min)
limit default 100, max 999
conversations.replies
Tier 3 (50+/min)
conversations.list
Tier 2 (20+/min)
limit < 1000; Slack recommends ≤ 200/page
The binding constraint for oto is chat.postMessage: ~1 msg/sec/channel. An alert storm that fans out to one channel will queue. Design: a per-channel token-bucket at 1 msg/s, plus coalescing — prefer chat.update on an existing card (Tier 3, 50/min, and not per-channel-limited) over posting a new one. This is an independent, strong argument for the update-in-place card model.
Affected methods: ONLY conversations.history and conversations.replies.
New limits for affected apps:
1 request per minute (Tier 3 → effectively Tier 1)
limit default AND max both reduced to 15 objects
Who is affected:“commercially distributed apps that are not approved for the Slack Marketplace” — i.e. “unlisted” apps distributed outside the Marketplace.
Who is NOT affected — the key line for a self-hosted internal tool:
“Any internal customer-built apps will maintain their existing rate limits and will not be subject to the new posted limits.”
— clarification 2025-06-03
Marketplace-approved apps are also unaffected.
Dates: May 29, 2025 — new limits effective immediately for newly-created non-Marketplace apps and new installs of existing unlisted apps. June 30, 2025 — updated API ToS for apps created before May 29, 2025. A grace period for existing installations of unlisted apps was reportedly set to Sept 2, 2025 and later extended to Mar 3, 2026 — PARTIALLY VERIFIED; neither date appears in the current changelog page text (they come from FAQ search summaries). Both dates are now in the past, so assume enforcement is live.
Consequence for oto:
Deployed as an internal, single-workspace, non-distributed app → exempt. Keep Tier 3.
If oto is ever sold/distributed to customers outside the Marketplace, conversations.history/replies collapse to 1 req/min @ 15 objects.
Architectural mandate either way: never depend on reading Slack to reconstruct our own state. Our DB is the source of truth for (channel, ts, thread_ts, card state). Reading threads back must be, at most, a diagnostic nicety.
The payload arrives form-encoded as a single payload parameter — you must ParseForm() then JSON-decode payload, but you need the raw body first for signature verification (see below). Read the body once into a buffer.
The token field is deprecated — never use it for auth. Use signing secrets.
message.ts / container.message_ts is what you feed to chat.update.
actions[].value (2,000 chars) carries our per-alert state. Put an opaque ID, not a serialized blob — look it up in our DB. Never trust value as authoritative.
“your app must reply to the HTTP POST request with an HTTP 200 OK response. This must be sent within 3 seconds of receiving the payload.”
— Handling user interaction
In Go: verify signature → write 200 → hand off to a worker goroutine/queue. Everything else is async.
Related: trigger_id expires in 3 seconds and may only be used once — so views.open must happen synchronously in the handler, before the ack, or immediately after.
“bypasses channel posting permissions” — works even without channel scopes
needs chat:write in the channel
Ephemeral messages
only way to update them
cannot touch them
Payload options
response_type (in_channel / ephemeral), replace_original, delete_original (must be sole attribute), thread_ts
full message rewrite
Design call: use response_url with replace_original: true for the immediate optimistic UI response to a button click (fast, permission-free, within the 30-min window). Use chat.update for all subsequent long-lived state changes (a resolve arriving 6 hours later). Do not build the long-lived path on response_url — the 30-min/5-use cap will bite.
“we recommend using HTTP for production applications.”“we recommend using Socket Mode when developing your app and using it locally. Once deployed and published for use in a team setting, we recommend using HTTP request URLs.”
Socket Mode
HTTP Events API
Transport
WebSocket, bidirectional, realtime
Request/response
Public endpoint
Not required
Required (public HTTPS Request URL)
Concurrency
max 10 concurrent WebSocket connections per app
scales horizontally
Scaling
stateful → “more difficult to scale”
“scales well horizontally”
Reliability
long-lived connections inherently less reliable
short-lived connections “inherently more reliable”
Slack Marketplace
❌ “Apps using Socket Mode are not currently allowed in the public Slack Marketplace.” HTTP is “a requirement” for Marketplace submission
✅ required
Signature verification
not needed — “you’re receiving the events over a pre-authenticated WebSocket”
required
Ack mechanism
ack each event with its envelope_id over the socket
HTTP 200 within 3s
Firewall / on-prem
✅ ideal — “if you’re building an on-premise integration, have no ability to receive external HTTP requests”
❌ blocked by corporate firewalls
Socket Mode apps can still be distributed org-wide via enterprise deployment options — just not the public Marketplace. The 10-connection allowance is deliberate: graceful restarts, load balancing under high event throughput, temporary redundancy — which gives us zero-downtime deploys of a Go service.
HTTP retry behaviour (relevant to idempotency): Slack retries failed deliveries up to 3 times with exponential backoff — nearly immediate, then after 1 minute, then after 5 minutes — with x-slack-retry-num and x-slack-retry-reason headers. If >95% of deliveries fail in a 60-minute window (SSL errors, >3s responses, >2 redirects, non-2xx), Slack temporarily disables your event subscriptions and emails the app owner.
Support both; default to Socket Mode for self-hosted.
The overwhelming majority of self-hosted installs sit behind a firewall with no inbound HTTPS. Socket Mode is the difference between “works in 5 minutes” and “please ask your network team for an ingress rule + TLS cert.” That onboarding delta dominates.
Socket Mode also removes an entire class of bugs: no public endpoint to secure, no signature verification path, no Slack-side auto-disable when we deploy.
Offer HTTP mode as a config flag for installs that already have ingress, want horizontal scale beyond 10 connections, or intend to list on the Marketplace.
Build the interaction handler transport-agnostically — one func handleBlockActions(payload slack.InteractionCallback) error, with two thin adapters. slack-go makes this straightforward since both paths yield the same InteractionCallback type.
Note the ack semantics differ: Socket Mode requires acking by envelope_id; HTTP requires a 200 within 3s. Both are “ack fast, work async.”
Caveat on “3 open issues”: the repo runs actions/stale with days-before-stale: 45 / days-before-close: 10 for issues and PRs. The low count reflects aggressive auto-closing, not an absence of gaps.
Norberto Lopes created the library in 2015, stepped away for years, and returned to maintenance in January 2025 — explicitly because he joined incident.io, which uses the library, and found it struggling. Source: nlopes.dev.
v0.23.1 (2026-05-10) is a security patch: NewSecretsVerifier now rejects empty signing secrets“to avoid accepting forged request signatures when applications are misconfigured.” The release calls this an intentional breaking change “so that there are no footguns.” An earlier version with an accidentally-empty SLACK_SIGNING_SECRET env var would accept forged requests. — v0.23.1 release
Policy for oto: pin an exact version in go.mod, floor at v0.23.1, and read the changelog on every bump. Do not use @latest in CI.
NewSecretsVerifier(header, secret) + Write(body) + Ensure() in security.go. HMAC-SHA256 over v0:timestamp:body, constant-time hmac.Equal, rejects timestamps > 5 min old, returns ErrInvalidConfiguration on empty secret. Exactly implements the documented spec — do not hand-roll this.
Attachment has both Color string and Blocks Blocks (json:"blocks,omitempty") — the attachment-wrapping-blocks pattern is directly supported.
Reactions
AddReaction(name, ItemRef), RemoveReaction, GetReactions, ListReactions (+ Context variants) — useful for lightweight state signalling.
Rate limiting
OptionRetry() enables automatic retry for 429 only (“5xx and connection errors are not retried”); OptionRetryConfig() for broader strategies. 429s surface as *RateLimitedError{RetryAfter time.Duration} with Retryable() bool.
container block is missing — issue #1575 (2026-07-17). Slack shipped container on 2026-06-29: a collapsible wrapper with title/subtitle/icon, max 10 child blocks, width narrow|standard|wide|full, and default_collapsed. This is the one missing block genuinely relevant to alert cards — a collapsed “all labels” section would be ideal. Workaround: thread reply or modal.
#1571 (2026-07-10) — PostMessage/SendMessage discard the chat.postMessage response’s message object, so you cannot read back thread_ts and detect a silently un-threaded reply. Relevant to our deleted-parent handling (§B2): we cannot cheaply confirm the reply actually threaded.
#1577 (2026-07-25) — action_token parsed from assistant_thread but Slack sends it top-level; silent parse failure. Not relevant to us.
Bottom line: use slack-go/slack, pin ≥ v0.23.1, expect breaking changes on minor bumps, and budget for periodic dependency maintenance.
B8. How best-in-class tools build their Slack alert cards
Architecture: Block Kit blocks for content; attachments used purely as colored containers for state + buttons.
Layout:
section/mrkdwn — title
section/mrkdwn — body, truncated with "... Message has been trimmed. Check the whole content in Web"
If >1 alert in the group, a context block: :package: Showing the last alert only out of {alerts_count} total. Visit <{web_link}|the plugin page> to see them all.
Header format:
*<{grafana_oncall_link}|#{alert_group_id} {web_title}>* via {integration_name} (*<{source_link}|source>*)
Body template — the concrete field pattern to copy:
This is the best-tested palette in the open-source landscape. Adopt it. Severity goes in the emoji; state goes in the color bar. That split is the key insight — it means the color always answers “do I need to act?”, which is the question a human scanning a channel is actually asking.
Button state machine (single actions block, elements swap by state):
State
Buttons
Firing, unacked
Acknowledge, Resolve, Silence (static_select with duration options) / Unsilence, Responders, Attach to …, Add Resolution notes / Resolution notes [n], :fire: Declare incident (URL), :blue_book: Show Timeline
Acknowledged
:fire: Declare incident, Unacknowledge, Resolve, Silence/Unsilence, Responders, Attach to …, resolution notes, Show Timeline
Resolved
Unresolve, resolution notes, Show Timeline
Two details worth stealing:
The resolution-notes button is style: "primary" and its label carries a count — Add Resolution notes at 0, Resolution notes [3] otherwise.
Button value is a JSON blob {organization_id, alert_group_ppk, …} so the handler needs no ts → state lookup.
Silence is a static_select with duration options, not a fixed-duration button. Much better UX than our draft’s “Silence 1h”.
State text is appended as its own attachment above the buttons: "Resolved by @x" / "Acknowledged by @x".
Text: ..., Fields: nil/* TODO. Should be a config. */ }},
Fields is hard-coded nil — no fields grid at all. The docs confirm: “you cannot modify its visual appearance with custom blocks.”
Colors: firing #D63232, resolved #36a64f. Header: [FIRING:2] HighLatency checkout production — no emoji. Zero buttons. Silence/Dashboard/Panel/Source appear as bare URLs in the body text. Resolution posts a new message rather than editing.
Alertmanager’s own Slack integration — also legacy, also buttonless
send_resolved: false # note: FALSE by default for slack_config, unlike webhook_config
update_message: false
Block Kit support has been an open feature request since 2020 — alertmanager#2217, still open, 24 comments. This is the market gap oto fills.
Community best-practice templates converge on four link buttons — Runbook 📗, Query 🔍, Dashboard 📈, Silence 🔕 — with colors firing+critical=danger, firing+warning=warning, firing+other=#439FE0, resolved=good; severity emoji :fire: critical / :warning: warning. The Silence button is a deep link:
Cheap win: we can render a “Silence” URL button using that deep link with zero API calls and zero state — a great fallback when the user has not configured API credentials for us.
Sentry — pure blocks, no color bar, emoji-only severity
info :large_blue_circle: warning :large_yellow_circle:
ACTION_EMOJI = :white_circle: # replaces the level emoji once actioned
Key finding: Sentry issue alerts set no attachment color at all. Severity is conveyed purely by the leading emoji. Only metric alert incidents set a color. So the “no attachments” path is viable at the highest tier of the market.
Title is a section, NOT a header block::red_circle: *<link|TypeError: bar>*
Rationale: header is plain_text only — no bold, no links. Using a section buys the deep link back. This is a better choice than a header block for an alert card, and it is what Sentry, OnCall, and Alertmanager all do.
Fallback text is [project-slug] Title: description.
Context field labels: Events, Users Affected, State, First Seen, Approx. Start Time — rendered Key: *value* with three-space separators. Footer uses four-space separators: Project: <url|slug> Alert: <url|Rule Name> Short ID: BAR-1.
Noise-reduction trick worth copying: if State == "New" or event_count <= 1, Sentry deletes the Events and Users Affected fields entirely, and skips the backend query for the user count. Don’t render a count of 1. Generalize: suppress any field whose value carries no information.
Buttons — max 3, state-dependent: Unresolved → Resolve · Archive · Select Assignee… (an external_select, not a button); Resolved → Unresolve · Archive · assignee; Ignored → Resolve · Mark as Ongoing · assignee.
Updates: in place via replace_original: True on the response_url; the emoji flips to :white_circle: and an “Issue resolved by @user” block is appended. Errors go back ephemeral. Metric alerts thread: “a follow-up alert appearing as a threaded reply for future status changes.”
State routing: block_id carries JSON — {"issue": 123, "rule": 1}.
A complete literal Block Kit payload is published in Sentry’s test suite: build_test_message_blocks.
Cards “continuously update as the incident progresses” — in-place editing.
4 visible buttons: Acknowledge, Resolve, Join Dedicated Channel, Run a Workflow. 13 more behind a “More actions…” dropdown: Add a Note, Add Responder, View Details, Add Task, View Tasks, Assign Roles, Change Priority, Reassign, Escalate, Create Channel, New Status Update, Change Type, Run an Action.
Admin-tunable knobs that amount to a noise-control API — we should ship equivalents:
Thread incident updates — “Toggle on to add all updates to a thread and reduce clutter”
Verbosity presets: Highlights Only (default for incident channels) / Status Updates Only (default for notification channels) / No Updates / All Updates / Custom
Display emojis in incident notifications — “display emojis next to fields”, switchable off
Hide Create/Join Incident Channel Button
Channel naming: #inc-{{incident.incident_number}}
Fields include Priority (P1–P5) and Urgency (high/low). Exact color hex: UNVERIFIED — PagerDuty publishes no Block Kit JSON. Sources: Slack user guide, integration guide.
Opsgenie — clearest published color-bar convention
“Green means closed, red means open, and blue means acknowledged.”
— Atlassian docs
Colors update dynamically, and related alerts recolor too. Buttons: Acknowledge, Unacknowledge, Close, Snooze, Assign, Take Ownership, Add Note — and “buttons are changed accordingly” by state. Exact hex: UNVERIFIED.
Heavily Block Kit — “over 70 different Slack modals and almost as many messages” — but publishes no JSON.
Alert message actions: Declare incident / Join incident, Go to source, Silence, Resolve, Go to dashboard (docs).
Notable: there is no Acknowledge on alert messages. Ack belongs to escalations, and an “escalate to next level” button appears only after you have acknowledged. This is a deliberate separation of “the alert” from “the human response to the alert” — worth considering for our data model.
Messages are template-driven under Settings → Slack Messages: customizable Title (with dynamic status that auto-updates), Description toggle, and Attributes — “choose which alert attributes to display and in what order”, with per-attribute custom emoji (message customization).
With grouping enabled: “the first alert posts as a regular message and subsequent alerts turn it into a group message summarizing the number of alerts and their status” — edit-in-place with a count. Color bar / hex / default attribute list: UNVERIFIED.
Rootly — status emoji, no color bar, and a genuinely novel idea
Status emoji in the message, not a color bar: 🔴 Started, 🟠 Mitigated, 🟢 Resolved, 🔀 In triage (hex UNVERIFIED).
Steal this:“When the status or severity of an incident are updated, the previous state will be shown in strikethrough text (e.g. Started → Mitigated).”
— Rootly changelog
This solves a real problem with edit-in-place: a reader who saw the card an hour ago cannot tell what changed. Strikethrough-the-old-value gives you the history inside the current render, for free, with no extra blocks.
Update model is both: the announcement block auto-updates in place, and update blocks are posted into its thread. Workflow message options are explicitly Pin to Channel / Send as Ephemeral / Thread under parent / Update Parent Message / Broadcast Thread Reply.
Severity, Title, Started, Resolved, Opened By, Resolved By, Assigned Roles and People, links (channel / meeting bridge / ticket), buttons to Command Center and Internal Status Page — and “The message will also automatically update in place when these details update.”
— Runbook step: notify Slack channel
Custom messages support mrkdwn + template vars with a 3,000-character limit (matching the section limit). Signals alerts arrive as DMs with acknowledge / escalate / dismiss / open-incident buttons; escalations are added as threaded replies to the alert message. Also has emoji-reaction quick-declare: react with a configured emoji and the bot replies with a button to open an incident.
Screen readers read the top-level text field and NOT the content of interior blocks.
So either put everything a user needs into text, or omit text entirely when using blocks.
Combined with text also being the push notification, the sidebar preview, and the search result snippet, this makes text the highest-leverage string in the whole product. It deserves as much design attention as the blocks.
Messaging: “Ephemeral messages should only ever be sent in response to some user action, they must never be unexpected or unsolicited.” → never deliver an alert ephemerally; use ephemeral only for button errors/confirmations.
The filter test:“if the on-call engineer cannot take a specific action to resolve it, the alert should not exist.”
Teams receive “over 2,000 alerts weekly” with “around 3% needing immediate action.”
Benchmark: below 20% alert-to-actionable-incident conversion = noise problem; target 30–50%. — a metric oto could actually compute and surface.
Audit each alert: real/imminent user impact? runbook or clear action? ignored more than twice in 30 days?
“A database going down fires alerts for every dependent service within seconds. Without deduplication, your on-call engineer wakes up to 40 pages from one incident.”
P0 = “Production down, revenue impacted, or customer data at risk.”
Batch: instead of three “Server N restarted” messages → “3 servers restarted: server-1, server-2, server-3.”
Thread: parent per event, replies for updates — “The channel will see one message, and users who want details can expand the thread.” Use reply_broadcast: true for urgent updates. Start a fresh parent when a thread exceeds 20–30 replies.
“A great Slack message uses its header block the same way a great email uses its subject line.”
Button color: only default/primary/danger — “save green and red for clear confirm/accept or cancel/reject use cases.” Use emoji in default-style buttons to differentiate navigation actions.
Dividers separate items in digests; “if you’re just sending single block messages with text, dividers are probably overkill.”
B8c. Cross-tool synthesis — where the good ones converge
One leading emoji encodes severity, paired with text. Never color alone — Slack’s own accessibility guidance requires it, and colorblind users need it.
The color bar is optional and forces an attachment wrapper. Accept the legacy/truncation risk, or skip it and lead with emoji as Sentry does.
Color encodes STATE, not severity, in every tool that uses it well. OnCall’s palette is the best-tested.
Title as a section with a bold link, not a header block — headers are plain_text only, so a header costs you the deep link.
3–4 visible buttons, everything else in an overflow. PagerDuty: 4 visible / 13 hidden. Sentry: 3.
Buttons are state-dependent. Resolve↔Unresolve, Ack↔Unack, Silence↔Unsilence. Never render a no-op action.
Edit in place with chat.update for state changes; thread only for a stream of ongoing updates (notes, escalations, screenshots). Make it a per-channel toggle, as PagerDuty does.
Metadata in context blocks, Key: *value* with wide separators — not sections.
Suppress low-information fields (Sentry drops counts at ≤1).
Runbook is a first-class link/button, not buried in annotation text — exactly where Grafana Alerting and Alertmanager fail.
Encode routing state in block_id / value as JSON so handlers need no ts → state lookup. (Ours should be an opaque ID, not a payload — see §B5.)
Always set a meaningful top-level text.
Show a group count when collapsing: OnCall’s context line and incident.io’s group summary both show “N total” and render only the latest.
"text": ":rotating_light: [FIRING] HighErrorRate — 3 of 12 api instances in prod-eu above 5% errors. Severity critical, team payments, firing since 09:14 UTC. Runbook: https://runbooks.example.com/HighErrorRate",
"unfurl_links": false,
"unfurl_media": false,
"attachments": [
{
"color": "#a30200",
"fallback": "[FIRING] HighErrorRate on prod-eu",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": ":rotating_light: *<https://oto.example.com/g/01J9XQ2K7M3T|HighErrorRate>* · `prod-eu`\n_Error rate above 5% for 10m_"
}
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "3 of 12 `api` instances are returning >5% of requests as 5xx. Checkout and payment confirmation are affected."
The text field is a full sentence, not a slug. It carries alertname, scope, severity, team, start time, and the runbook URL — because it is the push notification and the screen-reader content (§B8b).
Every url button still fires an interaction payload you must ack:
“If you’re using url, you’ll still receive an interaction payload and will need to send an acknowledgement response.” — button element
Hence the oto.noop.runbook naming convention: the handler needs an explicit no-op branch returning 200 for oto.noop.*. Forgetting this produces “This app is not responding” toasts. Overflow options with a url behave the same way — oto.more must also ack even when the chosen option was a link.
style: "primary" is used on exactly one button, per Slack’s guidance that primary “should only be used for one button within a set” and danger“even more sparingly”. Destructive actions (Force resolve) live in the overflow behind a confirm dialog rather than being styled danger inline.
UNVERIFIED: whether {ago} composes with other tokens in one <!date^…> (docs show {date} at {time} composing; {ago} is only shown alone). The card above avoids the question by using {time} in the date tag and computing “Firing for 21m” server-side. Re-render on update to keep it fresh.
color → #dddddd, emoji → :mute:, Status → ":mute: ~Firing~ → Silenced by <@U…> until <!date^…>", and the Silence select flips to an Unsilence button carrying the Alertmanager silenceID.
Because Alertmanager stops delivering silenced alerts entirely (§A6), this state is only ever set by us — from our own silence action or from the API v2 reconciler polling status.silencedBy. There is no webhook that will tell us.
Update-vs-post policy, driven by notification_reason
chat.update the card and post a thread reply “+2 instances now firing”.
some alerts resolved
chat.update only — no thread noise.
all alerts resolved
chat.update to resolved state; optional thread reply with duration.
repeat interval elapsed
chat.update only. Never post a new message.
(empty — AM < 0.32.0)
Fall back to diffing the fingerprint set against stored state.
repeat interval elapsed → update-only is the single biggest noise reduction available to us, and it is the exact behaviour Grafana Alerting and stock Alertmanager get wrong (both repost). It is also the cheapest: chat.update is Tier 3 (50/min) while chat.postMessage is ~1/sec/channel.
Escalation for a genuinely unacknowledged critical: post a threaded reply with reply_broadcast: true rather than a new top-level card — it surfaces in-channel while keeping the thread as the single record. Slack advises using reply_broadcast“sparingly”, so gate it on severity + unacked duration. Note that Alertmanager’s own repeat_interval default of 4h (§A10) is far too slow for this — oto must run its own escalation clock.
Thread hygiene: per Knock’s guidance, start a fresh parent card when a thread exceeds 20–30 replies.
Configuration knobs to ship (modelled on PagerDuty)
