Remedies

A Remedy is a change to a cluster that an Investigator proposes with its Finding, and that oto executes through one of your write Tools only after one or two different people approve it — two unless your risk rules say one (ADR 0054). It is the one place oto writes to a cluster, and this page is what you need to set one up, to know what each part does, and to know where oto’s control ends and your ToolServer’s begins.

proposed ──► approved ──► executed | failed
    │            │
    ├──► declined ◄┘
    └──► expired ◄─┘

Every transition is made by a named actor — the Investigator proposes, a person approves or declines, oto (system) expires — and every transition is sent as a fact to the Incident it is declared to (none, for a Case no Incident holds — see What goes outbound).

What you configure

1. A write ToolServer. oto ships no ToolServer: a Remedy runs through an MCP server you run, configured in oto with access: write. Only a write ToolServer’s Tools can carry out a Remedy, and no Investigator ever holds one — an allowlist naming a write Tool is refused, and a run is never offered one. Discover it once, so oto knows the Tools it lists.

2. Who may approve. Approval is a grant held on that ToolServer, given and taken only from the host shell:

   oto grant remedy-approver --org <slug> --toolserver <name> --email <address>
   oto revoke remedy-approver --org <slug> --toolserver <name> --email <address>

   No route in oto can create, change or delete a grant — an in-app grant would let one holder mint a second approver and approve alone. GET /api/v1/tool-servers/{id}/remedy-approvers lists the holders, read-only.

   On the Helm chart, “the host shell” is the oto pod. The image’s entrypoint is the oto binary itself, so run the subcommand in the API Deployment (<release>-oto-api, or <release>-api when the release name already contains oto), which already holds the database credentials:

   kubectl -n <namespace> exec deploy/<release>-oto-api -- \
     oto grant remedy-approver --org <slug> --toolserver <name> --email <address>

   oto remedy-rules apply runs the same way, with the file on standard input (kubectl -n <namespace> exec -i deploy/<release>-oto-api -- oto remedy-rules apply --org <slug> -f - < rules.yaml). ⚠️ Whoever may exec into the oto pod is the authority that mints approvers and writes the risk rules — the pod holds the database credentials. Restrict pods/exec on that namespace to the people you would trust with that.

3. An Investigator that may propose. Add oto_propose_remedy to its Tool allowlist, and oto_write_tools so it can read which write Tools exist and what arguments they take. Both are built in; neither calls a ToolServer.

4. How long a Remedy waits. remedy_approval_window_s (Settings → Tuning → Remedies), 60 to 86400 seconds, default 3600 — see tuning.

5. Optionally, risk rules that let a harmless command need only one approval, and a risk model that may ask for two, changed in Settings → Remedy risk (one member proposes, a different member confirms) or applied from the host shell with oto remedy-rules apply — see Risk rules below. With none, every Remedy needs two.

What a Remedy says

The exact command first. A Remedy names the write Tool that would carry it out — <toolserver>__<tool> — and the exact arguments it would be sent, byte for byte, with their SHA-256. Directly under them: how many approvals it needs and what set that — the rule by name, no rule matched, the rules could not parse this command, or raised by the risk model. Then what it is made to (target), and only then the Investigator’s own description of the change. Read the command; the description is a model’s words.

Or it says that no configured Tool can carry it out. The Investigator proposes the change it judges right whether or not one of your Tools can make it. A Remedy that names no Tool says “no configured Tool can carry this out”, can never be approved (the API refuses it with 409 remedy_has_no_tool, and the screen offers no approve control), and can still be declined; it expires like any other. Its value is the Finding it stands on: make the change by hand, or configure a Tool and ask for another Investigation.

Approving

• One or two approvals, from different people, each holding the grant on the Remedy’s ToolServer. How many is set when the Remedy is proposed, by your risk rules, and never changes after. The same person approving twice — in the UI and again from somewhere else — counts once (409 remedy_already_approved).
• You approve the arguments you were shown. The approval names their SHA-256; if it is not the Remedy’s, it is refused (409 remedy_arguments_changed).
• The Tool is checked again, against your configuration as it is now. If the ToolServer was removed, re-declared read, or no longer lists the Tool, the Remedy cannot be approved (409 remedy_tool_unavailable) and says why — it is never executed against nothing.
• A person without a counting grant is refused (403 remedy_approver_required).
• Approve from a browser session, or from Slack — never with a token. A personal access token’s approval is refused (403 remedy_approval_needs_a_session) before the Remedy is read: an approval makes a command run, and a leaked or scripted token must not make one count. A Slack press is a signed click by a linked person and is unaffected.
• Before you decide, see what went outbound. Above Approve and Decline, the approval screen lists the last three facts sent outbound about the Remedy’s Incident — what each said, when, and whether it landed. A Remedy on a Case no Incident holds has none, and shows no list.
• Decline is any member’s, from the UI, a token or Slack: saying no is the safe direction, and a declined Remedy may be proposed again by a later Investigation. A Remedy that is proposed, or approved and not yet being executed, can be declined.

When the last approval it needs lands the Remedy is approved, and its window starts again: it must be executed within remedy_approval_window_s of its approval.

From Slack

When the Incident is a Slack conversation and a notification policy routing it names remedy_proposed (see What goes outbound), a proposed Remedy’s reply in its thread says the exact command first, then its target, how many approvals it needs and what set that, who has approved so far, and only then the Investigator’s description — with Approve and Decline under it.

• A press is applied as the oto user your Slack account is linked to, through the same approval and decline the UI makes: the grant, the window, the Tool check and the count of different people are the same, and approving in Slack and again in the UI counts once.
• An unlinked Slack account is refused, and nothing is recorded. The reply — seen only by you — carries a one-time link code: sign in to oto, open Account from your menu, enter it, check the Slack account and workspace it names, and confirm. From then on your Slack clicks count as you. See Linking your Slack account below. A linked account without the grant gets remedy_approver_required’s answer.
• Approve asks for confirmation first. It is not offered for a Remedy no configured Tool can carry out, nor for one whose arguments the reply had to cut — measured after Slack’s escaping, so arguments heavy in &, <, > or non-ASCII text are cut sooner than their length suggests; the reply then says so and still shows the target and the tier. Approve those in oto, where the arguments are shown whole. Decline is always offered.
• The facts it was proposed among are above it. The reply sits in the Incident’s thread, under the facts already sent there — which is what the approval screen’s list shows in the UI.
• The buttons stay on the reply after the Remedy moves on; a late press is answered with why it no longer applies. Each transition is its own reply, as it is to every other destination.

Linking your Slack account

A link decides whose approval a Slack click counts as, and double approval counts different oto users — so linking is something each person does for themselves, from a place only they can reach on each side:

1. Press Acknowledge or Un-acknowledge on any card, or a Remedy’s Approve or Decline, in Slack while unlinked. oto verifies Slack’s signature on the press and answers, in a message only you can see, with a code such as ABCDE-FGHJK. (An acknowledgement is still recorded, as your Slack account; a Remedy press records nothing.) It is bound to your Slack member and workspace, works once, and dies after ten minutes — or as soon as you press again and get a newer one.
2. Signed in to oto in your browser, open Account from the menu under your initials and enter the code. oto shows the Slack account and workspace it would link — “Clicks from this Slack account will count as you.” — and links nothing until you confirm.
3. Unlink from the same page at any time. Every link and unlink is recorded with who did it.

⚠️ A link code is a credential. Whoever enters your code in their own oto session makes your Slack clicks count as theirs — including approving a Remedy. Never paste it anywhere but oto’s Account page, never send it to anybody, and if the confirmation names a Slack account that is not yours, cancel. The confirmation screen, the ten-minute single-use life and the recorded fact of every link are the mitigation; nothing else is.

The rules oto keeps whatever you do:

• Only the signed-in person is ever linked. No route takes a user id, and linking needs a browser session — a personal access token cannot link, unlink or preview.
• A link to another active person is never moved. A code for a Slack account already linked to somebody else is refused (409 slack_identity_linked_elsewhere); they unlink it first. A link to the stand-in oto created when you first pressed a button is replaced, and the stand-in retired.
• A link to a disabled person moves to your own code. A disabled user cannot sign in to unlink, so a Slack account linked to one is issued a code like an unlinked one, and confirming it links you; the record of the link names the disabled user it displaced.
• Wrong codes are limited. A wrong, used or expired code is one answer (422 slack_link_code_invalid); five in fifteen minutes and further attempts are refused with 429 for a while. One code may be checked or confirmed five times at most, then it is dead.
• oto stores only a hash of the code.

Execution

When the last approval it needs lands, oto enqueues the Remedy’s execution in the same transaction. The executor then:

1. Checks everything again, against now. An approved Remedy past its window is recorded expired. If the ToolServer was removed or re-declared read, or no longer lists the Tool, it is failed with tool_unavailable. If its arguments no longer hash to what was approved, arguments_changed. If fewer different approvers of these arguments than it needs still hold the grant — one was revoked, or disabled — approvals_withdrawn. In every one of these nothing is sent, and the Remedy says so.
2. Claims it. The Remedy moves approved → executing, and that is committed before the write Tool is called.
3. Calls your write Tool once, through the MCP client, with the stored arguments — the exact bytes the approvers approved by hash, never decoded and re-encoded — inside the ToolServer’s own per-call timeout.
4. Records what came back. executed when the Tool did not report a failure; failed with tool_error when it did. Either way the answer is kept with your redaction rules applied, the ToolServer’s token scrubbed, and capped at 16 KiB — on the Remedy, not on the Incident fact.

At most once. Because the claim is committed before the call, a retried or duplicated job finds the Remedy executing (or ended) and sends nothing. A worker that dies mid-call leaves it executing; once seven minutes have passed with no answer, remedies.sweep records it failed with outcome_unknown — the change may or may not have been made, and the record says exactly that. A broken connection or a timeout during the call is outcome_unknown for the same reason. A ToolServer that could not be reached at all is tool_unavailable: no session, so no call.

A failed Remedy is never retried. failed is final. If the change still matters, ask for another Investigation: a retry is a new Remedy and a new approval.

An executed Remedy is followed up. Right after the record of it executed commits, oto asks, in a transaction of its own, for one Investigation of its Incident — the one its facts are declared to — so the timeline says whether the change helped. A Remedy on a Case that no Incident holds is followed up on that Case.

• The Investigator that proposed it runs it, whether or not it is opted into Incidents: it chose the command, and its earlier Finding is the “before” the new one is read against.
• Every control holds. A switched-off Investigator or org records nothing; a spent daily budget records the follow-up skipped with reason budget; it waits for the concurrency; and it coalesces under the Investigator’s minimum interval like a membership change — into a run of the same Investigator on the same subject that has not started yet, or after the interval since the last one began. Several Remedies executed close together are one follow-up.
• A failed Remedy is followed by nothing, and a retried execution job never asks twice. A follow-up that cannot be asked for is logged, and never costs the record of what the Tool answered.
• Its Finding goes outbound like any other Incident Finding (finding); it never decides whether anyone is notified.

Risk rules

How many approvals a Remedy needs is set once, when it is proposed, from rules you change in Settings → Remedy risk or write in a YAML file and apply from the host shell with oto remedy-rules apply — see Changing the rules. oto ships no rule: with none, every Remedy needs two. GET /api/v1/remedy-risk-rules reads them, and the change waiting for a second person.

What a rule says

A rule has a name (lower-case letters, digits, _, -; it is what the approval screen shows), 1 or 2 approvals, and one or more conditions, every one of which must hold. A rule that says 1 must name its tool — so you say which of your write Tools take kubectl-shaped arguments; a 1-rule without one is refused (single_needs_tool). A rule that says 2 may omit it.

Condition	Holds when
tool	the Remedy’s write Tool is exactly this <toolserver>__<tool>.
verbs	the command’s verb is one of these — delete, scale, rollout restart, set image.
kinds	the command’s resource kind is one of these. kubectl’s plurals, short names and group-qualified names for the built-in kinds fold onto one name (deploy, deployments, deployment.apps are all deployment). A kind oto does not know every spelling of — a CRD’s, say — is refused in a rule (unknown_kind), and a command naming one is unparseable.
namespaces	the command names one of these namespaces. A command that names no namespace, or names all of them (-A), matches no namespace condition.
reversibility	reversible: the verb is one oto knows to be reversible — rollout restart, rollout pause, rollout resume, rollout undo, scale, cordon, uncordon — read from a command line. irreversible: any other verb, including one oto does not know, a command with no verb, and every structured Remedy (its verb member is the model’s claim, not what the Tool does).

A rule with no condition is refused: it would match every command.

How the rules decide

1. A command the rules cannot parse needs two, whatever the rules say — see below. No rule and no model is asked.
2. Otherwise every rule is checked. The most severe matching rule wins: if any matching rule says 2, the Remedy needs two, named after the first such rule in your order; only if every matching rule says 1 does it need one, named after the first of those.
3. No rule matches: two.

Order therefore only decides which rule is named, never the tier. This is deliberate: under “first match wins”, a broad rule placed above a narrow one would silently lower what the narrow one guards — anything in staging → 1 above delete secret → 2 would let a secret be deleted on one approval. Here a rule that says 2 cannot be outvoted, and a rule that says 1 lowers only what no 2-rule matches. Write broad 1-rules and carve exceptions out of them with 2-rules.

Example:

rules:
  - name: restart-payments
    tool: k8s-write__kubectl
    verbs: [rollout restart]
    kinds: [deployment]
    namespaces: [payments]
    approvals: 1
  - name: secrets-need-two
    verbs: [delete]
    kinds: [secret]
    approvals: 2

kubectl rollout restart deployment/api -n payments, sent to k8s-write__kubectl, needs one approval (restart-payments); the same restart in checkout matches nothing and needs two; kubectl delete secret db -n payments needs two (secrets-need-two).

What the rules read: the command

A Remedy is a write Tool and its JSON arguments. The rules read them in one of two shapes:

• A command line — the arguments are exactly one member, command (a string, or an array of words) or args (an array of words), with nothing beside it. It is read as kubectl’s command line: kubectl first, or one of kubectl’s own verbs first. A string is split on spaces and tabs only, and nothing is interpreted — no quoting, escaping, expansion or globbing. Every word must be ASCII letters, digits and _ . : / = @ , + -. The verb is kubectl’s (two words for rollout, set, auth, certificate, top); the kind is the next word, or the type in type/name; cordon, uncordon and drain name a node; the namespace is -n/--namespace, a namespace name.
• Structured arguments — no command line, and only these members, each a JSON string: verb, kind (or resource, or resourceType), namespace and name. A Tool whose arguments carry none of these can still be matched by a rule on its tool.

Unparseable — two approvals, whatever the rules say:

• a word with any character outside that allowlist — so sh -c …, pipes, ;, &&, backticks, $(…), redirects, quotes, globs, and any other whitespace (a no-break space, an em space, a vertical tab, …), which is never split on;
• ; | & $ < > ` or a newline in any string anywhere in the arguments, keys included;
• a key twice at any depth, two keys that differ only in case, or a key that is not ASCII;
• a member beside command/args, or, in structured arguments, any member but the four above, or a value that is not a string (a manifest, a patch, a number, a nested object);
• a program other than kubectl (sh, bash, helm, …);
• -- and whatever follows it (kubectl exec … -- …), or - (standard input);
• a flag that names resources in a file (-f, -k, --raw), changes who kubectl runs as (--as, --token, --kubeconfig, …) or where it connects (--server, --context, --cluster, --tls-server-name), or --profile, --cache-dir;
• a flag oto does not know, in any form, --flag=value included;
• a kind oto does not know every spelling of;
• two kinds (secret,configmap, or deploy/a secret/b), or two different namespaces.

The Remedy records which of these it was, and the approval screen shows it.

The risk model

You may name one of your model endpoints as the risk model. When — and only when — the rules say one approval, oto asks it, once, whether the change should need two. It may raise one to two; it can never lower anything, and a Remedy the rules said needs two is not asked about.

• It sees only the command — the write Tool, its exact arguments, what kubectl’s verb, kind and namespace were read as — its target, and the rules’ verdict (the tier and the rule). Never the Investigation, its Steps, its Finding, the Investigator’s description, a log line or a Tool’s answer: logs are attacker-writable, and a log line is the obvious injection path.
• It fails closed. An error, a timeout (30 s), an answer without token usage, or an answer that is not 1 or 2 leaves the Remedy at two, recorded as the risk model failed with why.
• With no risk model named, the rules’ tier stands, and the Remedy records that none was asked.
• Its tokens are the org’s. What it said, which endpoint and model, and the tokens it cost are kept on the Remedy, and those tokens count against the org’s daily token budget (investigation_daily_tokens, Investigators) beside every Investigation’s.
• A spent budget is two. When the day’s budget is already spent, the risk model is not asked, and a Remedy the rules said needs one needs two, recorded as the day’s token budget was spent (approvals_set_by: risk_model_budget). The check fails closed: a question nobody paid for never lets one approval stand. The budget resets at 00:00 UTC.

Changing the rules

There are two ways in, and neither lets one person loosen the rules alone.

From Settings → Remedy risk: one member proposes, a different member confirms. Propose a change edits the whole rule set and the risk model, and saves it as a pending change. Saving changes no Remedy’s approvals. It takes effect only when a member other than the proposer presses Confirm, and the screen shows that member what they are agreeing to, including every rule that says one approval. The proposer sees Confirm disabled, with the reason; they can discard the change or amend it, and a newer proposal supersedes the pending one. Both steps need a signed-in browser session — a personal access token cannot propose or confirm, so a script holding two members’ tokens is not two people — while discard is open to a token, because saying no is the safe direction. The database refuses an applied change whose confirmer is its proposer, so this holds even if the application were wrong.

From the host shell, the same place an approval grant is given (oto grant remedy-approver):

oto remedy-rules apply --org acme -f rules.yaml     # replace the rules and the risk model
oto remedy-rules show  --org acme > rules.yaml      # print what stands, as a file that applies

An apply from the shell is the later word: it supersedes a pending change, so a stale proposal cannot be confirmed over it.

Why two people. A rule that says 1 lets one grant holder approve alone. If any member could write the rules, a grant holder could write a one-approval rule and then approve their own way through — the loophole the grant itself was moved out of the app to close, and that a first version of the rules API reopened. Rules only ever loosen (oto ships none, and with none every Remedy needs two), so no change is exempt: writing one has the authority of granting a second approver, and therefore takes a second person. GET /api/v1/remedy-risk-rules reads what stands and the pending change; the routes under /changes propose, confirm and discard.

The file:

# Optional: one of the org's model endpoints, by name, asked whether a Remedy the
# rules say needs one approval should need two. Omit it for none.
risk_model: risk-checker

# Required. The whole list, in order; `rules: []` makes every Remedy need two.
rules:
  - name: restart-payments          # required; lower-case, digits, _ and -; unique
    tool: k8s-write__kubectl        # required when approvals is 1: <toolserver>__<tool>
    verbs: [rollout restart]        # optional conditions — at least one is required
    kinds: [deployment]
    namespaces: [payments]
    approvals: 1                    # required; 1 or 2
  - name: secrets-need-two
    tool: k8s-write__kubectl        # optional when approvals is 2
    kinds: [secret]
    reversibility: irreversible     # reversible | irreversible; omit for either
    approvals: 2

• Applied whole, in one transaction. The file replaces every rule and the risk model; a rule you leave out is gone. -f - reads the file from standard input.
• A malformed file changes nothing and exits non-zero, naming the problem and where it is — YAML that does not parse, a key oto does not know (namespace: for namespaces: would silently drop a condition and broaden the rule, so it is refused), a missing rules:, a rule that breaks one of the rules above (rules/0/namespaces/0: …), a risk_model that is not one of the org’s endpoints, or an unknown org.
• On success it prints each rule with its tier, the risk model, and how many rules it replaced.
• The last apply and when are recorded and shown on the Settings screen.
• ⚠️ Treat a change to the rules like a change to who may approve. Applying them re-tiers no Remedy already proposed: decline any pending Remedy the new rules would have tiered higher.

The trust boundary is your ToolServer

oto holds no cluster credential. It holds your write ToolServer’s access token, sealed, and it sends that ToolServer the approved arguments. A ToolServer that accepts oto’s credential will run whatever oto sends it — oto ships no ToolServer and does not inspect yours — so oto’s approval (one or two different holders of the grant, as your risk rules say) is the only gate oto owns. Everything past that is what you granted the ToolServer.

So:

• Give a write ToolServer narrower RBAC than a read one. Run it under its own ServiceAccount, separate from your read ToolServer’s, with exactly the verbs, resources and namespaces the Remedies you want to allow need — patch on deployments in the namespaces you name, say, not * on *. The read ToolServer may see more; the write one should be able to change less.
• Expose only the Tools you mean. Every Tool a write ToolServer lists is one a Remedy may name. A Tool that runs an arbitrary command or shell is an arbitrary change, approved by two people reading it.
• Keep its token for oto alone, and rotate it like any credential that can change a cluster.
• Grant approval narrowly. oto grant remedy-approver per ToolServer, to the people who should be able to change what that ToolServer can change.

Expiry

A proposed Remedy that has not had its approvals within the window, or an approved one oto has not executed within the window after its approval, expires. It reads as expired the moment the window passes — nothing can approve, decline or execute it from then — and the remedies.sweep job records the transition (by system) within a minute and declares it. Recorded, never silent.

What goes outbound

Each transition is an Incident fact — remedy_proposed, remedy_approved, remedy_declined, remedy_expired, remedy_executed, remedy_failed — sent to the Incident the Remedy is about, or to the Incident holding its Case at the time. A Remedy on a Case that is in no Incident is declared nowhere: it has no outbound fact, so no Slack card and no list of facts on its approval screen; it is decided on the Case’s Investigation panel, and its transitions are on its record. Route them with a notification policy like any Incident fact — a policy must name remedy_proposed, or no Slack card with Approve and Decline is posted; the envelope’s incident.remedy carries the transition, the exact command first, with required_approvals and what set it (approvals_set_by, approvals_rule) (see the webhook envelope). They are facts: approval happens in oto, and oto reads nothing back from your incident tool.