Skip to content
Scarab

Pipeline Syntax

Writing Scarab pipelines: YAML frontend, the typed IR, CEL expressions, invoke, and matrix.

The model, in brief

Section titled “The model, in brief”

A step is the fundamental unit of work: an OCI image plus a command (a contract) that Scarab runs as a single Kubernetes Pod. A pipeline is a group of steps, plus how they are triggered and wired together. Structurally a pipeline is a flat recursive DAG (Pipeline → Step).

The real DSL is a typed IR; YAML is a frontend and CEL provides expressions. invoke is reuse: inlining a library pipeline into the caller’s DAG at compile time. matrix is a modifier that fans a step out over a set of inputs.

Compilation is static: the whole DAG (every matrix combination, every inlined library step) is materialised and validated at submit time, so a cycle, a dangling needs, or an empty matrix dimension is rejected up front, never discovered mid-run.

A repo’s CI lives in .scarab/. A triggered pipeline is a YAML file such as .scarab/ci.yaml; its file name is the default display name (ci.yaml reads as ci, overridable with a top-level name:). Reusable library pipelines live under .scarab/lib/ and are pulled in with invoke. Start from Run locally to submit your first pipeline.

A minimal pipeline is an on: block and a list of steps:

on:
  push: {}
  pull_request: {}


steps:
  - id: checkout
    clone: {}


  - id: check
    image: rust:1-bookworm
    needs: [checkout]
    command: ["cargo", "check"]

The rest of this page documents each vocabulary term in turn, from triggers and interface, through the step shape, wiring, and reuse, to deploys and operational concerns.

Triggers: on:

Section titled “Triggers: on:”

on: maps a trigger kind to a filter. The kinds Scarab understands are push, pull_request, tag, release, comment, cron, manual, api, and upstream. An empty on: means the pipeline has no automatic triggers: it runs only when launched explicitly.

on:
  push: {}
  pull_request: {}

Each kind takes an optional when: CEL guard over the event context, so a trigger can fire selectively:

on:
  push:
    when: "event.branch == 'main'"

Manual dispatch is opt-in: a pipeline is dispatchable from the UI/API only if it declares on: manual. The trigger context also feeds each run’s provenance and Headline (the trigger title shown against the run).

on:
  manual: {}
  push: {}

Interface: interface.inputs (and outputs)

Section titled “Interface: interface.inputs (and outputs)”

A pipeline declares its typed launch parameters under interface.inputs. This is the same interface an invoke caller satisfies with with:, one contract for both launching and reuse. A bare string is shorthand for a required string; the map form is fully typed:

on:
  manual: {}


interface:
  inputs:
    - { name: region, type: choice, options: [us-east-1, eu-west-1], description: "target region" }
    - { name: replicas, type: number, required: false, default: 2 }
    - { name: force, type: boolean, required: false, default: false }


steps:
  - id: ship
    image: busybox
    command: ["deploy", "${{ inputs.region }}", "n=${{ inputs.replicas }}"]
  • type is one of string (default), boolean, number, choice. A choice needs a non-empty options list.
  • required defaults to true. An optional parameter (required: false) must carry a default, keeping the resolved parameter set total.
  • An optional validate: CEL predicate over the resolved value (bound as value) rejects a bad launch, failing closed.

Resolved parameters are read in steps as ${{ inputs.<name> }} and are also injected into the step environment as SCARAB_PARAM_<NAME>. A reusable library pipeline additionally declares the outputs it exposes to its caller (see Reuse: invoke).

Steps: the step shape

Section titled “Steps: the step shape”

The step contract is an OCI image plus a command; everything else is a modifier. env is a list of [NAME, VALUE] pairs; artifacts publishes globs of files the step wrote to /scarab/artifacts/. A step names the secrets it needs with secrets: [KEY, ...] (resolution and scoping is covered under Environments & gates).

- id: check
  image: rust:1-bookworm
  needs: [checkout]
  env:
    - [CARGO_TARGET_DIR, /tmp/scarab-target]
  command:
    - bash
    - -c
    - |
      set -euxo pipefail
      cargo check
      mkdir -p /scarab/artifacts
      rustc --version > /scarab/artifacts/toolchain.txt
  artifacts:
    - "*.txt"

Privilege above the hardened baseline is requested per step under security:. run_as_root is self-service (root inside the unprivileged sandbox, with its capabilities already dropped, does not escape it); add_capabilities and privileged are governed and take effect only if the environment whitelists the image digest.

- id: check
  image: rust:1-bookworm      # its USER is root
  security:
    run_as_root: true

Some steps are kinds with no image that the engine runs on your behalf (you set no image/command):

  • clone: {}: source provisioning (see The clone step).
  • build: { image: ..., push: true }: a rootless BuildKit image build.
  • gate: manual: a durable approval/wait point (see Environments & gates).
  • invoke: .scarab/lib/...: reuse at compile time (see Reuse: invoke).

The clone step

Section titled “The clone step”

Source is provisioned by a first-class clone step. It needs no configuration by design: the repo, ref, resolved SHA, and token all come from the run’s trigger context, so clone: {} is the common case. Downstream steps inherit the cloned workspace through ordinary needs.

- id: checkout
  clone: {}          # depth 1, current ref pinned to its SHA


- id: full
  clone:
    depth: full      # complete history (`1` is the shallow default)
    submodules: true
    lfs: true

Scarab is multi-forge: GitHub and Forgejo are both supported adapters; the clone step is identical across them.

Ordering: needs

Section titled “Ordering: needs”

needs: declares the DAG edges. A step runs once all of its needs succeed and inherits their workspaces:

- id: deploy
  image: deployer
  needs: [build]

Fan-out: matrix

Section titled “Fan-out: matrix”

matrix fans one authored step into the cartesian product of its dimensions, expanded statically at submit time. An exclude list of CEL predicates (evaluated against each combination’s own coordinate) drops combinations:

- id: test
  image: rust:1-bookworm
  matrix:
    dimensions:
      os: [linux, windows]
      arch: [amd64, arm64]
    exclude:
      - "os == 'windows' && arch == 'arm64'"

Each instance gets a deterministic id like test[arch=amd64,os=linux], and a downstream needs: [test] fans in over every surviving instance.

Conditions: when

Section titled “Conditions: when”

A when: CEL guard skips a step whose predicate is false; expressions read the run’s context (e.g. event, and any matrix coordinate):

- id: deploy
  image: deployer
  needs: [build]
  when: "event.branch == 'main'"

Reuse: invoke

Section titled “Reuse: invoke”

invoke inlines a library pipeline (under .scarab/lib/ by convention) into the caller’s DAG at compile time, not a runtime call. It is local only, forever: repo-relative paths, read at the caller’s ref; no ../ escape and no cross-repo references (cross-repo causation is on: upstream). The invoked steps are namespaced by the invoke step’s id (deploy/plan, deploy/url), and matrix on an invoke step fans out the whole subgraph.

A library declares its contract with interface (the inputs it requires and the outputs it exposes) and the caller supplies inputs with with::

.scarab/lib/deploy.yaml
interface:
  inputs:  [region, replicas]
  outputs: [url]
steps:
  - { id: plan, image: tf, command: ["plan"] }
  - { id: url,  image: tf, command: ["apply"], needs: [plan] }
.scarab/ci.yaml
steps:
  - id: deploy
    invoke: .scarab/lib/deploy.yaml
    with: { region: us-east-1, replicas: "3" }


  - id: notify
    image: busybox
    needs: [deploy]
    command: ["echo", "${{ outputs.deploy.url }}"]

The interface is validated at compile: every required input present, no unknown extras, and every referenced output declared. Supplied inputs reach each inlined step as SCARAB_PARAM_<NAME>.

Results & interpolation

Section titled “Results & interpolation”

A step publishes named, typed results that a downstream step reads at launch via ${{ outputs.<step-id>.<name> }}; launch parameters are read the same way as ${{ inputs.<name> }}. Interpolation is resolved at launch time against the upstream step’s captured results, so a consumer must needs: the producer.

Results are not artifacts. A result is a small typed value the DAG interpolates into a later step’s command or environment; an artifact is a file published from /scarab/artifacts/ that you download after the run.

Publishing a result

Section titled “Publishing a result”

A step publishes result <name> by writing the file /scarab/results/<name>.json — the result’s name is the filename minus .json, and the file must hold valid JSON: "abc123" (quotes included) for a string, 42 for a number — never bare text.

- id: version
  image: busybox
  command:
    - sh
    - -c
    - |
      set -eu
      mkdir -p /scarab/results
      printf '"%s"' "1.4.2" > /scarab/results/tag.json     # string result "tag"
      printf '%s' 42 > /scarab/results/build.json          # number result "build"


- id: notify
  image: busybox
  needs: [version]
  command: ["echo", "release ${{ outputs.version.tag }}"]

Capture is otherwise transparent: on Kubernetes a trusted per-Pod egress sidecar drains the step’s results back to the control plane — nothing to wire up in the pipeline. A build: step emits an image result natively (BuildKit writes --metadata-file /scarab/results/image.json); its author writes nothing.

The trap: a bare filename publishes nothing, and the step still goes green

Section titled “The trap: a bare filename publishes nothing, and the step still goes green”

The sidecar drains *.json files only. Writing a bare /scarab/results/tag — the obvious guess — publishes nothing: the glob matches no file, the sidecar logs only no results to drain, and the step still exits 0. The run goes green while ${{ outputs.version.tag }} silently resolves to empty downstream. Nothing tells the author they produced no result.

So make the consumer assert that its interpolation resolved — that is what turns the silent no-op into a failure at the point of use:

- id: notify
  image: busybox
  needs: [version]
  command:
    - sh
    - -c
    - |
      TAG='${{ outputs.version.tag }}'
      test -n "$TAG"    # a producer-side typo fails HERE instead of shipping ""
      echo "release $TAG"

Environments & gates

Section titled “Environments & gates”

This section covers only the YAML you author. For the concepts behind them (how secrets are stored and scoped, and what an environment governs), see Secrets and Environments.

Referencing a top-level environment: makes the pipeline a deploy: its runs are enforced against that environment’s protection rules at admission. Secrets a step needs are named with secrets: [KEY, ...]; each key is resolved against the run’s scope and made available to the step under the same name (a run from a fork’s pull request is locked out and receives none).

environment: prod


steps:
  - id: ship
    image: deployer
    secrets: [DEPLOY_TOKEN]
    command: ["./deploy.sh"]

Gates are durable suspend points with no image: gate: manual is an approval, gate: timer waits gate_after seconds, gate: external releases via API/webhook. Placed before a deploy step, a manual gate is the approval mechanism for a governed environment:

- id: approve
  gate: manual
  needs: [build]


- id: release
  image: deployer
  needs: [approve]

Placement & resources

Section titled “Placement & resources”

By default steps land on the operator’s default placement profile. An author targets scheduling by naming profiles (never raw Kubernetes) via placement_profiles, and requests exact compute with resources:

- id: heavy
  image: builder
  placement_profiles: [arm64, critical]
  resources: { cpu_millis: 8000, memory_mib: 16384 }

For a rare dynamic per-job Kubernetes need there is a governed k8s_overlay escape hatch: a raw pod-spec fragment merged last onto the Pod. It carries no authority: it is admitted only if the run’s target environment permits raw overlays, else the run is rejected fail-closed. Prefer a named profile for anything static.

- id: heavy
  image: builder
  k8s_overlay:
    spec:
      schedulerName: my-scheduler

Retries, timeouts & reruns

Section titled “Retries, timeouts & reruns”

A step may set a timeout: (seconds) and an opt-in retry: policy. Retry is at-least-once: declaring it is your assertion that the step is idempotent or fenced; a side effect that does not cooperate will fire twice.

- id: flaky
  image: tester
  timeout: 600
  retry: { on: failure, max: 3 }

Re-execution that a human initiates is modelled as Takes: a rerun re-executes a run from the top, a retry resumes from a failed step, and an earlier attempt becomes superseded or shadowed by the newer one. These are concepts from a run’s history surfaced in the UI. There is nothing to author for them.

References

Section titled “References”
Get StartedUsageReferencesDemo

Get Started

OverviewInstallationRun locallyLocal Development

Usage

Pipeline SyntaxSecretsEnvironmentsEnvironment Variables

References

APIThesis & language (CONTEXT)