Dynamic CI Buildkite Plugin

Skips the steps your diff does not need, using your organization’s own CI history. Trunk decides at pipeline-upload time, so a skipped step never acquires an agent.

Usage

Pipe your pipeline through trunk-dynamic-ci-filter on its way to buildkite-agent pipeline upload. That works whether your pipeline is a file or generated by a script:

steps:
  # a static file
  - label: ":pipeline: upload"
    secrets:
      - TRUNK_TOKEN
    command: |
      cat .buildkite/pipeline.yml \
        | trunk-dynamic-ci-filter \
        | buildkite-agent pipeline upload
    plugins:
      - trunk-io/dynamic-ci#v<version>: ~

steps:
  # generated by a script — identical, and it does less work: a pipeline that is
  # already JSON is never re-rendered
  - key: upload-pipeline
    label: ":pipeline: build + upload"
    secrets:
      - TRUNK_TOKEN
    command: |
      ./ci/emit-pipeline.sh \
        | trunk-dynamic-ci-filter \
        | buildkite-agent pipeline upload
    plugins:
      - trunk-io/dynamic-ci#v<version>: ~

Throughout this README, v<version> stands for the release you are pinning — replace it with an exact tag from Releases, for example #v0.1.0. See Versioning for why the pin should be exact.

The plugin contributes one thing: the trunk-dynamic-ci-filter command, put on your PATH for the duration of the step. It reads a pipeline on stdin — JSON or YAML — and writes one on stdout. On any failure it writes back exactly what it was given, byte for byte.

Do not add --no-interpolation to your pipeline upload. For a YAML pipeline the filter renders without interpolation, so your upload performs the single pass. If it does not, nothing interpolates and $$VAR reaches the shell, which reads $$ as its own process id — $$FOO becomes something like 206FOO. Not an error and not a blank: a plausible string that changes every run.

The plugin says so when there is something to lose: it stays quiet if your pipeline has nothing to interpolate, and if your step’s command is visible it speaks up only when that command really does pass the flag.

Options

Option	Default
mode	filter	filter (default) puts trunk-dynamic-ci-filter on PATH for your pipe; step decides about one individual step.
token-env	TRUNK_TOKEN	Name of the environment variable holding your Trunk organization API token. Expose it to the step (secrets:, or your agent’s environment) and name it here — never put the token in plugin configuration or an env: value.
only-keys	unset	Comma-separated step keys to consider. Every other step runs untouched. Works in both modes.
exclude-keys	unset	Comma-separated step keys to never skip. Trunk still runs them; it just never decides about them. Applied after only-keys, so a key in both is excluded.
debug	false	Print the requested step keys, the request body and the plan to the build log, in collapsed groups. All of it goes to stderr, so it is safe in filter mode.
ignore-signals	unset	Comma-separated signal identifiers to exclude from the recommendation.
commit-sha	unset	The commit to report instead of BUILDKITE_COMMIT, such as the pull request’s head when your checkout builds a merge commit.
commit-sha-env	unset	Name of the environment variable holding that commit, read when the job runs. Use it when the pipeline is not generated per build. commit-sha wins when both are set.
base-sha	unset	The commit to diff against instead of the merge base with the pull request’s base branch. Compared to the checked-out commit directly.
base-sha-env	unset	Name of the environment variable holding that commit, read when the job runs. base-sha wins when both are set.

The token must reach the step without passing through the pipeline definition. Two things are interpolated into the uploaded pipeline at upload time and shown in the Buildkite UI: plugin configuration, and env: values. So put the token in neither. Expose it to the step — Buildkite secrets:, or your agent’s own environment — and point token-env at the variable’s name.

Every keyed step is a candidate

Worth being explicit, because it differs from how the GitHub Action works: once the plugin is on your upload step, every step with a key: is something Trunk may decide to skip. There is no per-step opt-in to wire up — which is convenient, and also means a step you never want skipped needs saying so.

Two ways to say it:

plugins:
  - trunk-io/dynamic-ci#v<version>:
      exclude-keys: deploy-master,publish-release

exclude-keys keeps a step out of Trunk’s reach entirely — it still runs, Trunk just never decides about it. Good for deploys, release gates, and anything whose cost of being wrongly skipped is high.

For a standing policy rather than pipeline configuration, set the job to Never skip on the repository’s Dynamic CI configuration in the Trunk web app. Same effect, kept with the rest of your Dynamic CI settings rather than in your pipeline, and it survives someone editing the YAML.

Trialling it on one step

Narrow the plugin to the steps you name, and leave the rest of the pipeline untouched:

- label: ":pipeline: upload"
  secrets:
    - TRUNK_TOKEN
  command: |
    cat .buildkite/pipeline.yml \
      | trunk-dynamic-ci-filter \
      | buildkite-agent pipeline upload
  plugins:
    - trunk-io/dynamic-ci#v<version>:
        only-keys: e2e

This is the way to start small. The named step is marked before it is dispatched, so it reports as genuinely skipped and no agent is acquired for it — which is what step mode below cannot do. only-keys works in both modes; in step mode it means “do not decide about this step unless it is on the list”, so one shared plugin block can sit on many steps while the list controls which are live.

If you want to try one step first

Step mode puts the plugin on a single step, which then asks about itself. Reach for it when your pipeline lives entirely in Buildkite’s settings editor and there is no upload step to attach to — otherwise prefer only-keys above, which gets you a real skip:

- key: e2e
  label: ":robot: E2E"
  command: make e2e
  secrets:
    - TRUNK_TOKEN
  plugins:
    - trunk-io/dynamic-ci#v<version>:
        mode: step

Two honest differences from filter mode, because the step has already been dispatched by the time Trunk is asked:

• The saving is smaller. The agent is acquired, the repository checked out, and plugins set up before anything is skipped. On a long step that is still most of the cost; on a short one it may be nearly all of it.
• Buildkite reports the step as passed, not skipped, because it ran and did no work. Filter mode marks the step before dispatch and so reports it as genuinely skipped.

A step in this mode must have a key: — it is the whole request — and the plugin fails the step rather than running it if one is missing, so you never get a step that looks filtered but is not.

Your steps need a key:

Trunk identifies a Buildkite step by its declarative key:, which is the only identity that stays stable as labels and matrix values change. A step without one has no history to reason about, so it always runs and the plugin says which steps those were.

- key: unit-tests # <- required for Trunk to decide about this step
  label: ":test_tube: Unit tests"
  command: make test

What it does to your pipeline, and what it does not

The plugin’s only effect is to add a skip: attribute to steps Trunk recommended skipping. Specifically:

• It never writes a file in your repository, and reads none of their contents. It runs git diff for the changed files’ names, statuses and line counts, which it sends to Trunk with the request. The pipeline is rendered by buildkite-agent pipeline upload --dry-run --format json, so your YAML is parsed by Buildkite’s own agent and the change happens to the rendered copy in flight.
• It never changes a command, adds a step, removes a step, or reorders anything.
• It never overrides a skip: you wrote yourself, including skip: false.
• A step the plan did not name is uploaded unchanged.

The whole mutation is seven lines of jq. It is small on purpose: this is the only part of the plugin that changes what your build does, so you should be able to read all of it.

Skipped steps stay visible in the build — use Show skipped steps — with the recommendation’s reason on hover. The log says the same thing in one place, naming every step it marked:

--- :trunk: Dynamic CI marked 2 step(s) to skip
    unit-tests — passed on the last 40 runs; no correlated paths changed
    lint — no correlated paths changed

The reasons there are the full ones, not the 70-character versions Buildkite shows on the step.

It installs nothing

The plugin ships the one tool it needs. vendor/ holds verified static jq binaries for linux-amd64, linux-arm64 and macos-arm64, with their upstream checksums in vendor/SHA256SUMS; the hook checks the checksum before executing, using whichever of sha256sum or shasum your agent has. On a platform we do not ship — or with neither checksum tool present — the plugin says so and uploads your pipeline unchanged.

Working out why nothing skipped

Set debug: true and the plugin prints three collapsed groups to the build log: the step keys it asked about, the request body it sent, and the plan it got back. Between them they answer most questions — whether your steps were seen, whether the repository and pipeline resolved to what you expected, whether baseSha came out null, and what Trunk actually decided.

plugins:
  - trunk-io/dynamic-ci#v<version>:
      debug: true

All of it goes to stderr, so it is safe in filter mode, where stdout carries the pipeline being uploaded. None of it contains your token — that travels in a request header, never in the body.

It fails open

Any failure — an unreachable API, a timeout, a plan this version cannot read, a missing jq — uploads your pipeline unmodified and logs why. This step is on the critical path of every build; the plugin will not be what breaks one.

After each plan request it reports the outcome, the plugin version and how long the request took to Trunk, with the same token. Set TRUNK_DISABLE_TELEMETRY=true to turn this off.

Development

pnpm test

# the request body, without making a call — the shape the tests validate
# against the service’s own schema
lib/request-plan.sh --print-body '["unit-tests"]'

Point the plugin at another Trunk deployment with TRUNK_PUBLIC_API_ADDRESS (a base address; the endpoint path is part of the contract and is always appended). It is the same variable the test-results uploader and the GitHub Action read.

The plugin and the Dynamic CI GitHub Action speak one wire contract — Trunk’s published API schema, synced into both. They release independently.

Versioning

Pin an exact version. Buildkite agents cache a plugin’s checkout by the ref you name, and BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH is off by default — so an agent that has already checked out a moving tag keeps what it has, and moving that tag does not reach it. Pin one exact release tag — #v1.2.3, the form the examples above write as #v<version> — not a major alias and not #main, and a version you audited is the version that runs. A full commit SHA works too and is equally immutable, if you prefer naming one.

The current release is on the Releases page.