Run a suite at scale

Hand-written tests are great for a handful of scenarios. But when you capture production sessions in bulk — hundreds, thousands, maybe tens of thousands of *.case.json files — writing one Jest test per case doesn't scale.

softprobe suite run reads one YAML file and executes every case deterministically, in parallel, emitting JUnit XML for your CI. It's the recommended path for regression-sweeping captured production traffic.

When should I use a suite?

You have…	Use
<10 scenarios, lots of custom assertions	Write Jest / pytest / JUnit / Go tests
10–100 captures with shared behavior	Write a suite; keep a few hand-written tests for edge cases
Thousands of production captures, nightly regression	Suite is the only reasonable option

Suite execution is roughly 10–20× faster per case than launching Jest/pytest worker processes, because there's no test-framework overhead per case — just HTTP.

The one file you write: suite.yaml

# suites/checkout.suite.yaml
name: checkout-nightly
cases: cases/checkout/*.case.json   # glob — can be a list

defaults:
  request:
    source: case.ingress    # replay the ingress request from the case
    transform: unmaskCard   # optional hook name (see "Hooks")

  mocks:
    - name: fragment
      match: { direction: outbound, host: fragment, pathPrefix: /shipping }
      source: case
    - name: stripe
      match: { direction: outbound, hostSuffix: stripe.com, pathPrefix: /v1/payment_intents }
      source: case

  assertions:
    status: 200
    headers:
      include:
        content-type: application/json
    body:
      mode: json-subset
      ignore:
        - "$.paymentId"
        - "$.createdAt"
      custom: assertTotalsMatchItems   # optional hook name

Three sections:

1. cases — which case files to run. Supports globs and lists.
2. defaults — how to build the request, what to mock, and how to assert. Per-case overrides live in cases: [{ path, overrides: {...} }] if you need them.
3. hooks references — optional named functions resolved by the executor (see Write a hook).

Full reference: Suite YAML.

Run it

softprobe suite run suites/checkout.suite.yaml \
  --parallel 32 \
  --junit out/checkout.xml \
  --report out/checkout.html

Flag	Purpose
--parallel N	Run N cases concurrently. Defaults to min(32, cpu * 4).
--junit PATH	Emit JUnit XML (consumed by most CI systems).
--report PATH	Emit a standalone HTML report with per-case diffs.
--hooks PATH[,PATH]	TypeScript hook files (see below).
--filter GLOB	Run only cases matching the substring/glob.
--fail-fast	Stop on first failure (default: run everything).
--env-file FILE	Load environment variables (e.g. TEST_CARD).

What the CLI does per case

for each case in cases:
  1. start a replay session on the runtime
  2. POST /v1/sessions/$ID/load-case with the case file
  3. for each mock in defaults.mocks (and case-specific overrides):
       findInCase on the loaded case
       if transform hook declared → call it with the captured response
       POST /v1/sessions/$ID/rules (accumulated)
  4. build the request from case.ingress
     if transform hook declared → call it with the request
  5. send the request to APP_URL with x-softprobe-session-id: $ID
  6. compare the actual response with case.response
     using assertions (status, headers, body, ignores, redactions)
     if assert-body hook declared → call it with (actual, captured)
  7. close the session
  8. emit one <testcase> into the JUnit XML

Cases run in parallel at step granularity — the runtime handles hundreds of concurrent sessions.

Hooks — when declarative isn't enough

Suites cover 80% of cases with pure YAML. The other 20% need code: PII masking, signature recomputation, custom invariants. For that, suites reference named hooks that the executor resolves at runtime.

In v1, the CLI executor supports TypeScript/JavaScript hooks via a Node sidecar:

mocks:
  - name: stripe
    match: { hostSuffix: stripe.com }
    source: case
    hook: stripe.unmaskCard          # → hooks/stripe.ts export

// hooks/stripe.ts
import type { MockResponseHook, BodyAssertHook } from '@softprobe/softprobe-js/hooks';

export const unmaskCard: MockResponseHook = ({ capturedResponse, env }) => {
  const body = JSON.parse(capturedResponse.body);
  body.source.card.number = env.TEST_CARD ?? '4111111111111111';
  body.source.card.exp_year = 2030;
  return { ...capturedResponse, body: JSON.stringify(body) };
};

export const assertTotalsMatchItems: BodyAssertHook = ({ actual, captured }) => {
  const items = JSON.parse(captured.ingressBody).items;
  const expected = items.reduce((s: number, i: any) => s + i.price, 0)
                 + actual.shipping;
  if (Math.abs(actual.total - expected) > 0.01) {
    return [{ path: '$.total', expected, actual: actual.total }];
  }
  return [];
};

Pass them in at run time:

softprobe suite run suites/checkout.suite.yaml --hooks hooks/stripe.ts

The CLI spawns a Node sidecar once per suite run and streams JSON requests over stdin/stdout for each hook invocation. See Write a hook for the full contract.

Sharing hooks with framework tests

The same hooks/*.ts file can be imported from a Jest runSuite helper, so the custom logic is reused in both CI nightly (CLI) and dev-loop IDE debugging:

// __tests__/checkout.replay.test.ts
import { runSuite } from '@softprobe/softprobe-js/suite';
import * as hooks from '../hooks/stripe';

runSuite('suites/checkout.suite.yaml', { hooks });
// → registers one describe()/it() per case

The pytest, JUnit, and Go adapters also load the same YAML, but their hooks are in the framework's native language (Python/Java/Go). Suites stay the source of truth; hooks localize by executor. See the design note for the rationale.

Overriding per case

When most cases share defaults but a few need tweaks:

cases:
  - path: cases/checkout/happy.case.json
  - path: cases/checkout/card-declined.case.json
    overrides:
      mocks:
        - name: stripe
          response:
            status: 402
            body: '{"error":{"type":"card_error","code":"card_declined"}}'
      assertions:
        status: 402

Overrides shallow-merge into defaults.

Using environment variables

Any ${VAR} in the YAML is expanded at runtime:

request:
  headers:
    authorization: "Bearer ${TEST_TOKEN}"

assertions:
  body:
    ignore:
      - "$.expiresAt"
      - "$.clientSecret"

Pass them via --env-file or the shell:

TEST_TOKEN=eyJ... softprobe suite run suites/checkout.suite.yaml

Output formats

JUnit XML (--junit out/report.xml)

<testsuites name="checkout-nightly" tests="120" failures="2" time="38.41">
  <testsuite name="checkout-nightly" tests="120" failures="2">
    <testcase classname="checkout" name="cases/checkout/happy.case.json" time="0.31"/>
    <testcase classname="checkout" name="cases/checkout/declined.case.json" time="0.29">
      <failure message="body mismatch at $.total">...</failure>
    </testcase>
    ...
  </testsuite>
</testsuites>

Consumable by GitHub Actions, CircleCI, Jenkins, GitLab CI, etc.

HTML report (--report out/report.html)

Self-contained page with per-case pass/fail, durations, diffs of expected vs. actual body. No server needed — open it in a browser or upload as a CI artifact.

JSON (--json)

softprobe suite run suites/checkout.suite.yaml --json > result.json

Stream of JSON objects, one per case, for further processing.

Performance tips

Crank up --parallel for read-heavy cases. Replay is mostly I/O. --parallel 64 on a laptop typically runs 2× as fast as --parallel 16 until you saturate the runtime (one CPU is usually enough).

Put the case files on local SSD. If your cases live on a network mount, suite startup can stall at glob expansion. Copy to /tmp/cases/ in CI.

Scope your captures. A 50-MB case is probably three scenarios pretending to be one. Split them.

Warm up the runtime. The first session's load-case pays for JSON parsing of the case. For very large cases, suite runners pool a case-content cache across parallel workers automatically.

Next

• Write a hook — TypeScript hooks for unmasking PII, custom asserts, dynamic request bodies.
• Suite YAML reference — every field.
• CI integration — running suites in GitHub Actions / GitLab CI.