Guide / Test and run

The scenario file

What a scenario file holds, which Camunda Process Test instructions Penstock runs on a local engine, which it refuses and why, and what Penstock adds under x-penstock.

Pro

What it is

A scenario is a test case in the JSON format of Camunda Process Test. It is one list of instructions: start the process, answer the steps it reaches, and assert what it did. Penstock does not have a format of its own for it. The file is one that Camunda's own tooling reads, so you can run the same scenario in a Camunda Process Test suite, and Penstock plays it back step by step on a local engine and shows what every step received and changed.

This page is the reference for the file. For the tab that lists, generates, edits and runs scenarios, see Scenarios. Running a scenario and editing one are part of Pro, and Team in a pipeline. The file itself is plain JSON, and any editor can open it.

Where the file is

One file per diagram, named after the diagram, in a folder called test-cases:

src/test/resources/test-cases/<diagram>.json when the diagram is in src/main/resources of a Maven or Gradle module.
test-cases/<diagram>.json next to the diagram everywhere else.

The editors, the command line and the test explorer all use the same place. A file that is not valid JSON, or has no testCases list, is left as it is, and Penstock says why instead of writing over it. When it generates scenarios it adds only the cases that are not in the file yet, by name, and keeps every case that is.

The shape of the file

{
  "$schema": "https://camunda.com/json-schema/cpt-test-cases/8.9/schema.json",
  "testCases": [
    {
      "name": "Order is approved",
      "description": "Approved by the clerk, shipped, done.",
      "instructions": [
        {
          "type": "CREATE_PROCESS_INSTANCE",
          "processDefinitionSelector": { "processDefinitionId": "order-process" },
          "variables": { "orderId": "A-1001", "amount": 120 }
        },
        {
          "type": "MOCK_JOB_WORKER_COMPLETE_JOB",
          "jobType": "validate-order",
          "variables": { "valid": true }
        },
        {
          "type": "COMPLETE_USER_TASK",
          "userTaskSelector": { "elementId": "Activity_approve" },
          "variables": { "approved": true }
        },
        {
          "type": "ASSERT_PROCESS_INSTANCE",
          "state": "IS_COMPLETED"
        },
        {
          "type": "ASSERT_VARIABLES",
          "variables": { "approved": true }
        }
      ]
    }
  ]
}

`$schema`	Added on save when the file has none: the Camunda test-case schema, version 8.9.
`testCases`	The list of cases. Required.
`name`	Required, and not empty. The name identifies the case: generating never replaces a case with the same name.
`description`	Optional text.
`instructions`	Required, with at least one instruction. They run top to bottom.
`x-penstock`	Penstock's own additions, below. Camunda's tools ignore it.

Every case is checked against the schema before it runs. An instruction that lacks a required property, or has a property the schema does not allow for its type, is reported by case and instruction number and the case does not run.

Selectors

Instructions point at things with selectors. Penstock reads these:

elementSelector and the entries of elementSelectors: elementId or elementName.
jobSelector: jobType or elementId.
userTaskSelector: elementId or taskName.
processInstanceSelector and processDefinitionSelector: processDefinitionId.

An element name that no step of the diagram has, or an id that is not in the diagram, fails the case with that message.

The instructions Penstock runs

Fifteen of the Camunda instructions run, and two more are known and refused, below. The first instruction that is not a mock must be CREATE_PROCESS_INSTANCE.

Instruction	What Penstock does with it
`CREATE_PROCESS_INSTANCE`	Starts the process with `variables`. `startInstructions` are passed to the engine. One per case.
`MOCK_JOB_WORKER_COMPLETE_JOB`	Answers every job of the type `jobType` with a completion and `variables`. It stands for the worker.
`MOCK_JOB_WORKER_THROW_BPMN_ERROR`	Answers every job of the type with a BPMN error: `errorCode` is required, `errorMessage` and `variables` are optional.
`COMPLETE_JOB`	Completes the waiting job that the `jobSelector` names, with `variables`.
`THROW_BPMN_ERROR_FROM_JOB`	Throws a BPMN error from that job instead of completing it. `errorCode` is required, `errorMessage` and `variables` are optional.
`COMPLETE_USER_TASK`	Waits for the user task the `userTaskSelector` names to be waiting, then completes it with `variables`.
`CORRELATE_MESSAGE`, `PUBLISH_MESSAGE`	Sends the message `name` with a `correlationKey` and `variables`.
`BROADCAST_SIGNAL`	Broadcasts the signal `signalName` to the engine, with `variables`, so that catch events and start events waiting for it go on.
`UPDATE_VARIABLES`	Sets `variables` on the instance, or on the element an `elementSelector` names.
`ASSERT_PROCESS_INSTANCE`	Checks the `state`: `IS_ACTIVE`, `IS_COMPLETED` or `IS_TERMINATED`, and `hasActiveIncidents` true or false. `IS_CREATED` is accepted and checks nothing.
`ASSERT_ELEMENT_INSTANCE`	Checks that an element has an instance in the `state` `IS_ACTIVE`, `IS_COMPLETED` or `IS_TERMINATED`. With `amount` the number of such instances must be exactly that.
`ASSERT_ELEMENT_INSTANCES`	The same for a list of elements, with the states `IS_ACTIVE`, `IS_COMPLETED`, `IS_TERMINATED`, `IS_NOT_ACTIVE`, `IS_NOT_ACTIVATED`, `IS_ACTIVE_EXACTLY` and `IS_COMPLETED_IN_ORDER`.
`ASSERT_VARIABLES`	Checks that the names in `variableNames` are set, and that every entry of `variables` is set to that value. With an `elementSelector` it looks at the variables local to that element.
`ASSERT_USER_TASK`	Checks the `state`: `IS_CREATED` (the task is waiting), `IS_COMPLETED`, `IS_CANCELED` or `IS_FAILED`, and that `elementId` is the task selected. With no state it checks that the task was reached.
`MOCK_CHILD_PROCESS`, `MOCK_DMN_DECISION`	Accepted by the schema, refused by the runner: Penstock says it cannot run them yet.

What Penstock refuses

A case that Penstock cannot run fails at once with a message that names the instruction, before anything is deployed. It does not run half of it. The first instruction that cannot run gives the message.

Instructions it does not know. Besides the two mocks above, the schema allows nine more: ASSERT_DECISION, ASSERT_PROCESS_INSTANCE_MESSAGE_SUBSCRIPTION, COMPLETE_JOB_AD_HOC_SUB_PROCESS, COMPLETE_JOB_USER_TASK_LISTENER, EVALUATE_CONDITIONAL_START_EVENT, EVALUATE_DECISION, INCREASE_TIME, RESOLVE_INCIDENT and SET_TIME. They are valid in the file and Camunda Process Test can run them, but Penstock says is not something Penstock can run.
More than one CREATE_PROCESS_INSTANCE. Penstock runs one instance per case.
A selector for another process. Penstock follows only the instance of the process the case started. An instruction whose selector names another processDefinitionId is refused.
useExampleData: true on a job or user task instruction.
Runtime instructions. A non-empty runtimeInstructions on CREATE_PROCESS_INSTANCE.
These fields of ASSERT_USER_TASK: assignee, candidateGroups, priority, name, dueDate and followUpDate.

What Penstock adds: `x-penstock`

A case can carry an x-penstock block. Camunda's tools ignore it. It has three parts.

"x-penstock": {
  "engines": ["camunda-8", "camunda-7"],
  "behaviours": {
    "ReserveStock": { "then": "complete", "fail": 2, "variables": { "reserved": 3 } },
    "WaitForPayment": { "then": "fire" },
    "CheckCredit": { "then": "error", "errorCode": "NO_CREDIT" }
  }
}

`engines`	The engines the case is for, from `camunda-8`, `camunda-7`, `operaton`, `cib-seven` and `classic`. On any other engine the case is skipped, not failed, and says which engines it is for. `camunda-7` covers Operaton and CIB seven.
`behaviours`	What a step does, keyed by the id of the element. It answers a service task or a timer without a mock instruction.
`recorded`	Written by Penstock when a case is recorded from a run. It names the run it came from: `from`, `instance`, `engine` and `at`.

A behaviour has a required then:

`then`	What the step does
`complete`	Completes the job, with `variables`.
`error`	Throws a BPMN error. `errorCode` is required, `errorMessage` is optional.
`fire`	For a timer: fires it at once instead of waiting for the time.
`hold`	Leaves the job unanswered and keeps its lock fresh, so the step stays waiting.

fail is a number. The step fails that many times, on purpose and with a message that says so, before it does what then says. It is how a scenario checks a retry.

A case with at least one behaviour is shown with the tag Penstock only: Camunda Process Test would run it differently, because it does not know the block. A block that is not an object, an engines list with an unknown engine, or a behaviour without a valid then is reported as a problem of the case.

What a run does with the file

It reads the file and checks every case. A case for another engine is skipped.
It builds a copy of the process for the test and deploys it to the local engine you chose. See Scenario runs are safe by design for what is and is not touched.
It runs the instructions in order. A step that waits for something that never comes fails after the step timeout, 30 seconds by default, with the name of the step that did not run. --timeout on the command line changes it.
It reads the instance after each step and checks the assertions. A failed assertion says what it expected, what it found, and which steps were active.
It removes its build from the engine and writes a trace of the run to .penstock/runs/. See Files Penstock reads and writes.

On the command line, interrupting a run once asks the engine to stop and the run goes on. Interrupting it twice cancels the running case, removes its build and stops, with exit code 130.

What you see

A file of cases that you can read, diff and review like any other file of the project.
In the Scenarios tab, each case with its steps, a Penstock only tag where it uses behaviours, and a message that names the instruction where a case cannot run.

Common problems

"is not something Penstock can run".: The instruction is valid in the Camunda schema but not one of the fifteen above. Run that case with Camunda Process Test, or rewrite it with the instructions Penstock runs.
"Penstock runs one instance per test case".: The case has two CREATE_PROCESS_INSTANCE. Split it into two cases.
The case is skipped, not run.: x-penstock.engines does not list the engine of the folder. Add it, or remove the list.
A run waits and then says the engine did not run a step.: Nothing took the job within the timeout. Add a MOCK_JOB_WORKER_COMPLETE_JOB for its job type, or a behaviour for the element. See Scenarios.
My file was not changed by Generate.: Generate adds a case only when its name is not in the file. Delete a case, or rename it, to have it written again.

The scenario file

See also