warning

This doc is deprecated and merged into test-suites.md. It will remain hidden for a time then removed.

Lifecycle of a test suite

Overview

Start with workflow v1

When you have a workflow that is ready for testing, prepare the test suite.

$ sandgarden workflows push --name=test-workflow --file=./examples/workflows/multifile-aws --entrypoint="handler.handler" 

✅ Workflow pushed successfully!
Workflow ID: wrk-1234
Workflow Version: 1

Run tests and review results

Run the test suite against the workflow for baseline results.

$ sandgarden batches start --workflow=test-workflow:1 --in={"input":{"width":10,"length":12},"validators":[{"regex":"12.*","jpath":"$.area"}]}

Batch created:
Batch ID: bat-123
Status: running

Push update to workflow v2

Update your workflow, then re-run the test suite.

$ sandgarden workflows push --name=test-workflow --file=./examples/workflows/multifile-aws --entrypoint="handler.handler" 

✅ Workflow pushed successfully!
Workflow ID: wrk-1234
Workflow Version: 2
$ sandgarden batches start --workflow=test-workflow:2 --in={"input":{"width":10,"length":12},"validators":[{"regex":"12.*","jpath":"$.area"}]}

Batch created:
Batch ID: bat-124
Status: running

Compare results

Finally, compare the results of the two workflow versions with sandgarden batches compare. This will show you whether there were improvements, regressions, or no changes from the earlier version of the workflow.

$ sandgarden batches compare $BATCH_ID_1 $BATCH_ID_2

bat_01jb9zc1psenr9q7fajp3y7p2r
Batch Results                                      Batch Comparison
╭────────────────────────────────────────────────╮ ╭────────────────────────────────────────────────╮
│ Batch bat_01jb9z9brcenqa5m6t803956mp           │ │ Shared tests      Shared tests between batches │
│                                                │ │                                                │
│ Passed                                       2 │ │ Passing                                      2 │
│ Failed                                       2 │ │ Failing                                      0 │
│ Total                                        4 │ │ Total                                        2 │
│                                                │ │                                          ----- │
│ Percentage                              50.00% │ │ Percentage                             100.00% │
│ ────────────────────────────────────────────── │ │                                                │
│ Batch bat_01jb9zc1psenr9q7fajp3y7p2r           │ │ Improvements                       0/0 (0.00%) │
│                                                │ │ Regressions                        0/0 (0.00%) │
│ Passed                                       4 │ ╰────────────────────────────────────────────────╯
│ Failed                                       0 │
│ Total                                        4 │
│                                                │
│ Percentage                             100.00% │
│              (+ 2 test added) (50.00% passing) │
│              (-2 test removed) (0.00% passing) │
╰────────────────────────────────────────────────╯