GitHub - scenarigo/scenarigo: An end-to-end scenario testing tool for HTTP/gRPC server.

A scenario-based API testing tool for HTTP/gRPC server.

Overview

Scenarigo is a scenario-based API testing tool for HTTP/gRPC server. It is written in Go and provides a plugin feature that enables you to extend by writing Go code. You can write test scenarios as YAML files and executes them.

title: get scenarigo repository
vars:
  user: scenarigo
  repo: scenarigo
steps:
- title: get repository
  protocol: http
  request:
    method: GET
    url: 'https://api.github.com/repos/{{vars.user}}/{{vars.repo}}'
  expect:
    code: OK
    body:
      id: '{{int($) > 0}}'
      name: '{{vars.repo}}'

Features

Multi-Protocol Support - Test both HTTP/REST and gRPC APIs
YAML-based Scenarios - Write test scenarios in a readable, declarative format
Template Strings - Dynamic value generation and validation with template expressions
Plugin System - Extend functionality by writing custom Go plugins
Variables and Secrets - Manage test data with variable scoping and secret masking
Retry Policies - Built-in retry with constant or exponential backoff strategies
Conditional Execution - Control test flow with conditional step execution
ytt Integration - Advanced templating and overlay capabilities for test scenarios

Quick Start

Installation

Install Scenarigo using Go:

$ go install github.com/scenarigo/scenarigo/cmd/scenarigo@latest

Your First Test

Create a simple test scenario file hello.yaml:

title: Hello Scenarigo
steps:
- title: Check GitHub API
  protocol: http
  request:
    method: GET
    url: https://api.github.com/repos/scenarigo/scenarigo
  expect:
    code: OK
    body:
      name: scenarigo

Running Tests

Create a configuration file and run the test:

# Initialize configuration
$ scenarigo config init

# Run the test
$ scenarigo run hello.yaml
ok      hello.yaml     0.123s

That's it! You've just run your first Scenarigo test. Continue reading to learn more advanced features.

Installation (Detailed)

go install command (recommend)

$ go install github.com/scenarigo/scenarigo/cmd/scenarigo@latest

from release page

Go to the releases page and download the zip file. Unpack the zip file, and put the binary to a directory in your $PATH.

You can download the latest command into the ./scenarigo directory with the following one-liner code. Place the binary ./scenarigo/scenarigo into your $PATH.

$ version=$(curl -s https://api.github.com/repos/scenarigo/scenarigo/releases/latest | jq -r '.tag_name') && \
    go_version=$(echo -n $(curl -s 'https://go.dev/VERSION?m=text' | head -n 1)) && \
    curl -sLJ https://github.com/scenarigo/scenarigo/releases/download/${version}/scenarigo_${version}_${go_version}_$(uname)_$(uname -m).tar.gz -o scenarigo.tar.gz && \
    mkdir ./scenarigo && tar -zxvf ./scenarigo.tar.gz -C ./scenarigo && rm scenarigo.tar.gz

Notes: If you use the plugin mechanism, the scenarigo command and plugins must be built using the same version of Go.

Setup

You can generate a configuration file scenarigo.yaml via the following command.

$ scenarigo config init

schemaVersion: config/v1

# global variables
vars:
  endpoint: http://api.example.com

scenarios: [] # Specify test scenario files and directories.

pluginDirectory: ./gen    # Specify the root directory of plugins.
plugins:                  # Specify configurations to build plugins.
  plugin.so:              # Map keys specify plugin output file path from the root directory of plugins.
    src: ./path/to/plugin # Specify the source file, directory, or "go gettable" module path of the plugin.

output:
  verbose: false # Enable verbose output.
  colored: false # Enable colored output with ANSI color escape codes. It is enabled by default but disabled when a NO_COLOR environment variable is set (regardless of its value).
  summary: false # Enable summary output.
  report:
    json:
      filename: ./report.json # Specify a filename for test report output in JSON.
    junit:
      filename: ./junit.xml   # Specify a filename for test report output in JUnit XML format.

Usage

scenarigo run executes test scenarios based on the configuration file.

schemaVersion: config/v1

scenarios:
- github.yaml

title: get scenarigo repository
steps:
- title: GET https://api.github.com/repos/scenarigo/scenarigo
  vars:
    user: scenarigo
    repo: scenarigo
  protocol: http
  request:
    method: GET
    url: "https://api.github.com/repos/{{vars.user}}/{{vars.repo}}"
  expect:
    code: OK
    body:
      name: "{{vars.repo}}"

$ scenarigo run
ok      github.yaml     0.068s

Alternatively, provide the paths to specific test files as arguments.

$ scenarigo run github.yaml

You can see all commands and options by scenarigo help.

scenarigo is a scenario-based API testing tool.

Usage:
  scenarigo [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  config      manage the scenarigo configuration file
  dump        dump test scenario files
  help        Help about any command
  list        list the test scenario files
  plugin      provide operations for plugins
  run         run test scenarios
  version     print scenarigo version

Flags:
  -c, --config string   specify configuration file path (read configuration from stdin if specified "-")
  -h, --help            help for scenarigo
      --root string     specify root directory (default value is the directory of configuration file)

Use "scenarigo [command] --help" for more information about a command.

How to write test scenarios

You can write test scenarios easily in YAML. A test scenario consists of steps that are executed sequentially from top to bottom. Each step represents an API request (HTTP or gRPC) and its expected response.

Scenarigo supports testing both HTTP/REST and gRPC APIs. The following sections describe how to write tests for each protocol.

HTTP Testing

Send HTTP requests

This simple example has a step that sends a GET request to http://example.com/message.

title: check /message
steps:
- title: GET /message
  protocol: http
  request:
    method: GET
    url: http://example.com/message

To send a query parameter, add it directly to the URL or use the query field.

title: check /message
steps:
- title: GET /message
  protocol: http
  request:
    method: GET
    url: http://example.com/message
    query:
      id: 1

You can use other methods to send data to your APIs.

title: check /message
steps:
- title: POST /message
  protocol: http
  request:
    method: POST
    url: http://example.com/message
    body:
      message: hello

By default, Scenarigo will send body data as JSON. If you want to use other formats, set the Content-Type header.

title: check /message
steps:
- title: POST /message
  protocol: http
  request:
    method: POST
    url: http://example.com/message
    header:
      Content-Type: application/x-www-form-urlencoded
    body:
      message: hello

Available Content-Type header to encode request body is the following.

application/json (default)
text/plain
application/x-www-form-urlencoded

Check HTTP responses

You can test your APIs by checking responses. If the result differs from the expected values, Scenarigo aborts the execution of the test scenario and notifies the error.

Scenarigo provides three ways to validate response values in the expect field:

Exact Matching - Compare values directly for equality
Template Expressions - Use conditional expressions with the actual value $
Assertion Functions - Use built-in assertion functions for common validations

Exact Matching

The simplest way to validate responses is to specify the expected values directly. Scenarigo will compare them for exact equality.

title: exact matching
steps:
- title: GET /message
  protocol: http
  request:
    method: GET
    url: http://example.com/message
    query:
      id: 1
  expect:
    code: OK
    header:
      Content-Type: application/json; charset=utf-8
    body:
      id: 1
      message: hello

This method is best when you know the exact expected value and want a simple equality check.

Template Expressions

For more flexible validations, you can use template string expressions with the actual value represented by $. This allows you to write conditional expressions and perform calculations.

title: template expressions
steps:
- title: GET /message
  protocol: http
  request:
    method: GET
    url: http://example.com/message
    query:
      id: 1
  expect:
    code: OK
    header:
      Content-Type: application/json; charset=utf-8
    body:
      id: '{{int($) > 0}}'                                    # Check if id is positive
      message: '{{"hello" + " world"}}'                       # String concatenation
      timestamp: '{{time($) > time("2024-01-01T00:00:00Z")}}' # Time comparison

Template expressions are useful when:

You need to perform range checks or comparisons
The exact value is unknown but must satisfy certain conditions
You want to perform type conversions before validation

Assertion Functions

Scenarigo provides the assert variable with built-in assertion functions for common validation patterns. These functions offer a more expressive and readable way to validate responses.

Available Assertion Functions:

Function	Usage	Description
any	`'{{assert.any}}'`	Always passes without validating the actual value
notZero	`'{{assert.notZero}}'`	Ensures the value is not a zero value
regexp	`'{{assert.regexp("^[a-z]+$")}}'`	Ensures the value matches the regular expression pattern
length	`'{{assert.length(3)}}'`	Ensures the length of a string, array, slice, or map equals the expected value
greaterThan	`'{{assert.greaterThan(10)}}'`	Ensures the value is greater than the expected value
greaterThanOrEqual	`'{{assert.greaterThanOrEqual(10)}}'`	Ensures the value is greater than or equal to the expected value
lessThan	`'{{assert.lessThan(100)}}'`	Ensures the value is less than the expected value
lessThanOrEqual	`'{{assert.lessThanOrEqual(100)}}'`	Ensures the value is less than or equal to the expected value
contains	`'{{assert.contains <-}}': value`	Ensures the array or slice contains the specified value (uses Left Arrow Function)
notContains	`'{{assert.notContains <-}}': value`	Ensures the array or slice does not contain the specified value (uses Left Arrow Function)
and	`'{{assert.and <-}}': [assertion1, assertion2]`	Ensures the value passes all assertions (uses Left Arrow Function)
or	`'{{assert.or <-}}': [assertion1, assertion2]`	Ensures the value passes at least one of the assertions (uses Left Arrow Function)

Example:

title: assertion functions
steps:
- title: GET /users/1
  protocol: http
  request:
    method: GET
    url: http://example.com/users/1
  expect:
    code: OK
    body:
      id: '{{assert.notZero}}'
      name: '{{assert.regexp("^[A-Za-z ]+$")}}'
      age: '{{assert.greaterThanOrEqual(0)}}'
      email: '{{assert.regexp("^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$")}}'
      tags: '{{assert.length(3)}}'

For advanced assertions, you can combine multiple conditions using the and and or functions with the Left Arrow Function syntax:

expect:
  body:
    # Ensures age is between 20 and 65
    age:
      '{{assert.and <-}}':
      - '{{assert.greaterThanOrEqual(20)}}'
      - '{{assert.lessThanOrEqual(65)}}'
    # Ensures status is either "active" or "pending"
    status:
      '{{assert.or <-}}':
      - active
      - pending

Adding Custom Assertions via Plugins

You can extend the assertion functions by creating a plugin that implements the assert.Assertion interface. This allows you to add domain-specific validations tailored to your testing needs.

Example Plugin (plugin/src/main.go):

package main

import (
	"fmt"
	"regexp"

	"github.com/scenarigo/scenarigo/assert"
)

// EmailFormat returns an assertion that validates email format
var EmailFormat = assert.AssertionFunc(func(v any) error {
	email, ok := v.(string)
	if !ok {
		return fmt.Errorf("expected string but got %T", v)
	}
	pattern := `^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$`
	matched, _ := regexp.MatchString(pattern, email)
	if !matched {
		return fmt.Errorf("%q is not a valid email format", email)
	}
	return nil
})

// InRange returns an assertion that validates a number is within a range
func InRange(min, max int) assert.Assertion {
	return assert.AssertionFunc(func(v any) error {
		var num int
		switch val := v.(type) {
		case int:
			num = val
		case float64:
			num = int(val)
		default:
			return fmt.Errorf("expected number but got %T", v)
		}
		if num < min || num > max {
			return fmt.Errorf("%d is not in range [%d, %d]", num, min, max)
		}
		return nil
	})
}

Usage in Test Scenarios:

title: custom assertions
plugins:
  myassert: myassert.so
steps:
- title: POST /users
  protocol: http
  request:
    method: POST
    url: http://example.com/users
    body:
      name: John Doe
      email: [email protected]
      age: 30
  expect:
    code: Created
    body:
      email: '{{plugins.myassert.EmailFormat}}'
      age: '{{plugins.myassert.InRange(18, 65)}}'

The assert.Assertion interface requires only one method:

type Assertion interface {
    Assert(v any) error
}

You can implement this interface directly or use the convenient assert.AssertionFunc adapter to convert a function into an assertion.

Combining Validation Methods

You can combine all three validation methods in a single test scenario to leverage the strengths of each approach:

title: combined validation
steps:
- title: POST /users
  protocol: http
  request:
    method: POST
    url: http://example.com/users
    body:
      name: John Doe
      email: [email protected]
  expect:
    code: Created                                             # Exact matching
    body:
      id: '{{assert.notZero}}'                                # Assertion function
      name: John Doe                                          # Exact matching
      email: '{{assert.regexp("^[^@]+@[^@]+$")}}'             # Assertion function
      createdAt: '{{time($) > time("2024-01-01T00:00:00Z")}}' # Template expression
      status: active                                          # Exact matching

Custom Client

Scenarigo allows you to use custom clients defined in plugins. You can pass custom clients through the client field in your test scenarios.

For HTTP tests, you can pass a custom *http.Client instance defined in your plugin:

package main

import (
	"net/http"
	"time"
)

var CustomHTTPClient = &http.Client{
	Timeout: 10 * time.Second,
	// Add custom transport, middleware, etc.
}

title: test with custom HTTP client
plugins:
  client: client.so
steps:
- title: GET /api/resource
  protocol: http
  request:
    method: GET
    url: http://example.com/api/resource
    client: '{{plugins.client.CustomHTTPClient}}'
  expect:
    code: OK

Using custom clients allows you to:

Configure custom timeouts and retry policies
Add authentication tokens or headers
Implement custom middleware or interceptors
Mock or stub external dependencies for testing

gRPC Testing

Scenarigo supports gRPC API testing with two methods for loading service definitions: Protocol Buffers files or gRPC reflection.

Send gRPC requests

Using Protocol Buffers files

You can load .proto files to define the service schema:

title: gRPC test with proto files
steps:
- title: Call Echo service
  protocol: grpc
  request:
    target: localhost:50051
    service: myapp.EchoService
    method: Echo
    message:
      text: "Hello, gRPC!"
    options:
      proto:
        imports:
        - ./proto        # Import paths for .proto files
        files:
        - service.proto  # Proto files to load
      auth:
        insecure: true   # Use insecure connection (for development)
  expect:
    status:
      code: OK
    message:
      text: "Hello, gRPC!"

Configuration options:

target: The gRPC server address (e.g., localhost:50051)
service: Full service name as defined in the .proto file
method: The RPC method name to call
message: The request message (as YAML/JSON matching the protobuf structure)
options.proto.imports: List of directories to search for .proto files
options.proto.files: List of .proto files to load
options.auth.insecure: Set to true for insecure connections (no TLS)

Setting Default Options in scenarigo.yaml:

To avoid repeating common options like proto imports and auth configuration in every test scenario, you can define them as defaults in your scenarigo.yaml configuration file:

schemaVersion: config/v1

scenarios:
- scenarios

protocols:
  grpc:
    request:
      proto:
        imports:
        - ./proto        # Default import paths for all gRPC tests
      auth:
        insecure: true   # Default auth settings for all gRPC tests

When default options are configured, individual test scenarios can omit these settings:

title: gRPC test with defaults
steps:
- title: Call Echo service
  protocol: grpc
  request:
    target: localhost:50051
    service: myapp.EchoService
    method: Echo
    message:
      text: "Hello, gRPC!"
    options:
      proto:
        files:
        - service.proto  # Only specify the files, imports come from defaults
  expect:
    status:
      code: OK
    message:
      text: "Hello, gRPC!"

If a test scenario specifies options that conflict with the defaults, the scenario-level settings take precedence. The settings are merged, allowing you to override specific values while keeping others from the defaults.

Using gRPC reflection

If your gRPC server supports gRPC reflection, Scenarigo can automatically discover service definitions:

title: gRPC test with reflection
steps:
- title: Call Echo service
  protocol: grpc
  request:
    target: localhost:50051
    service: myapp.EchoService
    method: Echo
    message:
      text: "Hello, gRPC!"
    options:
      reflection:
        enabled: true    # Enable gRPC reflection
      auth:
        insecure: true
  expect:
    status:
      code: OK
    message:
      text: "Hello, gRPC!"

gRPC reflection is particularly useful during development and testing as it eliminates the need to maintain .proto files in your test repository.

Check gRPC responses

Similar to HTTP testing, you can validate gRPC responses using the same three validation methods:

Exact Matching

expect:
  status:
    code: OK           # gRPC status code
  message:
    id: 123
    text: "exact match"

Template Expressions

expect:
  status:
    code: OK
  message:
    id: '{{int($) > 0}}'
    text: '{{size($) > 0}}'
    timestamp: '{{time($) > time("2024-01-01T00:00:00Z")}}'

Assertion Functions

expect:
  status:
    code: OK
  message:
    id: '{{assert.notZero}}'
    text: '{{assert.regexp("^[A-Za-z]+$")}}'
    items: '{{assert.length(5)}}'

Custom Client

For gRPC tests, you can pass an auto-generated gRPC client instance:

package main

import (
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	pb "path/to/your/protobuf/package"
)

func GetCustomGRPCClient() (pb.YourServiceClient, error) {
	conn, err := grpc.Dial(
		"localhost:50051",
		grpc.WithTransportCredentials(insecure.NewCredentials()),
		// Add custom interceptors, options, etc.
	)
	if err != nil {
		return nil, err
	}
	return pb.NewYourServiceClient(conn), nil
}

title: test with custom gRPC client
plugins:
  client: client.so
steps:
- title: Call RPC method
  protocol: grpc
  request:
    client: '{{plugins.client.GetCustomGRPCClient()}}'
    method: YourMethod
    message:
      field: value
  expect:
    status:
      code: OK

gRPC-specific features

Status codes

gRPC uses a different set of status codes than HTTP. Common status codes include:

OK (0): Success
CANCELLED (1): Operation cancelled
INVALID_ARGUMENT (3): Invalid argument
NOT_FOUND (5): Not found
ALREADY_EXISTS (6): Already exists
PERMISSION_DENIED (7): Permission denied
UNAUTHENTICATED (16): Unauthenticated

See the full list of gRPC status codes.

Authentication

For TLS-enabled gRPC servers, configure authentication options:

request:
  target: secure.example.com:443
  service: myapp.SecureService
  method: GetData
  options:
    auth:
      # TLS/SSL configuration
      cacert: /path/to/ca.pem        # CA certificate
      cert: /path/to/client-cert.pem # Client certificate
      key: /path/to/client-key.pem   # Client private key
      serverName: example.com        # Server name for TLS verification

For insecure connections (development only):

request:
  options:
    auth:
      insecure: true

Common Features

The following features are available for both HTTP and gRPC testing.

Variables

The vars field defines variables that can be referred by template string like '{{vars.id}}'.

title: get message 1
vars:
  id: 1
steps:
- title: GET /messages
  protocol: http
  request:
    method: GET
    url: 'http://example.com/messages/{{vars.id}}'

You can define step scope variables that can't be accessed from other steps.

title: get message 1
steps:
- title: GET /messages
  vars:
    id: 1
  protocol: http
  request:
    method: GET
    url: 'http://example.com/messages/{{vars.id}}'

If you want to pass the response data to the subsequent steps, use the bind field.

title: re-post message 1
vars:
  id: 1
steps:
- title: GET /messages
  protocol: http
  request:
    method: GET
    url: 'http://example.com/messages/{{vars.id}}'
  bind:
    vars:
      msg: '{{response.body.text}}'
- title: POST /messages
  protocol: http
  request:
    method: POST
    url: http://example.com/messages
    header:
      Content-Type: application/json
    body:
      text: '{{vars.msg}}'
  expect:
    code: OK
    body:
      id: '{{assert.notZero}}'
      text: '{{request.body.text}}'

You can also define global variables in the scenarigo.yaml. The defined variables can be used from all test scenarios.

schemaVersion: config/v1
vars:
  name: zoncoen

Secrets

The secrets field allows defining variables like the vars field. Besides, the values defined by the secrets field are masked in the outputs.

schemaVersion: scenario/v1
plugins:
  plugin: plugin.so
vars:
  clientId: abcdef
secrets:
  clientSecret: XXXXX
title: get user profile
steps:
- title: get access token
  protocol: http
  request:
    method: POST
    url: 'http://example.com/oauth/token'
    header:
      Content-Type: application/x-www-form-urlencoded
    body:
      grant_type: client_credentials
      client_id: '{{vars.clientId}}'
      client_secret: '{{secrets.clientSecret}}'
  expect:
    code: OK
    body:
      access_token: '{{$ != ""}}'
      token_type: Bearer
  bind:
    secrets:
      accessToken: '{{response.body.access_token}}'
- title: get user profile
  protocol: http
  request:
    method: GET
    url: 'http://example.com/users/zoncoen'
    header:
      Authorization: 'Bearer {{secrets.accessToken}}'
  expect:
    code: OK
    body:
      name: zoncoen

...
        --- PASS: scenarios/get-profile.yaml/get_user_profile/get_access_token (0.00s)
                request:
                  method: POST
                  url: http://example.com/oauth/token
                  header:
                    Content-Type:
                    - application/x-www-form-urlencoded
                  body:
                    client_id: abcdef
                    client_secret: {{secrets.clientSecret}}
                    grant_type: client_credentials
                response:
...
                  body:
                    access_token: {{secrets.accessToken}}
                    token_type: Bearer
                elapsed time: 0.001743 sec
        --- PASS: scenarios/get-profile.yaml/get_user_profile/get_user_profile (0.00s)
                request:
                  method: GET
                  url: http://example.com/users/zoncoen
                  header:
                    Authorization:
                    - Bearer {{secrets.accessToken}}
...

Timeout/Retry

You can set timeout and retry policy for each step or for the entire scenario. Duration strings are parsed by time.ParseDuration. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h".

Step-level Retry

You can set retry policy for individual steps:

steps:
- protocol: http
  request:
    method: GET
    url: http://example.com
  expect:
    code: OK
  timeout: 30s           # default values is 0, 0 means no timeout
  retry:                 # default policy is never retry
    constant:
      interval: 5s       # default value is 1s
      maxRetries: 1      # default value is 5, 0 means forever
      maxElapsedTime: 1m # default value is 0, 0 means forever

Scenarigo also provides the retry feature with an exponential backoff algorithm.

steps:
- protocol: http
  request:
    method: GET
    url: http://example.com
  expect:
    code: OK
  timeout: 30s             # default values is 0, 0 means no timeout
  retry:                   # default policy is never retry
    exponential:
      initialInterval: 1s  # default value is 500ms
      factor: 2            # default value is 1.5
      jitterFactor: 0.5    # default value is 0.5
      maxInterval: 180s    # default value is 60s
      maxRetries: 10       # default value is 5, 0 means forever
      maxElapsedTime: 10m  # default value is 0, 0 means forever

The actual interval is calculated using the following formula.

initialInterval * factor ^ (retry count - 1) * (random value in range [1 - jitterFactor, 1 + jitterFactor])

For example, the retry intervals will be like the following table with the above retry policy.

Note: maxInterval caps the retry interval, not the randomized interval.

Retry #	Retry interval	Randomized interval range
1	1s	[0.5s, 1.5s]
2	2s	[1s, 3s]
3	4s	[2s, 6s]
4	8s	[4s, 12s]
5	16s	[8s, 24s]
6	32s	[16s, 48s]
7	64s	[32s, 96s]
8	128s	[64s, 192s]
9	180s	[90s, 270s]
10	180s	[90s, 270s]

Scenario-level Retry

You can also set retry policy for the entire scenario. This will retry all steps in the scenario if any step fails:

title: my test scenario
retry:                   # retry policy for the entire scenario
  constant:
    interval: 5s
    maxRetries: 3
    maxElapsedTime: 30s
steps:
- protocol: http
  request:
    method: GET
    url: http://example.com
  expect:
    code: OK
- protocol: http
  request:
    method: POST
    url: http://example.com/data
    body:
      key: value
  expect:
    code: Created

When a scenario has a retry policy, if any step fails, the entire scenario will be retried according to the policy. This is useful when you want to retry a sequence of dependent steps together.

You can also use exponential backoff for scenario-level retry:

title: my test scenario
retry:
  exponential:
    initialInterval: 1s
    factor: 2
    maxRetries: 5
    maxElapsedTime: 1m
steps:
- protocol: http
  request:
    method: GET
    url: http://example.com
  expect:
    code: OK

Using conditions to control step execution

You can use if field to prevent a step from execution unless a condition is met. The template expression must return a boolean value. For example, you can access the results of other steps like {{steps.step_id.result}}. There are three result kinds of steps: passed, failed, and skipped.

Scenarigo doesn't execute subsequent steps if a step fails in default. If you want to continue running the test scenario even if a step fails, set true to the continueOnError field.

For example, the second step will be executed in the following test scenario when the first step fails only.

schemaVersion: scenario/v1
title: create item if not found
vars:
  itemName: foo
  itemPrice: 100
steps:
- id: find # need to set id to access the result of this step
  title: find by name
  continueOnError: true # the errors of this step don't fail the test scenario
  protocol: http
  request:
    method: GET
    url: 'http://example.com/items?name={{vars.itemName}}'
  expect:
    code: OK
    body:
      name: '{{vars.itemName}}'
  bind:
    vars:
      itemId: '{{response.body.id}}'
- title: create
  if: '{{steps.find.result == "failed"}}' # this step will be executed when the find step fails only
  protocol: http
  request:
    method: POST
    url: 'http://example.com/items'
    header:
      Content-Type: application/json
    body:
      name: '{{vars.itemName}}'
      price: '{{vars.itemPrice}}'
  expect:
    code: OK
    body:
      name: '{{vars.itemName}}'
  bind:
    vars:
      itemId: '{{response.body.id}}'

Template String

Template strings are a core feature of Scenarigo that enable dynamic values and validations throughout your test scenarios. This section provides a complete reference for template string syntax and capabilities.

Overview

Scenarigo provides a powerful template string feature evaluated at runtime. You can use expressions with a pair of double braces {{}} in YAML strings. All expressions return an arbitrary value.

For instance, '{{1}}' is evaluated as an integer 1 at runtime.

vars:
  id: '{{1}}' # id: 1

You can mix the templates into a raw string if all expressions' results are a string.

vars:
  text: 'foo-{{"bar"}}-baz' # text: 'foo-bar-baz'

Syntax

The grammar of the template is defined below, using | for alternatives, [] for optional, {} for repeated, () for grouping, and ... for character range.

ParameterExpr   = "{{" Expr "}}"
Expr            = UnaryExpr | BinaryExpr | ConditionalExpr
UnaryExpr       = [UnaryOp] (
                    ParenExpr | SelectorExpr | IndexExpr | CallExpr |
                    INT | FLOAT | BOOL | STRING | IDENT
                  )
UnaryOp         = "!" | "-"
ParenExpr       = "(" Expr ")"
SelectorExpr    = Expr "." IDENT
IndexExpr       = Expr "[" INT "]"
CallExpr        = Expr "(" [Expr {"," Expr}] ")"
BinaryExpr      = Expr BinaryOp Expr
BinaryOp        = "+" | "-" | "*" | "/" | "%" |
                  "&&" | "||" | "??" |
                  "==" | "!=" | "<" | "<=" | ">" | ">=" 
ConditionalExpr = Expr ? Expr : Expr

The lexis is defined below.

INT           = "0" | ("1"..."9" {DECIMAL_DIGIT})
FLOAT         = INT "." DECIMAL_DIGIT {DECIMAL_DIGIT}
BOOL          = "true" | "false"
STRING        = `"` {UNICODE_VALUE} `"`
IDENT         = (LETTER {LETTER | DECIMAL_DIGIT | "-" | "_"} | "$") - RESERVED

DECIMAL_DIGIT = "0"..."9"
UNICODE_VALUE = UNICODE_CHAR | ESCAPED_CHAR
UNICODE_CHAR  = /* an arbitrary UTF-8 encoded char */
ESCAPED_CHAR  = "\" `"`
LETTER        = "a"..."Z"
TYPES         = "int" | "uint" | "float" | "bool" | "string" |
                "bytes" | "time" | "duration" | "any"
RESERVED      = BOOL | TYPES | "type" | "defined" | "size"

Types

The template feature has abstract types for operations.

Template Type	Description	Go Type
int	64-bit signed integers	int, int8, int16, int32, int64
uint	64-bit unsigned integers	uint, uint8, uint16, uint32, uint64
float	IEEE-754 64-bit floating-point numbers	float32, float64
bool	booleans	bool
string	UTF-8 strings	string
bytes	byte sequence	[]byte
time	time with nanosecond precision	time.Time
duration	amount of time	time.Duration
any	other all Go types	any

Type Conversions

The template feature provides functions to convert types.

Function	Type	Description
int	(*int) -> int	type conversion (returns an error if arg is nil)
	(uint) -> int	type conversion (returns an error if result is out of range)
	(float) -> int	type conversion (rounds toward zero, returns an error if result is out of range)
	(string) -> int	type conversion (returns an error if arg in invalid int string)
	(duration) -> int	type conversion
uint	(int) -> uint	type conversion (returns an error if result is out of range)
	(*uint) -> uint	type conversion (returns an error if arg is nil)
	(float) -> uint	type conversion (rounds toward zero, returns an error if result is out of range)
	(string) -> uint	type conversion (returns an error if arg in invalid uint string)
float	(int) -> float	type conversion
	(uint) -> float	type conversion
	(*float) -> float	type conversion (returns an error if arg is nil)
	(string) -> float	type conversion (returns an error if arg in invalid float string)
bool	(*bool) -> bool	type conversion (returns an error if arg is nil)
string	(int) -> string	type conversion
	(uint) -> string	type conversion
	(float) -> string	type conversion
	(*string) -> string	type conversion (returns an error if arg is nil)
	(bytes) -> string	type conversion (returns an error if arg contains invalid UTF-8 encoded characters)
	(time) -> string	convert to string according to RFC3339 format
	(duration) -> string	convert to string according to time.Duration.String format
bytes	(string) -> bytes	type conversion
bytes	(*bytes) -> bytes	type conversion (returns an error if arg is nil)
time	(string) -> time	parse RFC3339 format string as time
time	(*time) -> time	type conversion (returns an error if arg is nil)
duration	(int) -> duration	type conversion
	(string) -> duration	parse string as duration by time.ParseDuration
	(*duration) -> duration	type conversion (returns an error if arg is nil)

Operators

Operator	Type	Description
! _	(bool) -> bool	logical not
- _	(int) -> int	negation
	(float) -> float	negation
	(duration) -> duration	negation
_ + _	(int, int) -> int	arithmetic
	(uint, uint) -> uint	arithmetic
	(float, float) -> float	arithmetic
	(string, string) -> string	concatenation
	(bytes, bytes) -> bytes	concatenation
	(time, duration) -> time	arithmetic
	(duration, time) -> time	arithmetic
	(duration, duration) -> duration	arithmetic
_ - _	(int, int) -> int	arithmetic
	(uint, uint) -> uint	arithmetic
	(float, float) -> float	arithmetic
	(time, time) -> duration	arithmetic
	(time, duration) -> time	arithmetic
	(duration, duration) -> duration	arithmetic
_ * _	(int, int) -> int	arithmetic
	(uint, uint) -> uint	arithmetic
	(float, float) -> float	arithmetic
_ / _	(int, int) -> int	arithmetic
	(uint, uint) -> uint	arithmetic
	(float, float) -> float	arithmetic
_ % _	(int, int) -> int	arithmetic
_ % _	(uint, uint) -> uint	arithmetic
_ == _	(A, A) -> bool	equality
_ != _	(A, A) -> bool	inequality
_ < _	(int, int) -> bool	ordering
	(uint, uint) -> bool	ordering
	(float, float) -> bool	ordering
	(string, string) -> bool	ordering
	(bytes, bytes) -> bool	ordering
	(time, time) -> bool	ordering
	(duration, duration) -> bool	ordering
_ <= _	(int, int) -> bool	ordering
	(uint, uint) -> bool	ordering
	(float, float) -> bool	ordering
	(string, string) -> bool	ordering
	(bytes, bytes) -> bool	ordering
	(time, time) -> bool	ordering
	(duration, duration) -> bool	ordering
_ > _	(int, int) -> bool	ordering
	(uint, uint) -> bool	ordering
	(float, float) -> bool	ordering
	(string, string) -> bool	ordering
	(bytes, bytes) -> bool	ordering
	(time, time) -> bool	ordering
	(duration, duration) -> bool	ordering
_ >= _	(int, int) -> bool	ordering
	(uint, uint) -> bool	ordering
	(float, float) -> bool	ordering
	(string, string) -> bool	ordering
	(bytes, bytes) -> bool	ordering
	(time, time) -> bool	ordering
	(duration, duration) -> bool	ordering
_ && _	(bool, bool) -> bool	logical and
_ \|\| _	(bool, bool) -> bool	logical or
_ ?? _	(A, B) -> A \| B	nullish coalescing operator. The operator returns the left-hand side if it is defined and not null, otherwise the operator returns the right-hand side.
_ ? _ : _	(bool, A, A) -> A	ternary conditional operator

Predefined Variables

Variables	Description
ctx	scenarigo context
vars	user-defined variables
secrets	user-defined secrets
plugins	loaded plugins
env	environment variables
request	request data
response	response data
assert	assert functions
steps	results of steps

Predefined Functions

Function	Description	Example
type	returns the abstract type of expression in string	`type(0) == "int"`
defined	tells whether a variable is defined or not	`defined(a) ? a : b`
size	returns the string length	`size("foo")`
	returns the bytes length	`size(bytes("foo"))`
	returns the number of list elements	`size(items)`
	returns the number of map elements	`size(index)`

Plugin

Scenarigo has a plugin mechanism that enables you to add new functionalities you need by writing Go code. This feature is based on Go's standard library plugin, which has the following limitations.

Supported on Linux, FreeBSD, and macOS only.
All plugins (and installed scenarigo command) must be built with the same version of the Go compiler and dependent packages.

Scenarigo loads built plugins at runtime and accesses any exported variable or function via template string.

See the official document for details of the plugin package.

How to write plugins

A Go plugin is a main package with exported variables and functions.

package main

import "time"

var Layout = "2006-01-02"

func Today() string {
	return time.Now().Format(Layout)
}

You can use the variables and functions via template strings like below in your test scenarios.

{{plugins.date.Layout}} => "2006-01-02"
{{plugins.date.Today()}} => "2022-02-22"

Scenarigo allows functions to return a value or a value and an error. The template string execution will fail if the function returns a non-nil error.

package main

import "time"

var Layout = "2006-01-02"

func TodayIn(s string) (string, error) {
	loc, err := time.LoadLocation(s)
	if err != nil {
		return "", err
	}
	return time.Now().In(loc).Format(Layout), nil
}

{{plugins.date.TodayIn("UTC")}} => "2022-02-22"
{{plugins.date.TodayIn("INVALID")}} => failed to execute: {{plugins.date.TodayIn("INVALID")}}: unknown time zone INVALID

How to build plugins

Go plugin can be built with go build -buildmode=plugin, but we recommend you use scenarigo plugin build instead. The wrapper command requires go command installed in your machine. Scenarigo always builds plugins with the same go version that is used to build its own. Because of that, scenarigo add toolchain directive to the go.mod files of plugins.

Scenarigo builds plugins according to the configuration.

schemaVersion: config/v1

scenarios:
- scenarios

pluginDirectory: ./gen  # Specify the root directory of plugins.
plugins:                # Specify configurations to build plugins.
  date.so:              # Map keys specify plugin output file path from the root directory of plugins.
    src: ./plugins/date # Specify the source file, directory, or "go gettable" module path of the plugin.

.
├── plugins
│   └── date
│       └── main.go
├── scenarigo.yaml
└── scenarios
    └── echo.yaml

In this case, the plugin will be built and written to date.so.

$ scenarigo plugin build

.
├── gen
│   └── date.so     # built plugin
├── plugins
│   └── date
│       ├── go.mod  # generated automatically if not exists
│       └── main.go
├── scenarigo.yaml
└── scenarios
    └── echo.yaml

Scenarigo checks the dependent packages of each plugin before building. If the plugins depend on a different version of the same package, Scenarigo overrides go.mod files by the maximum version to avoid the build error.

Now you can use the plugin in test scenarios.

title: echo
plugins:
  date: date.so # relative path from "pluginDirectory"
steps:
- title: POST /echo
  protocol: http
  request:
    method: POST
    url: 'http://{{env.ECHO_ADDR}}/echo'
    body:
      message: '{{plugins.date.Today()}}'
  expect:
    code: 200

Scenarigo can download source codes from remote repositories and build it with go get-able module query.

plugins:
  uuid.so:
    src: github.com/zoncoen-sample/scenarigo-plugins/uuid@latest

Advanced features

Setup Funciton

plugin.RegisterSetup registers a setup function that will be called before running scenario tests once only. If the registered function returns a non-nil function as a second returned value, it will be executed after finished all tests.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/scenarigo/scenarigo/plugin"

	secretmanager "cloud.google.com/go/secretmanager/apiv1"
	secretmanagerpb "google.golang.org/genproto/googleapis/cloud/secretmanager/v1"
)

const (
	projectName = "foo"
)

func init() {
	plugin.RegisterSetup(setupClient)
}

var client *secretmanager.Client

func setupClient(ctx *plugin.Context) (*plugin.Context, func(*plugin.Context)) {
	var err error
	client, err = secretmanager.NewClient(context.Background())
	if err != nil {
		ctx.Reporter().Fatalf("failed to create secretmanager client: %v", err)
	}
	return ctx, func(ctx *plugin.Context) {
		client.Close()
	}
}

func GetSecretString(name string) (string, error) {
	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()
	resp, err := client.AccessSecretVersion(ctx, &secretmanagerpb.AccessSecretVersionRequest{
		Name: fmt.Sprintf("projects/%s/secrets/%s/versions/latest", projectName, name),
	})
	if err != nil {
		return "", fmt.Errorf("failed to get secret: %v", err)
	}
	return string(resp.Payload.Data), nil
}

plugins:
  setup.so:
    src: ./plugins/date # call "setupClient" before running test scenarios

Similarly, plugin.RegisterSetupEachScenario can register a setup function. The registered function will be called before each test scenario that uses the plugin.

package main

import (
	"github.com/scenarigo/scenarigo/plugin"

	"github.com/google/uuid"
)

func init() {
	plugin.RegisterSetupEachScenario(setRunID)
}

func setRunID(ctx *plugin.Context) (*plugin.Context, func(*plugin.Context)) {
	return ctx.WithVars(map[string]string{
		"runId": uuid.NewString(),
	}), nil
}

title: echo
plugins:
  setup: setup.so # call "setRunID" before running this test scenario
steps:
- title: POST /echo
  protocol: http
  request:
    method: POST
    url: 'http://{{env.ECHO_ADDR}}/echo'
    header:
      Run-Id: '{{vars.runId}}'
    body:
      message: hello
  expect:
    code: 200

Custom Step Function

Generally, a step represents sending a request in Scenarigo. However, you can use a Go's function as a step with the plugin.

package main

import (
	"github.com/scenarigo/scenarigo/plugin"
	"github.com/scenarigo/scenarigo/schema"
)

var Nop = plugin.StepFunc(func(ctx *plugin.Context, step *schema.Step) *plugin.Context {
	ctx.Reporter().Log("nop step")
	return ctx
})

title: nop
plugins:
  step: step.so
steps:
- title: nop step
  ref: '{{plugins.step.Nop}}'

Left Arrow Function (a function takes arguments in YAML)

Scenarigo enables you to define a function that takes arguments in YAML for readability. It is called the "Left Arrow Function" since its syntax {{funcName <-}}.

package main

import (
	"errors"
	"fmt"

	"github.com/scenarigo/scenarigo/plugin"
)

var CoolFunc plugin.LeftArrowFunc = &fn{}

type fn struct{}

type arg struct {
	Foo string `yaml:"foo"`
	Bar string `yaml:"bar"`
	Baz string `yaml:"baz"`
}

func (_ *fn) UnmarshalArg(unmarshal func(interface{}) error) (interface{}, error) {
	var a arg
	if err := unmarshal(&a); err != nil {
		return nil, err
	}
	return &a, nil
}

func (_ *fn) Exec(in interface{}) (interface{}, error) {
	a, ok := in.(*arg)
	if !ok {
		return nil, errors.New("arg must be a arg")
	}
	return fmt.Sprintf("foo: %s, bar: %s, baz: %s", a.Foo, a.Bar, a.Baz), nil
}

title: echo
plugins:
  cool: cool.so
steps:
- title: POST /echo
  protocol: http
  request:
    method: POST
    url: 'http://{{env.ECHO_ADDR}}/echo'
    body:
      message:
        '{{plugins.cool.CoolFunc <-}}':
          foo: 1
          bar: 2
          baz: 3
  expect:
    code: 200

ytt Integration (templating and overlays)

Scenarigo integrates ytt to provide flexible templating and overlay features for test scenarios. You can use this experimental feature by enabling it in scenarigo.yaml.

input:
  yaml:
    ytt:
      enabled: true

Single File

All test scenarios are processed as ytt templates when the feature is enabled. For example, the following simple test scenario will set "hello" to the message field.

#@ msg = "hello"
schemaVersion: scenario/v1
title: echo
steps:
- title: POST /echo
  protocol: http
  request:
    method: POST
    url: http://example.com/echo
    header:
      Content-Type: application/json
    body:
      message: #@ msg
  expect:
    body:
      message: "{{request.body.message}}"

You can check the test scenarios generated by ytt integration with scenarigo dump sub-command.

$ scenarigo dump ./scenarios/simple.yaml
schemaVersion: scenario/v1
title: echo
steps:
- title: POST /echo
  protocol: http
  request:
    method: POST
    url: http://example.com/echo
    header:
      Content-Type: application/json
    body:
      message: hello
  expect:
    body:
      message: "{{request.body.message}}"

Multiple File

ytt/v1 schema type file allows giving multiple ytt files.

# This configuration equals the following command.
# ytt -f template.ytt.yaml -f values.ytt.yaml
schemaVersion: ytt/v1
files:
- template.ytt.yaml
- values.ytt.yaml

#@ load("@ytt:data", "data")
#@ for params in data.values:
---
schemaVersion: scenario/v1
plugins:
  plugin: plugin.so
title: #@ params.title
vars: #@ params.vars
steps:
- title: #@ "{} /{}".format(params.request.method, params.request.path)
  protocol: http
  request:
    method: #@ params.request.method
    url: #@ "http://example.com/{}".format(params.request.path)
    header:
      Content-Type: application/json
    body:
      message: "{{vars.message}}"
  expect: #@ params.expect
#@ end

#@data/values
---
- title: success
  vars:
    message: hello
  request:
    method: POST
    path: echo
  expect:
    code: OK
    body:
      message: "{{request.body.message}}"

- title: invalid method
  vars:
    message: hello
  request:
    method: GET
    path: echo
  expect:
    code: Method Not Allowed

- title: invalid path
  vars:
    message: hello
  request:
    method: POST
    path: invalid
  expect:
    code: Not Found

This example will run three test scenarios.

$ scenarigo dump ./scenarios/scenarios.yaml
schemaVersion: scenario/v1
title: success
plugins:
  plugin: plugin.so
vars:
  message: hello
steps:
- title: POST /echo
  protocol: http
  request:
    method: POST
    url: http://example.com/echo
    header:
      Content-Type: application/json
    body:
      message: "{{vars.message}}"
  expect:
    code: OK
    body:
      message: "{{request.body.message}}"
---
schemaVersion: scenario/v1
title: invalid method
plugins:
  plugin: plugin.so
vars:
  message: hello
steps:
- title: GET /echo
  protocol: http
  request:
    method: GET
    url: http://example.com/echo
    header:
      Content-Type: application/json
    body:
      message: "{{vars.message}}"
  expect:
    code: Method Not Allowed
---
schemaVersion: scenario/v1
title: invalid path
plugins:
  plugin: plugin.so
vars:
  message: hello
steps:
- title: POST /invalid
  protocol: http
  request:
    method: POST
    url: http://example.com/invalid
    header:
      Content-Type: application/json
    body:
      message: "{{vars.message}}"
  expect:
    code: Not Found

⚠️ You should exclude ytt files specified from ytt/v1 type test scenarios by setting regular expressions to excludes field.

input:
  excludes:
  - \.ytt\.yaml$
  yaml:
    ytt:
      enabled: true

Default ytt Files

The files set to defaultFiles field will be used to generate all test scenarios.

input:
  yaml:
    ytt:
      enabled: true
      defaultFiles:
      - default.yaml

#@ load("@ytt:overlay", "overlay")
#@overlay/match by=overlay.map_key("schemaVersion"), expects="0+"
---
schemaVersion: scenario/v1
steps:
#@overlay/match by=overlay.all, expects="0+"
-
  #@overlay/match when=0
  timeout: 30s

This example set 30 sec. as the default timeout for all test scenarios.

$ scenarigo dump ./scenarios/simple.yaml
schemaVersion: scenario/v1
title: echo
plugins:
  plugin: plugin.so
steps:
- title: POST /echo
  protocol: http
  request:
    method: POST
    url: http://{{plugins.plugin.ServerAddr}}/echo
    header:
      Content-Type: application/json
    body:
      message: hello
  expect:
    body:
      message: "{{request.body.message}}"
  timeout: 30s

Examples

This repository includes practical examples demonstrating various Scenarigo features. You can find them in the examples/ directory:

Running Examples

To run any example:

# Navigate to the example directory
$ cd examples/grpc

# Build plugins
$ scenarigo plugin build

# Run the tests
$ scenarigo run

Each example directory contains:

scenarigo.yaml - Configuration file
scenarios/ - Test scenario files
README.md (in some examples) - Detailed explanation

Name		Name	Last commit message	Last commit date
Latest commit History 1,386 Commits
.github		.github
assert		assert
cmd/scenarigo		cmd/scenarigo
color		color
context		context
errors		errors
examples		examples
internal		internal
logger		logger
mock		mock
plugin		plugin
protocol		protocol
reporter		reporter
schema		schema
scripts		scripts
template		template
test/e2e		test/e2e
testdata		testdata
version		version
.codebeatignore		.codebeatignore
.gitignore		.gitignore
.golangci.yml		.golangci.yml
.tagpr		.tagpr
CHANGELOG.md		CHANGELOG.md
CREDITS		CREDITS
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
codecov.yml		codecov.yml
doc.go		doc.go
go.mod		go.mod
go.sum		go.sum
mod.go		mod.go
runner.go		runner.go
runner_test.go		runner_test.go
scenario.go		scenario.go
scenario_test.go		scenario_test.go
setup.go		setup.go
setup_test.go		setup_test.go
step.go		step.go

License

scenarigo/scenarigo

Folders and files

Latest commit

History

Repository files navigation

Overview

Features

Quick Start

Installation

Your First Test

Running Tests

Installation (Detailed)

go install command (recommend)

from release page

Setup

Usage

How to write test scenarios

HTTP Testing

Send HTTP requests

Check HTTP responses

Exact Matching

Template Expressions

Assertion Functions

Adding Custom Assertions via Plugins

Combining Validation Methods

Custom Client

gRPC Testing

Send gRPC requests

Using Protocol Buffers files

Using gRPC reflection

Check gRPC responses

Exact Matching

Template Expressions

Assertion Functions

Custom Client

gRPC-specific features

Status codes

Authentication

Common Features

Variables

Secrets

Timeout/Retry

Step-level Retry

Scenario-level Retry

Using conditions to control step execution

Template String

Overview

Syntax

Types

Type Conversions

Operators

Predefined Variables

Predefined Functions

Plugin

How to write plugins

How to build plugins

Advanced features

Setup Funciton

Custom Step Function

Left Arrow Function (a function takes arguments in YAML)

ytt Integration (templating and overlays)

Single File

Multiple File

Default ytt Files

Examples

Running Examples

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 69

Packages 0

Uh oh!

Contributors 12

Uh oh!

Languages

Packages