This repository contains YAML test procedure definitions for use with the CSIP-AUS Client/Server Test Harnesses.

This repository also provides Python dataclasses to make it easier for Python code to work with these definitions. In addition, there are also helper functions for creating valid instances of these dataclasses directly from the YAML test procedure definition files.

Client Test Procedures can be found in the cactus_test_definitions/client/procedures/ directory

Server Test Procedures can be found in the cactus_test_definitions/server/procedures/ directory

This repository also contains a small number of tests that verify that test definitions can be sucessfully converted to their equivalent python dataclasses.

First install the development and testing dependencies with,

python -m pip install --editable .[dev,test]

Once installed, run the tests with,

pytest

See cactus_test_definitions/server/README.md

A TestProcedure is a YAML configuration that describes how a CSIP-Aus Client test case should be implemented by a server / compliance body. It's designed to be human readable but interpretable by a test harness for administering a test. At its most basic level, a TestProcedure is a series of a "events" that a client must trigger in sequence in order to demonstrate compliance.

Here is a minimalist definition of a test case

Description: Minimal Test Case Definition
Category: Explaining Schemas
Classes:
  - Descriptive Test Tag 1
  - Descriptive Test Tag 2
Preconditions: 
    # See definitions below for more info
    actions:
    checks:
Criteria:
    # See definitions below for more info
    checks:
Steps:
    # See definitions below for more info
    STEP_NAME:

For how to actually interpret and "run" these test cases against a CSIP-Aus Server implementation, please see cactus-runner for a reference implementation.

The most basic building block of a TestProcedure is a Step. Each Step will always define an Event which dictates some form of trigger based on client behavior (eg sending a particular request). When an Event is triggered, each of it's Action elements will fire which will in turn enable/disable additional Steps with new events and so on until the TestProcedure is complete. Event's can also define a set of Check objects (see doco below) that can restrict an Event from triggering if any Check is returning False/fail.

When a TestProcedure first starts, normally only a single Step will be active but more can be enabled/disabled in response to client behaviour.

Step Schema:

Steps:
  DESCRIPTIVE_TITLE_OF_STEP: # This is used to reference this step in other parts of this test procedure
    event: # 
      type:  # string identifier of the event type - see table below
      parameters: # Any parameters to modify the default behaviour of the event - see table below
      checks: # A list of Check definitions that will need to be true for this event to trigger - see section on Checks below 
    actions:
       # See action schema in the Action section below

These are the currently defined Event types that a Step can define

NOTE: The endpoint parameter used by the various -request-received events supports a rudimentary * wildcard. This will match a single "component" of the path (portion deliminated by / characters).

eg:

• /edev/*/derp/456/derc will match /edev/123/derp/456/derc
• /edev/* will NOT match /edev/123/derp/456/derc (the * will only match the 123 portion - not EVERYTHING)

These are the currently defined Action elements that can be included in a test. Action's can trigger at the beginning of a test (as a precondition) or whenever a Step's Event is triggered.

Action's are defined with the following schema, always as a list under an element called actions

actions: 
  - type: # string identifier of the action type - see table below
    parameters: # Any parameters to supply to the executed Action - see table below

This is an example of two Action elements that will enable a different Step and create a DERControl:

actions: 
  - type: enable-steps
    parameters:
      steps: NAME_OF_STEP_TO_ENABLE
  - type: create-der-control
    parameters:
      start: $now
      duration_seconds: 300
      opModExpLimW: 2000

A Check is a boolean test of the state of the utility server. They are typically defined as a success/failure condition to be run at the end of a TestProcedure.

Check's are defined with the following schema, always as a list under an element called checks

checks: 
  - type: # string identifier of the check type - see table below
    parameters: # Any parameters to supply to the executed Check - see table below

This is an example of two Check elements that will check that all steps are marked as complete and that there has been a DERStatus submitted with specific values:

checks: 
  - type: all-steps-complete
    parameters: {}
  - type: der-status-contents
    parameters:
      genConnectStatus: 1

The following are csipaus.org/ns/v1.3-beta/storage extension specific checks implemented

doeModesEnabled_set modesEnabled_set doeModesSupported_set and modesSupported_set all expect a hexbinary string to be supplied, which contains the hi assertion bits to be equal to one e.g. doeModesEnabled_set: "03" would test to ensure that at least bits 0 and 1 are set hi (==1) for the given DERSetting.doeModesEnabled, ignoring all others The corresponding _unset performs the inverse operation such that every bit set to 1 in the mask is expected to correspond to a zero in the corresponding value e.g. doeModesEnabled_unset: "03" would test to ensure that at least bits 0 and 1 are set lo (==0) for the given DERSetting.doeModesEnabled, ignoring all others

If a common bit is set between a _set and _unset corresponding pair of parameters, the check will always fail.

Any parameter element expects a series of name/value pairs to pass to the "parent" Action, Check or Event. For example:

parameters:
    number_param: 123
    text_param: Text Content
    date_param: 2020-01-02 03:04:05Z

But placeholder variables may also be used to reference things that aren't known until the test is underway. For example, the following would instead set number_param to the current DERSetting.setMaxW supplied by the client while date_param would be set to the moment in time that the Action, Check or Event is being evaluated.

parameters:
    number_param: $setMaxW
    text_param: Text Content
    date_param: $now

The following are all the NamedVariable types currently implemented

The following are csipaus.org/ns/v1.3-beta/storage extension specific NamedVariable types implemented

Placeholder variables can also be used in some rudimentary expressions to make variations on the returned value. For example:

parameters:
    number_param: $(setMaxW / 2)
    text_param: Text Content
    date_param: $(now - '5 mins')
    setMaxW: $(this < rtgMaxW)
    setMaxVA: $(this >= rtgMaxWh)

Would resolve similar to above, but instead number_param will be half of the last supplied value to DERSetting.setMaxW and date_param will be set to 5 minutes prior to now. setMaxW param will return boolean result DERSettings.setMaxW is less than DERCapability.rtgMaxW. setMaxVA param will return boolean result DERSettings.setMaxVA is greather than or equal to DERCapability.rtgMaxWh.