Evaluating files with the JSON evaluator

Evaluating files with the JSON evaluator#

Note

This tutorial does not configure any fetcher. It only uses the JSON evaluator to check if a manually uploaded file fulfills certain conditions.

Introduction#

With the JSON evaluator, you can evaluate JSON files based on their content.

Use-case#

For this example, we are using the following use case:

  • We manually upload a JSON file to our service and then the JSON evaluator evaluates the file based on its content.

  • We want to verify that the total test coverage in the JSON file is bigger than 70%.

Preparation#

Download resources#

Please go ahead and download the following files. They will be required in the subsequent steps.

Adjusting the config files#

The qg-config.yaml#

Feel free to have a look at the downloaded file first, to get an overview of it. For the JSON evaluator, there are only two environment variables that need to be set, additional optional variables can however be specified for more precise control.

 1metadata:
 2  version: v1
 3header:
 4  name: JSON
 5  version: 1.2.3
 6autopilots:
 7  json-evaluator-autopilot:
 8    run: |
 9      json-evaluator
10    env:
11      JSON_CONFIG_FILE: json-evaluator-config.yaml
12      JSON_INPUT_FILE: coverage-data.json
13chapters:
14  "5":
15    title: Verification / validation
16    requirements:
17      "5.1":
18        title: The test strategy was defined/adhered to.
19        text: |-
20          QG0: Test strategy was defined
21          QGR/QG4: test strategy was adhered to
22        checks:
23          "1":
24            title: The test strategy was defined/adhered to._check
25            manual:
26              status: PENDING
27              reason: Not answered
28      "5.2":
29        title: All unit tests were performed and their results are good.
30        text: |-
31          QG0: Unit test coverage is greater than 25%
32          QGR/QG4: Unit test coverage is greater than 85%
33        checks:
34          "1":
35            title: Check unit test coverage
36            automation:
37              autopilot: json-evaluator-autopilot

Lines 11 and 12 require the filename of the JSON formatted data to evaluate JSON_INPUT_FILE and the the path to the JSON evaluator’s configuration file JSON_CONFIG_FILE.

The JSON evaluator config file is described in the next section. We already provided a config file for this example, so you don’t have to adapt the config file for this example.

Note

Please refer to the Evaluator Background Information for more detailed information and a comprehensive list of all available environment variables for the JSON evaluator.

The JSON evaluator config#

Now that you have configured the qg-config.yaml, you can go ahead and create the JSON evaluator config.

This file should be created with the same filename as given by the JSON_CONFIG_FILE environment variable. Here, you can define the conditions to be checked for the evaluation of the input JSON data.

1checks:
2  - name: has_good_coverage
3    ref: $.totals.percent_covered
4    condition: ($) > 70

In line 3, you need to provide a reference to the object in the JSON file that you want to evaluate. In this case, we want to evaluate the percent_covered property of the totals object (see also Using JSONPath Syntax) .

In line 4, you need to provide the condition that has to be met for the percent_covered property. In this case, it checks whether the value of the percent_covered property is bigger than 70%. The result is GREEN in case the check passes and RED if it does not.

Running the example#

Now, you can run the example. Upload the json-evaluator-config.yaml, using the same name as stated in the JSON_CONFIG_FILE variable (line 15 in the qg-config.yaml) as well as the qg-config.yaml. Don’t forget to also upload the coverage-data.json.