JSON object for displaying unit test output

Run tests using any unit testing framework of your choice and output the results in a structure unit test format.

We now have an easier way of parsing unit test output from frameworks that aren't supported at the Gold or Silver level. Read more about that here:

When creating a unit test via the Custom Test block, you need to write the output of the test to a JSON object so that the Custom Test block generates the output in this structured way.

How to render custom unit tests:

1. Create a grading script that runs your unit tests and writes the output to a separate file.

2. Using a grading script or by coding directly in the editor of your test block, parse the test output file into the following categories:

3. Organize these into a JSON object and write the object to the 3 file. Here is code snippet to illustrate how to write the JSON object for generating unit test output to the 3 file using Python:

import json

with open(3,'w') as f:
    f.write(
        json.dumps({
            'tag': 'unit-test',
            'results': [{
                'id': 'Check whether the multiply function returns the expected result',
                'name': 'JUnit Jupiter',
                'testCases': [
                    {
                        'name': 'Check whether the addition function returns the expected result.',
                        'weight': 1,
                        'status': 'success',
                        'stdout': 'This is printed to std output',
                        'stderr': 'This is printed to std err'
                    },
                    {
                        'name': 'Check whether the multiply function returns the expected result',
                        'weight': 2,
                        'status': 'failure',
                        'reason': 'expected: <6.0> but was: <5.0>',
                        'stdout': 'This is printed to std output',
                        'stderr': 'This is printed to std err'
                    }
                ]
            }]
        })
    )
    
    

Unit test output JSON object format

• Tag (string) - The tag must always be "unit-test".

• Results (list) - A list of objects containing each test suite with the following information:

  • ID (string) - This should be a unique string to distinguish between test suites. It is for internal use and doesn't impact the presentation of the tests.

  • Name (string) - This is the name of the test suite.

  • Test cases (list) - A list of objects containing each test case with the following information:

    • Name (string) - The name of the test case

    • Weight (int) - The weight of the test case

    • Status (string) - This should be either "success" or "failure"

    • Reason (string) - Only applicable in the case of a "failure". The traceback or reason for the failure of the test.

The structured unit test JSON object will not render points. These still need to be calculated and written to a "points" JSON object. Learn more about how to do that in our Create a Custom Test guide.

pageCreate a Custom Test

This output format is supported by all AutoTest v2 test blocks including the Script block, the IO Test block and the Custom Test block.