Writing Unit Test wrapper scripts

CodeGrade offers an ever expanding list of unit test frameworks that are built in to AutoTest. In this guide we explain how you can add your own by writing a very simple wrapper script for it.

CodeGrade's Unit Test step can be used to automatically assess code using your unit test scripts. It will show each individual test to the students in a very insightful way. For most use cases, our built in unit test frameworks will be more than sufficient. Some advanced assignments however ask for specific frameworks. How to integrate these tools with CodeGrade by writing a custom wrapper script will be explained in this guide. Want to learn more about our Unit Test step? Read the guide below!

pageCreating Unit Tests

In this guide we will recreate a simplified version of the cg-junit5 wrapper script to run unit tests on Java code with JUnit 5. We will write the script in Python because it's a bit more flexible and powerful than a shell script, and also because AutoTest runners have the cg_at_utils library installed automatically, which is a Python package that provides utility functions for writing wrapper scripts like these.

The skeleton of a wrapper script

Unit test wrapper scripts generally consist of three stages:

• install: Installs any dependencies of your wrapper script that is not available by default on CodeGrade's AutoTest runners. This is usually executed in the global setup script, and may not be necessary at all if all dependencies are already installed by default.

• compile: Perform compilation if necessary. May be skipped if the programming language is not a compiled one, e.g. a scripting language like Python.

• run: Run the tests and generate the test output. The output is a JUnit XML file (note: there does not exist a formal specification of the JUnit XML format, but the specification found in this link is supported by CodeGrade, except for the <properties> tag). This file must be written to the randomized location specified in the $CG_JUNIT_XML_LOCATION environment variable (available in AutoTest, just like e.g. the $FIXTURES environment variable).

The cg_at_utils package provides a make_wrapper function that takes three functions — one for each of the stages described above — to help reduce the amount of boilerplate code you have to write. This uses the typer library to help defining the command line arguments to your script, which are automatically inferred from the types of the arguments of the functions you define.

We use Python for our script so that we can use the CodeGrade library called cg_at_utils . This library is available automatically for all Python scripts run in CodeGrade AutoTest and offers many functions to make your life easier.

#!/usr/bin/env python3

import os
import typing as t
import tempfile

import cg_at_utils.utils as utils
import typer


def install() -> t.NoReturn:
    """Install dependencies"""
    # TODO


def compile() -> t.NoReturn:
    """Compile student code"""
    # TODO


def run() -> t.NoReturn:
    """Run the unit tests"""
    # TODO


app = utils.cli.make_wrapper(
    # The name of the script and a helpful message that appear in the `--help`
    # output.
    name='cg-junit5',  
    help='A wrapper script for running JUnit5 tests',

    install=install,   # The function for the `install` stage.
    compile=compile,   # The function for the `compile` stage.
    run=run,           # The function for the `run` stage.
)

if __name__ == '__main__':
    app()

The install stage

This is the step that is run in the Global Setup Script step and can be used to install the necessary software to AutoTest. In our case, to run the JUnit 5 tests, we will need a Java runtime and SDK and the JUnit 5 .jar file somewhere on the machine. We will also need a program to combine the multiple .xml files that are generated by JUnit 5 into one. We will use the junitparser Python library for this.

All these packages we need are already installed in the base image, but we will write the install stage anyway for instructional purposes.

We can install the Java runtime with apt, but the versions of JUnit 5 and the junitparser library on apt are quite outdated, so we will fetch the JUnit 5 .jar of version 1.6.2 from the Maven repository and save it in $FIXTURES/junit5.jar and we will use pip to install junitparser. We will use utility functions provided by the cg_at_utils package to simplify these steps!

def install() -> t.NoReturn:
    """Install dependencies"""

    # Install packages with `apt`. This will quit the wrapper with an error code
    # if an error occurred.
    utils.pkg.apt('default-jdk', 'default-jre')

    # Download the JUnit 5 .jar to /opt/my-junit5/junit5.jar.
    utils.run_cmds(
        ['sudo', 'mkdir', '/opt/my-junit5'],
        ['sudo', 'wget', '-O', '/opt/my-junit5/junit5.jar', 'https://repo1.maven.org/maven2/org/junit/platform/junit-platform-console-standalone/1.6.2/junit-platform-console-standalone-1.6.2.jar'],
        check=True,
        exit_after=False,
    )

    # Install packages with `pip`. This will quit the wrapper with an error code
    # if an error occurred.
    utils.pkg.pip('junitparser==2.0.0')

The compile stage

The compile stage is fairly straightforward. We must update the CLASSPATH environment variable to include the student directory containing the code we want to compile, and the location to the junit5.jar we have downloaded in the install stage, so that the Java compiler can find the required classes and link them correctly.

The cg_at_utils library provides a function for obtaining the student directory, so we will use that. It also has a utility function for modifying PATH-like environment variables — that is, variables containing a list of paths to search, delimited by a :. Because we will need the same class path in the run stage we have extracted this to a separate function named set_classpath.

The above information is specific to the JUnit 5 unit testing framework and may differ for yours. Please consult the documentation of your unit testing framework to find out what is needed for a successful compilation of code and tests.

Finally, we invoke the Java compiler with a list of files to be compiled: javac File1.java File2.java .... We use the run_cmds function in cg_at_utils to run the command, and pass check=True to make it quit with an error code if the compilation failed.

def set_classpath() -> str:
    """Update the CLASSPATH environment variable to include the student
    directory with the code to compile and the junit5.jar.
    """
    return utils.env.add_to_path(
        'CLASSPATH',
        prepend=[utils.path.get_student_dir()],
        append=['/opt/my-junit5/junit5.jar'],
    )


def compile(
    files: t.List[str] = typer.Argument(..., help='The files to compile')
) -> t.NoReturn:
    """Compile student code"""

    set_classpath()
    utils.run_cmds(['javac', *files], check=True)

The run stage

The run stage is usually the most complicated one. Because there is no formal specification of the JUnit XML format, most unit test runners invent their own formats which all closely resemble each other, but are not quite the same most of the time. Because of this it can happen that you need to write your own converter for the output format of your test runner to a format that is recognized by CodeGrade. Luckily the XML produced by JUnit 5 is almost exactly what CodeGrade expects, so we just need to write a very simple converter function to remove the <properties> tags that are not supported:

def parse_junit_xml(tree):
    """Read the xml file, parse it, and remove any <properties> tags, as those
    are not yet supported by CodeGrade.
    """
    for _, el in tree:
        _, _, el.tag = el.tag.rpartition('}')
​
    root = tree.root
    for child in list(root):
        if child.tag != 'testcase':
            root.remove(child)
​
    return root

Then, in our run function we set the class path using the function we defined in the previous stage. We will also have to know the location where we must write the JUnit XML file containing the test results to. We can get that from the CG_JUNIT_XML_LOCATION environment variable, but the cg_at_utils library also provides a function junit_xml.get_location to do this for us. We pass as arguments delete=True to this function, which will also unset the CG_JUNIT_XML_LOCATION environment variable after it has read it. It is generally recommended to do this, so that students can not get its value from within their code.

Next we run JUnit 5 with java -jar /opt/my-junit5/junit5.jar ... and the arguments that were passed on the command line, e.g. -c MyTestClass. We also explicitly pass the -cp argument to JUnit 5 to ensure that it is consistent with the value we have set in the CLASSPATH environment variable and we create a temporary directory where it can write the test reports to. Finally, we pass exit_after=False because otherwise run_cmds will exit after running all the commands it was given, even if none of them produced an error.

The last step is to combine the resulting XML reports. JUnit 5 has multiple modes of running tests: jupiter mode which can run JUnit 5 tests, and vintage mode which can handle JUnit 4 tests. One report file is produced per mode, so we need to combine them into one file.

def run(
    runner_args: t.List[str] = typer.Argument(
        ...,
        help=(
            'Arguments to pass to the JUnit ConsoleLauncher'
            ' https://junit.org/junit5/docs/current/user-guide/#running-tests-console-launcher'
        ),
        metavar='JUNIT_ARGS',
    ),
) -> t.NoReturn:
    """Run the unit tests"""
​
    # We cannot import `junitparser` at the top of this file, because it is
    # not yet installed (this happens in the `install` stage).
    import junitparser.junitparser as jparser
​
    classpath = set_classpath()
    junit_xml = utils.junit_xml.get_location(delete=True)
​
    with tempfile.TemporaryDirectory() as xml_dir:
        utils.run_cmds(
            [
                'java',
                '-jar',
                '/opt/my-junit5/junit5.jar',
                f'-cp={classpath}',
                f'--reports-dir={xml_dir}',
                *runner_args,
            ],
            exit_after=False,
        )
​
        merged_report = jparser.JUnitXml()
​
        for report_file in [
            f'{xml_dir}/TEST-junit-jupiter.xml',
            f'{xml_dir}/TEST-junit-vintage.xml',
        ]:
            report = parse_junit_xml(
                jparser.etree.iterparse(report_file),
            )
            merged_report.append(jparser.JUnitXml.fromelem(report))
​
        merged_report.write(junit_xml)
​
        utils.exit(0)

Wrapping it up

We have now written a script that handles all the steps in running unit tests on submitted code: installing dependencies, compiling the code, and running the tests. We combine that to our final script and then have to run it in AutoTest.

The entire wrapper script

#!/usr/bin/env python3
​
import os
import typing as t
import tempfile
​
import cg_at_utils.utils as utils
import typer
​
​
def install() -> t.NoReturn:
    """Install dependencies"""
​
    # Install packages with `apt`. This will quit the wrapper with an error code
    # script if an error occurred.
    utils.pkg.apt('default-jdk', 'default-jre')
​
    # Download the JUnit 5 .jar to /opt/my-junit5/junit5.jar.
    utils.run_cmds(
        ['sudo', 'mkdir', '/opt/my-junit5'],
        ['sudo', 'wget', '-O', '/opt/my-junit5/junit5.jar', 'https://repo1.maven.org/maven2/org/junit/platform/junit-platform-console-standalone/1.6.2/junit-platform-console-standalone-1.6.2.jar'],
        check=True,
        exit_after=False,
    )
​
    # Install packages with `pip`. This will quit the wrapper with an error code
    # script if an error occurred.
    utils.pkg.pip('junitparser==2.0.0')
​
​
def set_classpath() -> str:
    """Update the CLASSPATH environment variable to include the student
    directory with the code to compile and the junit5.jar.
    """
    return utils.env.add_to_path(
        'CLASSPATH',
        prepend=[utils.path.get_student_dir()],
        append=['/opt/my-junit5/junit5.jar'],
    )
    
​
def compile(
    files: t.List[str] = typer.Argument(..., help='The files to compile')
) -> t.NoReturn:
    """Compile student code"""
​
    set_classpath()
    utils.run_cmds(['javac', *files], check=True)
​
​
def parse_junit_xml(tree):
    """Read the xml file, parse it, and remove any <properties> tags, as those
    are not yet supported by CodeGrade.
    """
    for _, el in tree:
        _, _, el.tag = el.tag.rpartition('}')
​
    root = tree.root
    for child in list(root):
        if child.tag != 'testcase':
            root.remove(child)
​
    return root
​
​
def run(
    runner_args: t.List[str] = typer.Argument(
        ...,
        help=(
            'Arguments to pass to the JUnit ConsoleLauncher'
            ' https://junit.org/junit5/docs/current/user-guide/#running-tests-console-launcher'
        ),
        metavar='JUNIT_ARGS',
    ),
) -> t.NoReturn:
    """Run the unit tests"""
​
    # We cannot import `junitparser` at the top of this file, because it is
    # not yet installed (this happens in the `install` stage).
    import junitparser.junitparser as jparser
​
    classpath = set_classpath()
    junit_xml = utils.junit_xml.get_location(delete=True)
​
    with tempfile.TemporaryDirectory() as xml_dir:
        utils.run_cmds(
            [
                'java',
                '-jar',
                '/opt/my-junit5/junit5.jar',
                f'-cp={classpath}',
                f'--reports-dir={xml_dir}',
                *runner_args,
            ],
            exit_after=False,
        )
​
        merged_report = jparser.JUnitXml()
​
        for report_file in [
            f'{xml_dir}/TEST-junit-jupiter.xml',
            f'{xml_dir}/TEST-junit-vintage.xml',
        ]:
            report = parse_junit_xml(
                jparser.etree.iterparse(report_file),
            )
            merged_report.append(jparser.JUnitXml.fromelem(report))
​
        merged_report.write(junit_xml)
​
        utils.exit(0)
​
​
app = utils.cli.make_wrapper(
    # The name of the script and a helpful message that appear in the `--help`
    # output.
    name='cg-junit5',  
    help='A wrapper script for running JUnit5 tests',
​
    install=install,   # The function for the `install` stage.
    compile=compile,   # The function for the `compile` stage.
    run=run,           # The function for the `run` stage.
)
​
if __name__ == '__main__':
    app()

The install stage

The installation of dependencies should typically only be done once, so we can put this also in the Global setup script field of our AutoTest configuration.

We must upload the wrapper script as a fixture in our test configuration (we have chosen the name my-junit5, but you can name it however you want) and make it executable with chmod to be able to run it in the AutoTest containers.

In the end our Global setup script looks like:

chmod +x $FIXTURES/my-junit5 && $FIXTURES/my-junit5 install

The compile stage

Next comes compilation. We can either compile the student code in the Per student setup script or in a separate test step (see also Where do I compile students’ code?). The following command will compile all .java files in the current directory. Put it in the Per student setup script field, or create a Run Program step in an AutoTest category and put it in the Program to test field.

We also copy MyTest.java in the fixtures directory to the current directory before compiling, as it contains the tests we want to run, which must also be compiled.

cp $FIXTURES/MyTest.java . && $FIXTURES/my-junit5 compile *.java

The run stage

Then finally we can run the tests with the command below. In our definition of the run function, we specified that my-junit5 run accepts a list of string arguments, all of which we pass directly to our invocation of the JUnit 5 test runner. In the example below these are -c and MyTest. The -- is ignored by the typer library, and tells it to stop parsing command line arguments, which is necessary here, because otherwise typer will try to parse the -c argument and throw an error because we have not defined a -c argument to our run function.

Create a Unit Test step in your category and put the following command in the Program to test field.

$FIXTURES/my-junit5 run -- -c MyTest

Advanced features

We have created a simple wrapper script that performs but the Unit Test step supports a few advanced features that we haven't covered, which we will discuss below.

Weighted tests

Sometimes a few tests in your test suite are more important than other ones. In this case you can set a weight per test case, which is done by adding a weight attribute to the <testcase> tags in the generated JUnit XML report. The weight must be a decimal number, and does not have to be present for all test cases. A weight of 1 will be assumed for tests without a weight attribute.