Unit Test

Unit Tests work out of the box in AutoTest using the unit test step, simply upload your testing scripts and add it to your category.

The Unit Test provides integration with unit testing frameworks that support JUnit XML output. The only input for this type of test is a command to be run, which must produce a JUnit XML file at the location stored in the $CG_JUNIT_XML_LOCATION environment variable. Students will then get a clear overview of all the tests that have been run, which ones failed, and the output that was produced in the case of failure.

The resulting score of this step is the fraction of successful tests divided by the total number of tests that were run, multiplied by the weight of the test.

All built in scripts will automatically produce the correct files in the right locations.

Use cases

Unit Tests are useful if you want to run many tests but condense their output into a single entry in the test category. They are also useful when you already have a unit testing setup, and do not want to translate them all into individual AutoTest tests. Naturally, unit tests are a great way to assess individual functions and methods of larger programming assignments.

Compatible scripts

We provide wrapper scripts for a growing list of frameworks that handle everything for you, these are installed by default on AutoTest. If your unit testing framework is not in the list below, it is possible to manually create a small wrapper script for this to output the results in the correct location.

Want to know if your preferred testing framework is supported, or does a script not work as you need it to? Please send an email to [email protected] so we can discuss the possibilities!

Supported frameworks

Framework

Language(s)

Script name

Check

C/C++

cg-check

xUnit

C#

cg-xunit

Tasty (Quickcheck)

Haskell

cg-quickcheck

Junit 4

Java

cg-junit4

Junit 5

Java

cg-junit5

Jest

Javascript

cg-jest

Mocha

Javascript

cg-mocha

MOxUnit

Matlab/Octave

cg-moxunit

Nbgrader

Jupyter Notebook

cg-nbgrader

PHPUnit

PHP

cg-phpunit

Pytest

Python 2+3

cg-pytest

Wrapper script usage

All wrapper scripts share a similar interface and procedure for running tests, which we describe below, replace SCRIPT for the unit test script you want to use from the list above.

  • Installing: install the script in the global setup script to run by calling cg-SCRIPT compile.

  • Compiling: For compiled languages it can be tricky to get the compiler to find all libraries and other dependencies needed to compile the students’ code and your tests. For these languages the wrapper scripts provide a compile command that configures the compiler to be able to find the required libraries. The compile command takes file names as arguments and compiles those files. Run the command cg-SCRIPT compile <file1> <file2> ...to compile the student code and test classes.

  • Running: The wrapper scripts run the unit tests with their run command.

    The scripts extract the output location from $CG_JUNIT_XML_LOCATION and then unset it before running the tests. The testing framework is then configured to output its results at that location and the tests are executed. Run the command cg-SCRIPT run <file1> <file2> ...to compile the student code and test classes.

The exact arguments to the run command of each script may differ between scripts. E.g. cg-junit5 is run via cg-junit5 run -- -c TestClass.

​