Testing the tests: Autograding Java unit tests

Learn how to use JaCoCo, a JUnit code coverage assessment tool, to automatically grade students JUnit unit tests.

This is a summary of a guide published on our Blog page, read the full guide here:

Prefer to watch a video tutorial? Watch our webinar on Autograding Java assignments with I/O tests, Unit tests, and much more:

Unit Test assessment metrics

There are multiple ways in which we can effectively assess unit tests, of which code coverage and mutation testing are the two most common ones. Both serve different purposes and differ in setup complexity. In this guide, we will only discuss Code Coverage testing which should be sufficient for most educational purposes.
  • Code Coverage is the most common metric to assess unit test quality. It very simply measures what percentage of source code is covered by the unit tests that you have written. Different metrics can be used to calculate this, for instance the percentage of subroutines or the percentage of instructions of the code that the tests cover.
  • Mutation Testing is a more advanced metric to assess unit test quality, that goes beyond simple code coverage. During mutation testing, a tool makes minor changes to your code (known as mutations), that should break your code. It then checks whether a mutation makes your unit tests fail (this is what we want if our unit test is correct and complete) or not (meaning that our unit test was not correct or incomplete).
Using CodeGrade’s continuous feedback we can very effectively motivate our students with instant feedback to go for a 100% code coverage score. However, for more advanced software engineering courses, you may want to consider using Mutation Testing, which not only measures the number of lines we cover, but also how well these lines are actually covered by our tests. This metric can be somewhat off putting for beginning students, but very useful in more advanced courses.

Code Coverage Assessment for Java using JaCoCo

Setting up Code Coverage in CodeGrade’s autograder is very straightforward. The resulting coverage percentage can be simply converted to the points we give our students.
​JUnit, one of the most common unit testing frameworks for Java and supported out of the box by CodeGrade, works together very well with JaCoCo (Java Code Coverage). JaCoCo is a lightweight and flexible code coverage library that works well with tools like Ant and Maven, but also simply from the command line in CodeGrade AutoTest. Ant and Maven are supported in CodeGrade, but for this example we will stick to the CLI.

Setting up AutoTest

Creating the test steps

  • Run Unit test with JaCoCo This is done simply using the Unit Test step. We can use the following command to run JUnit5 tests with the JaCoCo agent (Keep in mind the test classes are selected using the -c flag, in this case TestReadSudoku) :
cg-junit5 run --java-args="-javaagent:$FIXTURES/jacocoagent.jar" -- -c TestReadSudoku
  • Set up the folder structure for JaCoCo In our example, we create three folders: src, tests and compiled. And moved all the student’s files to the correct folders:
    • all *.java files to src
    • all regular .class files to compiled
    • and compiled test classes, such asTest*.class, to tests.
    we can do this by creating a small bash script called (short for "configure environment"), uploading it as a fixture and the running it in a Run Program step:
mkdir src tests compiled
mv src
mv Board.class Cell.class SudokuSolver.class compiled
mv TestReadSudoku.class tests
  • Run JaCoCO We can generate the JaCoCo report in another Run Program step. This runs the JaCoCo CLI, calls the report function and uses the jacoco.exec file (generated in the Unit Test step) to generate this report. It then finds all files it needs in the folders that we created and finally generates the output XML in cov.xml:
java -jar $FIXTURES/jacococli.jar report jacoco.exec --classfiles compiled --sourcefiles src --xml cov.xml
  • Convert coverage score to test score In a subsequent Capture Points test, we can now very easily capture the code coverage by running a simple python script called, which we uploaded as a fixture:
import xml.etree.ElementTree as ET
tree = ET.parse('out.xml')
root = tree.getroot()
instructions = root[2].attrib
covered = int(instructions.get('covered'))
total = covered + int(instructions.get('missed'))
print(covered / total)
Full JaCoCo AutoTest category (including 2 optional steps)
Manually grading the code coverage report
JaCoCo provides the option to generate very useful HTML code coverage reports. These reports are not only very useful for manual grading by a teacher, but also interesting to open and investigate for students too. Next to automatically parsing the code coverage percentages, we can also save this code coverage report to the $AT_OUTPUT directory so that it is visible in the Code Viewer for grading.
In JaCoCo, we can use the following command (very similar to the one discussed above) to generate an HTML report and save it to the $AT_OUTPUT directory in CodeGrade:
java -jar $FIXTURES/jacococli.jar report jacoco.exec --classfiles compiled --sourcefiles src --html $AT_OUTPUT
JaCoCo HTML code coverage report rendered in the CodeGrade interface
In this guide we discuss the best way to autograde student JUnit unit tests for Java assignments using Code Coverage tests. We have covered the theory behind testing unit tests in Python in another guide: