☕Grading test coverage with JaCoCo

Writing good unit tests is a skill students need to develop, which means you need a way to grade the tests they write — not just whether they pass, but how thoroughly they exercise the code. JaCoCo is a Java code coverage library that measures what percentage of a reference implementation is exercised by a set of unit tests. The higher the coverage, the more of the reference code the student's tests are touching.

This guide walks through uploading a reference implementation alongside the JaCoCo jars, running the student's tests under the JaCoCo agent, and converting the resulting coverage data into a numeric grade.

This guide covers a scenario where students write the tests and you grade them against a reference implementation you provide. For the opposite scenario — instructor-written tests grading student code — see JUnit5 guide.

Before you start

You will need the following files ready to upload:

• jacocoagent.jar — the JaCoCo Java agent. Attached to the JVM with -javaagent, it instruments classes at runtime and records which lines are executed.

• jacococli.jar — the JaCoCo command-line tool. It converts the raw coverage data into an XML report that can be graded.

• cov_grade.sh — a shell script that reads the XML report and prints a numeric grade for the rubric. See the Appendix for the full script.

• Your reference implementation (e.g. Calculator.java).

Both JaCoCo jars can be downloaded from the official JaCoCo release page.

Step 1: Install the JUnit console launcher

JaCoCo instruments the JVM directly, so the standard cg junit5 helpers cannot be used here. Instead, download the JUnit5 platform console launcher jar from Maven Central during Setup.

1. Navigate to the Setup tab under the AutoTest settings and install JUnit5.

AutoTest Setup to install JUnit5 using Script block

1. Navigate to the Test tab and add an Allow Internet block.

2. Add a Script block inside the Allow Internet block and paste the following:

curl -O https://repo1.maven.org/maven2/org/junit/platform/junit-platform-console-standalone/1.10.0/junit-platform-console-standalone-1.10.0.jar
mv junit-platform-console-standalone-1.10.0.jar junit-platform-console-standalone.jar

An Allow Internet block wrapping a Script that downloads the JUnit5 console launcher.

The Allow Internet block is required in Test because curl needs to reach Maven Central. Renaming the jar drops the version number so the rest of your scripts can reference a stable path.

Step 2: Compile the student's tests against a reference implementation

1. In Tests tab, add an Upload Files block and attach jacocoagent.jar, jacococli.jar, cov_grade.sh, and your reference implementation (e.g. Calculator.java).

2. Add a Connect Rubric block and select the appropriate rubric category from the drop-down menu.

3. Add a Script block nested inside the Connect Rubric block and paste the following:

mv $UPLOADED_FILES/* .
javac -cp ".:$STUDENT:junit-platform-console-standalone.jar" \
  Calculator.java \
  $STUDENT/CalculatorTest.java

A Connect Rubric containing the Compilation script.

Moving the uploaded files into the working directory first ensures that the compiler and the JaCoCo jars can all find each other on the classpath.

Step 3: Run the tests under JaCoCo and grade the coverage

This step uses two Custom Test blocks nested inside the same Connect Rubric block from Step 2.

Custom Test 1 — Run student unit tests

Add a Custom Test block and paste the following:

# Run tests with JaCoCo agent, output to JUnit XML
java -javaagent:./jacocoagent.jar \
  -cp ".:$STUDENT:junit-platform-console-standalone.jar" \
  org.junit.platform.console.ConsoleLauncher \
  execute \
  --disable-banner \
  --details=none \
  --reports-dir=junitxml \
  --select-class=CalculatorTest

cg unit-tests junitxml junitxml/*.xml

The -javaagent flag attaches the JaCoCo agent to the JVM, which instruments the reference implementation's classes at runtime and writes the raw coverage data to jacoco.exec. cg unit-tests parses the JUnit XML output so pass/fail information appears in the submission view.

Custom Test 2 — Grade code coverage

Add a second Custom Test block immediately after and paste the following:

# Directory layout required by JaCoCo's report command
mkdir src tests compiled
mv Calculator.java src
mv Calculator.class compiled
mv $STUDENT/CalculatorTest.class tests
rm $STUDENT/CalculatorTest.java

# Generate the XML coverage report
java -jar jacococli.jar report jacoco.exec \
  --classfiles compiled \
  --sourcefiles src \
  --xml cov.xml

# Grade based on the XML report
./cov_grade.sh cov.xml

The Custom Test blocks for running student unit tests and grading code coverage.

The directory structure (src, tests, compiled) is required by JaCoCo's report command. Skipping or renaming these folders will cause the report step to fail.

Conclusion

You have set up an automatically graded assignment that measures how thoroughly students test a reference implementation. For background on standard instructor-written JUnit 5 tests, or for adding Checkstyle to your Java assignments, see the guides below. For questions about custom grading scripts or coverage thresholds, reach out to our support team at [email protected].

Appendix: cov_grade.sh

The script below parses the JaCoCo XML report and outputs a line coverage grade as a fraction (e.g. 3/4). CodeGrade uses this fraction to award rubric points proportionally. If LINE coverage data is not present in the report, the script falls back to INSTRUCTION coverage. If the XML cannot be parsed at all, it returns a grade of 0/1.

#!/usr/bin/env bash
set -uo pipefail

COV_XML="${1:-cov.xml}"

# If FD 3 is open, write structured JSON there. Otherwise skip JSON output.
if { true >&3; } 2>/dev/null; then
  JSON_FD=3
else
  JSON_FD=""
fi

python3 - "$COV_XML" "${JSON_FD:-}" < 2 else ""

def emit_json(obj) -> None:
    if not json_fd:
        return
    data = (json.dumps(obj) + "\n").encode("utf-8")
    os.write(int(json_fd), data)

try:
    root = ET.parse(path).getroot()
except Exception:
    covered = 0
    missed = 1
    total = 1
    frac = Fraction(0, 1)

    print("Code Coverage Grade")
    print("-----------------------")
    print(f"Covered={covered}")
    print(f"Missed={missed}")
    print(f"Total={total}")
    print(f"Grade={frac.numerator}/{frac.denominator}")

    emit_json({
        "tag": "points",
        "points": f"{frac.numerator}/{frac.denominator}"
    })
    sys.exit(0)

covered = None
missed = None

for counter in root.findall("counter"):
    if counter.get("type") == "LINE":
        missed = int(counter.get("missed", "0"))
        covered = int(counter.get("covered", "0"))
        break

if covered is None:
    for counter in root.findall("counter"):
        if counter.get("type") == "INSTRUCTION":
            missed = int(counter.get("missed", "0"))
            covered = int(counter.get("covered", "0"))
            break

if covered is None:
    covered = 0
    missed = 1
    total = 1
    frac = Fraction(0, 1)
else:
    total = covered + missed
    frac = Fraction(covered, total) if total > 0 else Fraction(0, 1)

print("Code Coverage Grade")
print("-----------------------")
print(f"Covered={covered}")
print(f"Missed={missed}")
print(f"Total={total}")
print(f"Grade={frac.numerator}/{frac.denominator}")

emit_json({
    "tag": "points",
    "points": f"{frac.numerator}/{frac.denominator}"
})
PY

exit 0

The script expresses the grade as an exact fraction (e.g. 7/10) rather than a percentage. CodeGrade reads this fraction directly to calculate rubric points, so the output format must not be changed.

When LINE coverage data is absent from the report — which can happen with certain JVM configurations — the script automatically falls back to INSTRUCTION coverage. Both metrics are present in every standard JaCoCo XML report.

If the XML file cannot be parsed (for example because the JaCoCo agent failed to write jacoco.exec), the script returns 0/1 rather than exiting with an error, so the AutoTest run completes and the student receives a zero for the coverage category instead of a broken result.