In this guide, we explore the advanced grading options available for C assignments. For more information about setting up a C assignment from scratch, see:
Check Unit Tests
is an industry-standard unit testing framework for C. It is particularly useful for grading assignments that require students to write functions and classes. Unit testing with check offers several advantages over conventional IO testing including the ability to use assertions, parametrize test cases, and provide better feedback for students. Consider the example submission below:
#include "bubble_sort.h"
void bubble_sort(int arr[], int n) {
int i, j, temp;
for (i = 0; i < n-1; i++) {
for (j = 0; j < n-i-1; j++) {
if (arr[j] > arr[j+1]) {
// Swap arr[j] and arr[j+1]
temp = arr[j];
arr[j] = arr[j+1];
arr[j+1] = temp;
}
}
}
}
In this example, students have been tasked with creating the bubble_sort() function. We can create robust test cases to check this submission using assertions from the check unit testing framework.
unit_test.c
#include <check.h>
#include "bubbleSort.h"
START_TEST(test_bubbleSort_basic) {
int arr[] = {5, 3, 8, 4, 2};
int expected[] = {2, 3, 4, 5, 8};
int n = 5;
bubbleSort(arr, n);
for (int i = 0; i < n; i++) {
ck_assert_int_eq(arr[i], expected[i]);
}
}
END_TEST
START_TEST(test_bubbleSort_sorted) {
int arr[] = {1, 2, 3, 4, 5};
int expected[] = {1, 2, 3, 4, 5};
int n = 5;
bubbleSort(arr, n);
for (int i = 0; i < n; i++) {
ck_assert_int_eq(arr[i], expected[i]);
}
}
END_TEST
START_TEST(test_bubbleSort_reverse) {
int arr[] = {9, 7, 5, 3, 1};
int expected[] = {1, 3, 5, 7, 9};
int n = 5;
bubbleSort(arr, n);
for (int i = 0; i < n; i++) {
ck_assert_int_eq(arr[i], expected[i]);
}
}
END_TEST
START_TEST(test_bubbleSort_duplicates) {
int arr[] = {4, 1, 3, 4, 2};
int expected[] = {1, 2, 3, 4, 4};
int n = 5;
bubbleSort(arr, n);
for (int i = 0; i < n; i++) {
ck_assert_int_eq(arr[i], expected[i]);
}
}
END_TEST
Suite* bubbleSort_suite(void) {
Suite *s;
TCase *tc_core;
s = suite_create("BubbleSort");
/* Core test case */
tc_core = tcase_create("Core");
tcase_add_test(tc_core, test_bubbleSort_basic);
tcase_add_test(tc_core, test_bubbleSort_sorted);
tcase_add_test(tc_core, test_bubbleSort_reverse);
tcase_add_test(tc_core, test_bubbleSort_duplicates);
suite_add_tcase(s, tc_core);
return s;
}
int main(void) {
int number_failed;
Suite *s;
SRunner *sr;
s = bubbleSort_suite();
sr = srunner_create(s);
srunner_set_xml(sr, "report.xml");
srunner_run_all(sr, CK_NORMAL);
number_failed = srunner_ntests_failed(sr);
srunner_free(sr);
return (number_failed == 0) ? 0 : 1;
}
"cg junitxml" command
We could simply compile and run the test cases to show students a simple pass or fail depending on the exit code of the tests. However, we wouldn't be able to award partial marks for each test case unless we parsed the results of the unit tests. To accomplish that, we can use the cg junitxml command. This command allows us to parse the number of test cases passed and the feedback from failed test cases from any unit test report written in the JUnit XML format.
Running the command cg junitxml --help shows the following information:
This is a parser for generic unit test coverage reports in JUnit XML format.
The parser can take in input multiple JUnit XML files. The final reported score
is an aggregate of all the reports. Skipped test cases do not count towards the
score.
Example use:
Input to the command (in a file called coverage.xml):
<?xml version="1.0" encoding="UTF-8" ?>
<testsuites id="Calculator suite" name="empty_name" tests="2" failures="1" time="0.001">
<testsuite id="uuid1" name="Addition and Multiplication" tests="2" failures="1" time="0.001">
<testcase id="uuid2" name="[1] Check whether the addition function returns the expected result." time="0.001"></testcase>
<testcase id="uuid2" name="[2] Check whether the multiply function returns the expected result." time="0.001">
<failure message="expected: 6.0 but was: 5.0" type="ERROR">
ERROR: Expected: 6.0 but was: 5.0
Category: Checking returns - Multiplication
File: /home/codegrade/student/calculator.py
Line: 2
</failure>
</testcase>
</testsuite>
</testsuites>
You would then call:
cg junitxml coverage.xml
Usage:
cg junitxml XMLS... [flags]
Flags:
-h, --help help for junitxml
--no-parse-weight Don't try to parse the weight from titles.
--no-score Don't calculate and output the final score based on the results of the input data.
-o, --output string The output file to to use. If this is a number it will use that file descriptor. (default "3")
The cg junitxml command works well in combination with the Custom Test block by displaying the results of the parsed JUnit XML report beautifully, making it easy to read and interpret, as shown in the image below.
Unfortunately, the check framework doesn't have a method for automatically outputting JUnit XML. Instead we can use the srunner_set_xml() command to generate an XML report (in check's own format) and use the following script to convert that into the JUnit XML format.
junit_xml.py
#!/usr/bin/env python3
# pylint: disable=missing-module-docstring,missing-function-docstring
import os
import sys
import typing as t
import subprocess
import xml.etree.ElementTree as ET
def _find_and_get_text(el: ET.Element, to_find: str) -> str:
found = el.find(to_find)
assert found is not None, (
'Expected to find %s in %s, but could not find it' % (to_find, el.tag)
)
return found.text or ''
def parse_case(el: ET.Element) -> ET.Element:
case = ET.Element('testcase')
case.set('classname', _find_and_get_text(el, 'description'))
case.set('name', _find_and_get_text(el, 'id'))
# Sometimes the 'duration' attr is -1.
case.set('time', str(max(0, float(_find_and_get_text(el, 'duration')))))
state = el.get('result')
if state is not None and state != 'success':
message = ET.Element(state)
message.text = _find_and_get_text(el, 'message')
case.append(message)
return case
def parse_suite(el: ET.Element) -> ET.Element:
cases = [parse_case(c) for c in el.findall('test')]
suite = ET.Element('testsuite')
suite.set('name', _find_and_get_text(el, 'title'))
suite.set('tests', str(len(cases)))
suite.set(
'failures',
str(sum(1 for c in cases if c.find('failure') is not None))
)
suite.set(
'errors', str(sum(1 for c in cases if c.find('error') is not None))
)
suite.set('time', str(sum(float(c.get('time') or '0') for c in cases)))
suite.extend(cases)
return suite
def parse_tree(el: ET.Element) -> ET.ElementTree:
assert el.tag == 'testsuites'
suites = [parse_suite(s) for s in el.findall('suite')]
tree = ET.Element('testsuites')
tree.set('time', _find_and_get_text(el, 'duration'))
tree.set('tests', str(sum(int(s.get('tests') or '0') for s in suites)))
tree.set(
'failures', str(sum(int(s.get('failures') or '0') for s in suites))
)
tree.set('errors', str(sum(int(s.get('errors') or '0') for s in suites)))
tree.extend(suites)
return ET.ElementTree(tree)
def run(
test: t.List[str]
) -> t.NoReturn:
"""Run a Check suite and parse its results
"""
status = subprocess.run(test)
# Remove namespaces in tag names.
tree = ET.iterparse("report.xml")
for _, el in tree:
_, _, el.tag = el.tag.rpartition('}')
root = tree.root # type: ignore
parse_tree(root).write("report.xml")
sys.exit(status)
if __name__ == '__main__':
run(sys.argv[1:])
Instructions
In the AutoTest settings, navigate to the Setup tab.
Add an Install GCC block to your setup configuration.
Add a Script block to your setup configuration and install check with the following command.
sudo apt install check
In the AutoTest settings, navigate to the Tests tab.
Add an Upload Files block. Upload bubble_sort.h, unit_test.c, and junit_xml.py.
Add a Script block. Move all the uploaded files to the student's directory and compile bubble_sort.c with the following commands.
Add a Connect Rubric block and a Custom Test block. Nest the Custom Test block within the Connect Rubric block. Run the unit tests and parse the results with the following commands.
Simply running clang-tidy would allow us to produce the default command line output. However, this wouldn't be particularly useful for students as they would have to spend time interpreting the output and would have to switch back and forth between the AutoTest output and their code. Instead, we can use the cg comments command to parse the output of clang-tidy and write the comments directly onto our students' code. The cg comments command will highlight each target line according to the severity of the comment and can be read by hovering over the line number with the mouse cursor. The comments are also placed on students' code in the editor, making it a powerful combination.
Instructions
In the AutoTest settings, navigate to the Setup tab.
Add a Script block. Install clang-tidy using the following command.
sudo apt update
sudo apt install clang-tidy
In the AutoTest settings, navigate to the Tests tab.
Add a Connect Rubric block and a Custom Test block. Nest the Custom Test block in the Connect Rubric block. Run clang-tidy and parse the output using the following commands.
is an industry-standard static analysis tool for C, C++, and Objective-C that allows you to diagnose and fix code style violations, interface misuse, and bugs. It is a useful tool for enforcing code styling best practices for beginner programmers.
