Step 1: Setup

CodeGrade AutoTest runs on Ubuntu (18.04.2 LTS) machines which you can configure in any way that you want.
In the setup section of your AutoTest, you can upload any files you might need for testing. For instance C++ library files, or unit testing scripts. These files are called Fixtures and will be placed in the $FIXTURES directory on the Virtual Server.
The file structure of the server is like this:
All of your uploaded fixtures will be here.
This is where the submission of student is placed.
Tests are executed from this directory.
After uploading any files you might need, you can run some setup to install any packages you might need.
  • Global setup script: this is where you install any additional packages you want to be available on the server, using sudo apt install <package>. The global setup only runs once and is then cached, so the student submission won't be available here yet.
  • Per-student setup script: this will run once when running the autograding for a student and will run before all tests. This is a good place to move $FIXTURES to the $STUDENT directory in case the student solution needs some input files to run correctly or if you are doing unit testing, as the unit tests have to be placed in the same directory as the student files. Move all fixtures with the following command: cp $FIXTURES/* $STUDENT
Use any command in the Global Setup Script field to install software or run your setup script

Step 2: Create your tests

Now that you have created the setup, it's time to create the actual tests. Do this by pressing the "Add Level" button and then the "Add Category" button.
All tests in AutoTest fill in a specific rubric category that you select. After selecting the category, you can start creating tests that will fill it in. Multiple test types are available in CodeGrade, which can be used together depending on your needs and wishes.

Compile the student's code

The first step in running any tests is compiling the code, that will give us an executable that we can use in subsequent tests.
Compiling and running C++ code in CodeGrade.
  1. 1.
    Create a Run Program test.
  2. 2.
    Enter the compilation command to compile the code:
    1. 1.
      g++ -o Example Example.cpp or similar. You can also run your make file, if you have uploaded this to the $FIXTURES folder. Or use another compiler if you prefer.

Set up an IO Test

IO Tests are great for console based programs and allow you to give an input to it and specify an expected output.
  1. 1.
    In Program to Test, execute the compiled student code, just like you would do in your own terminal: e.g. ./Example.
  2. 2.
    Give a name, so students know what the test is about.
  3. 3.
    Specify input and expected output. Several options are available to match a wider range of outputs.

Set up a GTest unit test

CodeGrade supports any unit test framework you'd like to use for C++. In this guide we will cover setting up GoogleTest, or GTest.
Follow these steps to set up GTest in CodeGrade, we will use Sample 1 from the Googletest Samples:
// A sample program demonstrating using Google C++ testing framework.
#include "sample1.h"
// Returns n! (the factorial of n). For negative n, n! is defined to be 1.
int Factorial(int n) {
int result = 1;
for (int i = 1; i <= n; i++) {
result *= i;
return result;
// Returns true if and only if n is a prime number.
bool IsPrime(int n) {
// Trivial case 1: small numbers
if (n <= 1) return false;
// Trivial case 2: even numbers
if (n % 2 == 0) return n == 2;
// Now, we have that n is odd and n >= 3.
// Try to divide n by every odd number i, starting from 3
for (int i = 3;; i += 2) {
// We only have to try i up to the square root of n
if (i > n / i) break;
// Now, we have i <= n/i < n.
// If n is divisible by i, n is not prime.
if (n % i == 0) return false;
// n has no integer factor in the range (1, n), and thus is prime.
return true;
// A sample program demonstrating using Google C++ testing framework.
// Returns n! (the factorial of n). For negative n, n! is defined to be 1.
int Factorial(int n);
// Returns true if and only if n is a prime number.
bool IsPrime(int n);
  1. 1.
    Upload your GTest tests as $FIXTURES, in our case, we upload: sample1_unittest.cpp:
#include "sample1.h"
#include <limits.h>
#include "gtest/gtest.h"
TEST(FactorialTest, Negative) {
// This test is named "Negative", and belongs to the "FactorialTest"
// test case.
EXPECT_EQ(1, Factorial(-5));
EXPECT_EQ(1, Factorial(-1));
EXPECT_GT(Factorial(-10), 0);
// Tests factorial of 0.
TEST(FactorialTest, Zero) { EXPECT_EQ(1, Factorial(0)); }
// Tests factorial of positive numbers.
TEST(FactorialTest, Positive) {
EXPECT_EQ(1, Factorial(1));
EXPECT_EQ(2, Factorial(2));
EXPECT_EQ(6, Factorial(3));
EXPECT_EQ(40320, Factorial(8));
// Tests IsPrime()
// Tests negative input.
TEST(IsPrimeTest, Negative) {
// This test belongs to the IsPrimeTest test case.
// Tests some trivial cases.
TEST(IsPrimeTest, Trivial) {
// Tests positive input.
TEST(IsPrimeTest, Positive) {
extern "C" int startTest(int x,char ** y) {
testing::InitGoogleTest(&x, y);
int code = RUN_ALL_TESTS();
return code;
extern "C" int __wrap_main(int x,char ** y) {
return startTest(x,y);
Please note that we have added two extra functions: startTest and __wrap_main, this is so that we can override the main method of student code, if they have one.
2. Upload the file to update CMake on CodeGrade, run it using ./ in the Global Setup Script:
#!/usr/bin/env bash
# Avoid debconf warnings about the frontend.
echo 'debconf debconf/frontend select Noninteractive' | sudo debconf-set-selections
sudo apt-get remove --purge cmake
wget --quiet -O - | sudo apt-key add - 2>/dev/null
sudo apt-add-repository 'deb bionic main'
sudo apt-get update -q
sudo apt-get install cmake -q
3. Upload your CMakeLists.txt, you may use the following as an example and edit it to fit your own assignment:
# GoogleTest requires at least C++11
GIT_TAG release-1.12.1
# For Windows: Prevent overriding the parent project's compiler/linker settings
set(gtest_force_shared_crt ON CACHE BOOL "" FORCE)
add_library(sample1 SHARED sample1.cpp)

Creating the unit test in CodeGrade

  1. 1.
    Upload the unit test file, CMakeLists.txt and as a $FIXTURE in the CodeGrade setup.
  2. 2.
    Run ./ in the Global setup script.
  3. 3.
    Copy the unit test file and CMakeLists.txt to the student directory in the per-student setup script and generate the Makefile using: cp $FIXTURES/* $STUDENT; cmake .;.
  4. 4.
    In the category where you want the test to live first create a Run Program Test with the build command: cmake --build ..
  5. 5.
    Create a Unit Test
  6. 6.
    Select Custom.
  7. 7.
    As the program to test write: ./sample1_unittest --gtest_output=xml:$CG_JUNIT_XML_LOCATION.
Build the student's C++ code and run the GTest Unit Test in CodeGrade.

Set up a Clang-tidy Code Quality test

Automatically run static code analysis on the student's code. In C++ you can do that with Clang-tidy.
​Clang-tidy is a static code analysis tool for C and C++ that is built-in to CodeGrade. It is a traditional linter: it checks for typical programming errors in the code.
Amongst other things, Clang-tidy can detect:
  • Style violations;
  • Interface misuse;
  • Common bugs that can be detected statically.

Create Code Quality test

After choosing the Code Quality test in your AutoTest category, you can simply select a linter from the dropdown list. Choose Clang-tidy for your C assignment.
After doing so, you must select extra arguments to select the checks and the file to check. Commonly a good argument to start with is --checks='cppcoreguidelines-*' sample1.cpp to enable all core C++ checks and select the file you want to check (replace sample1.cpp with your filename). Feel free to explore the other checks and options that are available for Clang-tidy.
Setting up a Code Quality test using the Clang-tidy linter for C++.

Set up a Code Structure test

CodeGrade integrates a tool called Semgrep to make it easy to perform more complex code analysis by allowing you to write rules in a human readable format. You can provide generic or language specific patterns, which are then found in the code. With its pattern syntax, you can find:
  • Equivalences: Matching code that means the same thing even though it looks different.
  • Wildcards / ellipsis (...): Matching any statement, expression or variable.
  • Metavariables ($X): Matching unknown expressions that you do not yet know what they will exactly look like, but want to be the same variable in multiple parts of your pattern.

Writing the tests

Semgrep is pre-installed in the Unit Test. You will write your Semgrep Code Structure tests in a YAML file and upload this as a fixture in the setup section of AutoTest. Semgrep has an online editor that can be used to check and create your patterns:
For instance, this is a ruleset to detect if a student uses an if-statement or not in their code.
- id: if-statement
match-expected: true
pattern: |
if ($COND) {
message: You must use an if-statement in your code!
severity: INFO
- cpp
We use the ellipses (the … in the pattern) here, so that any code can be inside the blocks of the if-statement. The $COND captures any variable / condition. Save this in a file named for instance rules.yml.
  • Patterns
    • The ellipsis (...) is used to capture anything
    • metavariables $EL (element) and $LST (list) capture the two parts of the for-loop declaration (the naming of these metavariables is irrelevant and could have been anything else).
    • The same is done for the while loop condition $COND
  • Messages We are able to provide understandable messages that are parsed by the wrapper script making our tests understandable for our students.
  • Match-expected Importantly, we have added the match-expected field (this is added in the CodeGrade wrapper script and cannot be tested in regular semgrep), with putting this field to True for the for-loop rule, we specify that we are expecting a match in order to pass that test.
  • Severity This dictates the level of severity of failing the rule. The penalty for each severity level can be set in the Unit Test step.
  • languages Here we specify the language of the scripts which semgrep will be checking.

Creating the test in CodeGrade

  1. 1.
    Upload the YAML (.yml) with your Code Structure tests in it as a $FIXTURE in the CodeGrade setup.
  2. 2.
    In the category where you want the test to live create a Unit Test.
  3. 3.
    Select semgrep.
  4. 4.
    Give the follow extra argument: $FIXTURES/your-test-file.yml $STUDENT.
If you run this on your student's code, it will then show up like this:

Other test types

  • Capture Points Tests: Use your own custom grading script, upload it as a $FIXTURE, execute it in this test and make sure it outputs a float between 0.0 and 1.0, this score will then be the number of points a student receives.
  • Checkpoints: Only execute tests below the checkpoint if the tests above the checkpoint reach a certain score. Great for separating simple and advanced tests.

Step 3: Start the AutoTest

Once you have created a configuration, press start and your AutoTest will start running. In the General Settings, you should have already uploaded a Test Submission, so you will see the results of this straight away.
Once it's started and your assignment is set to Open or Done, students can hand in and get immediate feedback!