☕
Java

CodeGrade launched AutoTest V2 in early 2023.
Interface of AutoTest V2
Overview
An AutoTest V2 configuration for a Java assignment has two stages:
The Setup stage: this is where you setup the auto grading environment. You can install packages and upload files. This builds into an image on which all tests are executed. All the configuration here only runs once for all submissions;
The Tests stage: this is what students will see and where you can configure all your tests that you want to run on every student’s code. Everything that you specify here is executed in the working directory of the student.
Steps are created in a “Scratch-like” block layout and can be dragged and dropped, reordered and nested to create an AutoTest program. We are continuously adding new blocks to AutoTest V2 to make it more powerful and easier to use.
AutoTest V2 is still available next to our original AutoTest, to start setting up an AutoTest V2 select AutoTest V2 when creating a new AutoTest.

Already have an original AutoTest? You can switch to AutoTest V2 by deleting it and pressing "Select another version of AutoTest" to finally select AutoTest V2. Please note that your original AutoTest will be deleted when pressing the "Delete" button!
Developing, snapshots and publishing to students
When developing your AutoTest V2 configuration, you can continuously test your configuration on the "Test Submission".
After configuring something, you press the “Build Snapshot” button in the bottom right corner of the screen. This will Build your AutoTest into a Snapshot (a history of your snapshots are available via the Snapshots tab).
After pressing "Build Snapshot", you can:
Test the configuration by seeing the results in seconds.
If you are ready to publish your AutoTest to your students press the big green Publish button.
If you make any changes, you build again and if you are satisfied, you can republish them to your students.
If you want to unpublish your snapshot, you can simply go to it in the green bar and press the red “Unpublish” button.
Step 1: Setup
CodeGrade AutoTest V2 runs on Ubuntu (20.04 LTS) machines which you can configure in any way that you want. Common software is pre-installed, most importantly for this guide: java is pre-installed.
In the setup section of your AutoTest, you can install software or packages and upload any files you might need for testing. The setup section will build an image and is only run once for all submissions. 
You can upload files using the "Upload Files" block, if you are intending on using Java Unit Testing fusing JUnit5 for instance, this is where you will upload your unit test file. These files will be placed in the $UPLOADED_FILES directory on the Virtual Server.
You can install software and packages (or configure the build image in any other way) using the "Script" block. You have network access by default in the Setup tab, so you may also download libraries you want to use.
Basic set up for an autograded Java assignment in AutoTest V2
Step 2: Tests
Now that you have created the setup, it's time to create the actual tests. This is done on the Tests tab. Select one of the many test-blocks to configure a AutoTest V2 procedure in a "scratch"-like format. Some tests can be nested and you can chose to connect tests to rubric categories with a certain weight, you can also hide aspects of tests and enable internet access for specific tests.
Set up an IO Test
IO Tests are great for console based programs and allow you to give an input to it and specify an expected output. The following will guide you to set up an IO Test:
A basic IO Test to check the output for a Java program.
1.You will have to compile your students' Java code somewhere. This may be in a "Script" block before your "IO Test" block. Compiling java is done via a command like javac Fibonacci.java or javac *.java to compile all files. 
2.Drag the "IO Test" block to your Test Procedure. In the code section, write the command to execute, for Java this could be something like java Fibonacci.
3.Drag one or more "Simple Test" block(s) inside your "IO Test" block. Specify an Input (given via stdin / standard input) and an Expected Output.
4.Give clear names to the numerous blocks you have created to make clear to the students what is being tested.
5.Optionally: drag in a "Connect Rubric" block and drag your "IO Test" block in this, to connect it to a rubric category and use it to grade your students. 
Set up a JUnit unit test
Upload a JUnit unit test file in the Setup tab, it can then easily be used to check the students' code. Until the "Unit Test" block is available in AutoTest V2, we have created a simple wrapper script to use JUnit5 unit tests easily in AutoTest V2.
Imagine students have to code a very simple calculator in a file called calculator.py:
public class Calculator {
  public static float add(float numberOne, float numberTwo) {
    return numberOne + numberTwo;
  }
  public static float subtract(float numberOne, float numberTwo) {
    return numberOne - numberTwo;
  }
  public static float multiply(float numberOne, float numberTwo) {
    return numberOne * numberTwo;
  }
  public static float divide(int numberOne, int numberTwo) {
    return numberOne / numberTwo; // Should return ArithmeticException
  }
}
A powerful way to test the individual methods, is to write unit tests. In this example, we can test the student's code with the following unit tests in a file called TestCalculator.java (make sure your test files always start with Test):
import java.util.*;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.DisplayName;
public class TestCalculator {
  @Test
  @DisplayName("Check whether the addition function returns the expected result.")
  public void testAdd() {
    assertEquals(Calculator.add(2, 3), 5, 0.2);
  }
  @Test
  @DisplayName("Check whether the subtraction function returns the expected result")
  public void testSubtract() {
    assertEquals(Calculator.subtract(5, 2), 3, 0.2);
  }
  @Test
  @DisplayName("Check whether the multiply function returns the expected result")
  public void testMultiply() {
    assertEquals(Calculator.multiply(2, 3), 6, 0.2);
  }
  @Test
  @DisplayName("Check whether the divide function returns the expected result and throws an ArithmeticException when dividing by 0")
  public void testDivide() {
    assertEquals(Calculator.divide(10, 2), 5, 0.2);
    assertThrows(ArithmeticException.class, () -> {
      Calculator.divide(10, 0);
    });
  }
}
Creating the unit test in CodeGrade
run_junit.py
11KB
Text
Download the run_junit.py script above.
run_junit.py
11KB
Text
Download the run_junit.py script above.
1.Upload the JUnit unit test file and the run_junit.py script (download above) to AutoTest V2 under the Setup tab. 
2.Install JUnit5 using python3 $UPLOADED_FILES/run_junit.py --install in a "Script" block.
3.Continue under the Tests tab: create and drag "Custom Test" block, using the run_pytest.py script we can run our Unit Test in here.
4.In the code field, use the uploaded wrapper script as follows: 
python3 $UPLOADED_FILES/run_junit.py TestCalculator
Of course, always replacing TestCalculator with the name of your uploaded unit test script. The wrapper script will now automatically compile all found Java files and copy TestCalculator to the correct directory. You may also use the --compile and flag arguments to specify which Java files to compile and the.
5.Optionally: Use a "Connect Rubric" block to connect your unit tests to a rubric category.
A JUnit5 unit test connected to a rubric in AutoTest V2.
Set up a Code Quality test
Automatically run static code analysis on the student's code, using Checkstyle or PMD. Until the "Code Quality" block is available in AutoTest V2, we have created simple wrapper scripts to use Checkstyle and PMD easily in AutoTest V2.
The wrappers will then run automated quality checking on the student's code and assign a score. Code Quality comments will be also generated and appended to your students' submissions.

Amongst other things, Checkstyle and PMD can detect:
Use of naming conventions (variables, methods, etc.)
Completeness of documentation (think Javadoc)
Use of vertical and horizontal whitespace
Conventions for parameters, headers and imports
Checkstyle
​Checkstyle is a static code analysis tool for Java that is built-in to CodeGrade. It is a traditional linter: it checks if the code style and presentation conform to a style guide, but it does not check or confirm code correctness or completeness.
Running Checkstyle in CodeGrade
run_checkstyle.py
7KB
Text
Download the run_checkstyle.py script above.
1.Upload the Checkstyle script (run_checksytle.py, download above) to AutoTest V2 under the Setup tab. Also, in a "Script" block, install it using python3 $UPLOADED_FILES/run_checkstyle.py --install.
2.Optionally: Upload a custom Checkstyle configuration .xml file.
3.Continue under the Tests tab: create and drag "Custom Test" block, using the run_checksytle.py script we can run Checkstyle here.
4.In the code field now use the wrapper script as follows: 
python3 $UPLOADED_FILES/run_checkstyle.py --run Calculator.java --max-errors 10 
The --run option allows you to select which file to scan. You may select multiple Java files.  The --max-errors  option indicates the penalty score that leads to 0.0 points, other scores are calculated linearly based on this.  The default checkstyle config is the Google Java Style Guide, use the --config option to select a custom configuration file.
5.Optionally: Use a "Connect Rubric" block to connect your code quality check to a rubric category.
A Checkstyle test connected to a rubric in AutoTest V2.
PMD
​PMD is a static source code analyzer that is mainly used with Java but can also support other languages. It is a traditional linter: it checks if the code style and presentation conform to a style guide, but it does not check or confirm code correctness or completeness.
Running PMD in CodeGrade
run_pmd.py
7KB
Text
Download the run_pmd.py script above
pmd_basic_config.xml
3KB
Text
Download a PMD ruleset that implements basic coding conventions for Java
1.Upload the PMD script (run_pmd.py, download above) to AutoTest V2 under the Setup tab. Also, in a "Script" block, install it using python3 $UPLOADED_FILES/run_pmd.py --install.
2.Upload a custom PMD ruleset as a .xml file (an example can be downloaded above). Learn more here about writing your own ruleset.
3.Continue under the Tests tab: create and drag a "Custom Test" block, using the run_pmd.py script we can run PMD here.
4.In the code field now use the wrapper script as follows: 
python3 $UPLOADED_FILES/run_pmd.py --run *.java --max-errors 10 --config $UPLOADED_FILES/pmd_basic_config.xml 
The --run option allows you to select which file to scan. You may select multiple Java files.  The --max-errors  option indicates the penalty score that leads to 0.0 points, other scores are calculated linearly based on this.  You may change in the run_pmd.py script how penalty scores are assigned according to the severity of the errors.  The --config  option is used to set your custom ruleset. You always need to provide a ruleset.
5.Optionally: Use a "Connect Rubric" block to connect your code quality check to a rubric category.
A PMD test connected to a rubric in AutoTest V2.
Set up a Code Structure test
CodeGrade integrates a tool called Semgrep to make it easy to perform more complex code analysis by allowing you to write rules in a human readable format. You can provide generic or language specific patterns, which are then found in the code. With its pattern syntax, you can find:
Optionally: Use a "Connect Rubric" block to connect your code quality check to a rubric category.
Equivalences: Matching code that means the same thing even though it looks different.
Wildcards / ellipsis (...): Matching any statement, expression or variable.
Metavariables ($X): Matching unknown expressions that you do not yet know what they will exactly look like, but want to be the same variable in multiple parts of your pattern.
Writing the tests
Semgrep can be installed in the Setup step using pip, use: pip3 install semgrep. The easiest way to use Semgrep in AutoTest V2 is via our structure.py script (this will be built-in in a later version). With this script, you will write your Semgrep patterns directly in the Input field in an IO Test. Try out Semgrep patterns using their online editor here:
Semgrep
For instance, this is a ruleset to detect too many nested if statements. 
pattern: |
  if (...) {
     ...
     if (...) {
         ...
         if (...) {
             ...
         }
     }
  }
We use the ellipses (the … in the pattern) here, so that any code can be in the condition and inside the blocks of the if-statements. As an if-statement itself can also be part of this, we automatically catch code that has more than three nested if-statements with this pattern too. 
Creating the test in CodeGrade
structure.py
2KB
Text
Download the structure.py script to run Semgrep in AutoTest V2.
1.Under the Setup tab, upload the structure.py file you can download above. Also, in a "Script" block, install Semgrep via pip3 install semgrep.
2.Continue in the Tests tab: first drag the "IO Test" block to your Test Procedure. Execute the script in the code field, follow: python3 $UPLOADED_FILES/structure.py <STUDENTFILE>. For a Calculator assignment, this could be: python3 $UPLOADED_FILES/structure.py Calculator.java.
3.Create and drag one or more "Simple Test" block(s) inside your "IO Test" block. Each block is one Semgrep structure to check. As Input write the Semgrep pattern (including pattern: or similar like pattern-either:).
4.As Expected Output write "Found!" if the student code should match or "Not found!" if the student code should not contain the pattern.
5.Optionally: Use a "Connect Rubric" block to connect your structure check to a rubric category.
6.Optionally: Use a "Hide" block to hide the "config" (i.e. input / pattern) of your tests so that students cannot see the pattern you are checking for.
A Semgrep structure check in an IO Test in AutoTest V2 (with optional Hide block to hide the pattern).
Step 3: Start the AutoTest
Once you have created a configuration, create a Snapshot to test it on a "Test Submission". In the General Settings, you should have already uploaded a Test Submission, so you will see the results of this straight away.
Once everything works as expected, press the green "Publish to students" button to publish your AutoTest V2 to students. If they use the editor they can run it instantly, and if they hand in in any other way they will get their AutoTest results instantly after any submission.
More Java autograding
🤖
Testing the tests: Autograding Java unit tests
Autograding Python using AutoTest V1
Testing the tests: Autograding Java unit tests
Last modified 1mo ago