☕
Autograding Java using AutoTest V1
Want to give your students instant feedback and use CodeGrade's latest features as they come out? Navigate to the AutoTest V2 guide to learn how to set up your Python assignment with our new and improved (beta) AutoTest system!
Learn how to set up a basic Java assignment in CodeGrade.
CodeGrade AutoTest runs on Ubuntu (18.04.2 LTS) machines which you can configure in any way that you want and Java comes preinstalled.
In the setup section of your AutoTest, you can upload any files you might need for testing, if you are intending on using Unit Testing using JUnit4 or JUnit5, this is where you will upload your unit test file. These files are called Fixtures and will be placed in the
$FIXTURES directory on the Virtual Server.The file structure of the server is like this:
$FIXTURES/
All of your uploaded fixtures will be here.
$STUDENT/
This is where the submission of student is placed.
Tests are executed from this directory.
After uploading any files you might need, you can run some setup to install any packages you might need.
- Global setup script: this is where you install any additional packages you want to be available on the server, using
sudo apt install <package>. The global setup only runs once and is then cached, so the student submission won't be available here yet. - Per-student setup script: this will run once when running the autograding for a student and will run before all tests. This is a good place to move
$FIXTURESto the$STUDENTdirectory in case the student solution needs some input files to run correctly or if you are doing unit testing, as the unit tests have to be placed in the same directory as the student files. Move all fixtures with the following command:cp $FIXTURES/* $STUDENT
Use any command in the Global Setup Script field to install software or run your setup script
Now that you have created the setup, it's time to create the actual tests. Do this by pressing the "Add Level" button and then the "Add Category" button.
All tests in AutoTest fill in a specific rubric category that you select. After selecting the category, you can start creating tests that will fill it in. Multiple test types are available in CodeGrade, which can be used together depending on your needs and wishes.
The first step in running any tests is compiling the code, that will give us an executable that we can use in subsequent tests.
- 1.Create a Run Program test.
- 2.Enter the compilation command to compile the code:
- 1.
javac Fibonacci.javawhere you replace Fibonacci with the student's program name. - 2.If you are compiling multiple files use:
javac *.java, this will compile all files in the student directory.
Also using unit testing? Change the compile command from
javac to cg-junit5 compile, so: cg-junit5 compile *.java. This will correctly build in JUnit to the tests.IO Tests are great for console based programs and allow you to give an input to it and specify an expected output.
- 1.In Program to Test, write
javafollowed by the filename of the student's program without the.javaextension. - 2.Give a name, so students know what the test is about.
- 3.Specify input and expected output. Several options are available to match a wider range of outputs.
Upload a JUnit5 unit test file as a
$FIXTURE and automatically run unit tests on the code. Imagine students have to code a very simple calculator in a file called
Calculator.java:public class Calculator {
public static float add(float numberOne, float numberTwo) {
return numberOne + numberTwo;
}
public static float subtract(float numberOne, float numberTwo) {
return numberOne - numberTwo;
}
public static float multiply(float numberOne, float numberTwo) {
return numberOne * numberTwo;
}
public static float divide(int numberOne, int numberTwo) {
return numberOne / numberTwo; // Should return ArithmeticException
}
}
A powerful way to test the individual methods, is to write unit tests. In this example, we can test the student's code with the following unit tests in a file called
TestCalculator.java (make sure your test files always start with Test):import java.util.*;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.DisplayName;
public class TestCalculator {
@Test
@DisplayName("Check whether the addition function returns the expected result.")
public void testAdd() {
assertEquals(Calculator.add(2, 3), 5, 0.2);
}
@Test
@DisplayName("Check whether the subtraction function returns the expected result")
public void testSubtract() {
assertEquals(Calculator.subtract(5, 2), 3, 0.2);
}
@Test
@DisplayName("[2] Check whether the multiply function returns the expected result (weight 2)")
public void testMultiply() {
assertEquals(Calculator.multiply(2, 3), 6, 0.2);
}
@Test
@DisplayName("[2] Check whether the divide function returns the expected result and throws an ArithmeticException when dividing by 0 (weight 2)")
public void testDivide() {
assertEquals(Calculator.divide(10, 2), 5, 0.2);
assertThrows(ArithmeticException.class, () -> {
Calculator.divide(10, 0);
});
}
}
- 1.Upload the unit test file as a $FIXTURE in the CodeGrade setup.
- 2.Copy the unit test file to the student directory in the per-student setup script using:
cp $FIXTURES/TestCalculator.java $STUDENT. - 3.In the category where you want the test to live first create a Run Program Test with the compile command:
cg-junit5 compile *.java. - 4.Create a Unit Test
- 5.Select JUnit5.
- 6.As the class to test:
TestCalculator
Automatic run static code analysis on the student's code. In Java you can do that with either Checkstyle or PMD.
Checkstyle
Checkstyle is a static code analysis tool for Java that is built-in to CodeGrade. It is a traditional linter: it checks if the code style and presentation conform to a style guide, but it does not check or confirm code correctness or completeness.
Amongst other things, Checkstyle can detect:
- Use of naming conventions (variables, methods, etc.)
- Completeness of documentation (think Javadoc)
- Use of vertical and horizontal whitespace
- Conventions for parameters, headers and imports
PMD
PMD is a cross-language source code analyzer that not only works with Java, but also with JavaScript, XML and PL/SQL. In comparison to Checkstyle, PMD actually detects coding bad practices and potential bugs, but does not assess style.
Amongst other things, PMD can be used to detect:
- Inefficient code
- Bad programming habits
- Overcomplicated expressions that can be simplified
- Potential bugs
- Dead code
- Duplicate code (via the included Copy-Paste-Detector, which also works for Python, C-languages, Ruby, PHP and many other languages)
PMD has many rules for specific bad practices, allowing you to make custom style guides. Moreover, PMD has an option to create totally custom rules, which is outside the scope of this guide, but can make it very powerful for any advanced Java course.
After choosing the Code Quality test in your AutoTest category, you can simply select a linter from the dropdown list. Choose Checkstyle or PMD for your Java assignment.
After doing so, you can select your style guide. You can upload and use a custom one, or simply use one of the default style guides available. For Checkstyle, we recommend the very common Google style guide, based on the style guide universally used at Google with hard-and-fast rules, or Sun style guide, a more traditional style guide enforced by Oracle / Sun that is last revised in 1999, which can be downloaded here. For PMD, we recommend Apache Maven’s style guide, which can be downloaded here.
After selecting your style guide and uploading this as a $FIXTURE, you can optionally provide extra arguments for the selected tool and update the penalties for the different message severity levels. If you wish to just inform your students of style and not grade them, these penalties can all be set to 0%.
Basic Java Code Quality (linter) set up in CodeGrade's autograder I
CodeGrade integrates a tool called Semgrep to make it easy to perform more complex code analysis by allowing you to write rules in a human readable format. You can provide generic or language specific patterns, which are then found in the code. With its pattern syntax, you can find:
- Equivalences: Matching code that means the same thing even though it looks different.
- Wildcards / ellipsis (...): Matching any statement, expression or variable.
- Metavariables ($X): Matching unknown expressions that you do not yet know what they will exactly look like, but want to be the same variable in multiple parts of your pattern.
Semgrep is pre-installed in the Unit Test. You will write your Semgrep Code Structure tests in a YAML file and upload this as a fixture in the setup section of AutoTest. Semgrep has an online editor that can be used to check and create your patterns:
For instance, this is a ruleset to detect too many nested if statements.
rules:
- id: nested-if
match-expected: false
pattern: |
if (...) {
...
if (...) {
...
if (...) {
...
}
}
}
message: No more than 2 nested if-statements were used.
severity: INFO
languages:
- java
We use the ellipses (the
… in the pattern) here, so that any code can be in the condition and inside the blocks of the if-statements. As an if-statement itself can also be part of this, we automatically catch code that has more than three nested if-statements with this pattern too. Save this in a file named for instance rules.yml.- Patterns
- The ellipsis (
...) is used to capture anything - metavariables
$EL(element) and$LST(list) capture the two parts of the for-loop declaration (the naming of these metavariables is irrelevant and could have been anything else). - The same is done for the while loop condition
$COND
- Messages We are able to provide understandable messages that are parsed by the wrapper script making our tests understandable for our students.
- Match-expected Importantly, we have added the match-expected field (this is added in the CodeGrade wrapper script and cannot be tested in regular semgrep), with putting this field to
Truefor the for-loop rule, we specify that we are expecting a match in order to pass that test. - Severity This dictates the level of severity of failing the rule. The penalty for each severity level can be set in the Unit Test step.
- languages Here we specify the language of the scripts which semgrep will be checking.
- 1.Upload the YAML (.yml) with your Code Structure tests in it as a $FIXTURE in the CodeGrade setup.
- 2.In the category where you want the test to live create a Unit Test.
- 3.Select semgrep.
- 4.Give the follow extra argument:
$FIXTURES/your-test-file.yml $STUDENT.
If you run this on your student's code, it will then show up like this:
- Capture Points Tests: Use your own custom grading script, upload it as a
$FIXTURE, execute it in this test and make sure it outputs a float between 0.0 and 1.0, this score will then be the number of points a student receives. - Checkpoints: Only execute tests below the checkpoint if the tests above the checkpoint reach a certain score. Great for separating simple and advanced tests.
Once you have created a configuration, press start and your AutoTest will start running. In the General Settings, you should have already uploaded a Test Submission, so you will see the results of this straight away.
Once it's started and your assignment is set to Open or Done, students can hand in and get immediate feedback!