To make CodeGrade AutoTest as flexible as possible, it is possible to customize the complete environment the tests are run in. Each separate assignment that makes use of AutoTest runs on its own Virtual Server in the cloud, to which you have superuser rights and network access during the setup phase. Every server on runs with the latest LTS version of Ubuntu, which is Ubuntu 18.04.2 LTS.
After the Setup Phase is finished, a snapshot is created which is used to initialize the containers used to run the actual tests on the student submissions. The Setup Phase consists of the default installed software, fixtures and setup script.
There are multiple options for setting up your environment, which makes AutoTest easy to use for simple cases yet very flexible for all advanced cases.
Each AutoTest VPS environment comes with pre-installed software that is commonly used for testing and running student submissions. This body of software is sufficient for most cases, which allows you to skip manually further setting up the environment.
The following software is automatically installed in all environments, all versions, except for Python, are lower bounds, as all packages are always updated to the latest version shipped by Ubuntu:
Python 2.7 (with pip)
Python 3.6 (with pip3)
Python 3.7 (with pip3)
Java 8 (openjdk-8-jdk)
Java 11 (openjdk-11-jdk)
R (r-base 3.4)
C/C++ (gcc 7 and clang 6 as compilers)
Go (golang 1.10)
Numpy (for Python2, Python3.6 and Python3.7)
SciPy (for Python2, Python3.6 and Python3.7)
Check (unit test framework for C)
Read the following sections to find out about extending this environment with other required software.
In addition to the software listed above, CodeGrade provides custom wrapper scripts for popular unit testing frameworks to make integration with Autotest a breeze. See the Unit Test chapter for a list of supported unit testing frameworks.
When a student uploads, AutoTest immediately runs on the handed in submission. You can choose whether you want the student to directly be able to view the AutoTest result (continuous feedback) or that the student can only view the result when the assignment is set to "Done" (just like with any other feedback).
The AutoTest rubric categories will be filled in automatically and directly in either case if you have not created any hidden steps.
The rubric calculation setting determines how discrete rubric categories will be filled-in.
There are two modes to fill in discrete rubric categories:
Minimum: A category's item will be chosen when the lower bound
of this item is reached (e.g. when a category has 4 items and 75% of the tests succeed, the maximum item is filled in).
Maximum: A category's item will be chosen when the upper bound
of this item is reached (e.g. you need 100% passed tests to have the
maximum item filled in).
Continuous rubric categories are filled in one way, this means this setting will have no influence on rubric calculation for this type of rubric category. The percentage achieved in the AutoTest level will also be the percentage achieved in the continuous rubric category.
When the preferred revision is set to "Teacher", and a teacher's revision is available for a submission, AutoTest is run against the teacher revision instead of the code submitted by the student. If no teacher's revision is present AutoTest will be run against the code of the student.
After the teacher has made their changes, the AutoTest should be manually restarted if it has already started or finished, to make it run against the teacher's revision. You can restart an AutoTest by going to a result, clicking on the arrow next to the state of the result, and selecting "Restart this result".
AutoTest caching will cache the state of your AutoTest environment after your configuration (e.g. after downloading data, installing additional software or packages or unarchiving). When it is restarted, it can use this cached state to restart instantly and allow for very fast feedback to students. It is also very useful when iteratively developing your AutoTest, as changes in test steps will not require a cache refresh.
AutoTest caching is turned on by default and should be left turned on for most configurations. There are however some uncommon reasons to disable it. Learn more in our guide here:
Fixtures can be optionally uploaded to the AutoTest VPS. Fixtures are files you can upload prior to the test, which will be available in every separate test container. Use cases are files used as setup script, unit tests, custom software to run or install and test input.
Select the fixtures to be uploaded and submit these to upload. A list of previously uploaded fixtures can be found above the upload dialog and managed here too. Uploading fixtures with the same name as current fixtures will by default result in an additional copy with
(1), the 'Override duplicates' option can be used to overwrite current fixtures with the same name.
It is sometimes desirable to limit student access to fixtures or to limit the visibility of your uploaded fixtures. For instance if one of your fixtures is a solution to the assignment you use to test student submissions against. By default fixtures and their contents are visible to students. Toggle the eye icon next to the fixture to toggle its visibility to students inside the CodeGrade interface.
Next to that, we offer multiple means of further limiting undesirable student access to fixtures. Firstly, the path to the fixtures is randomly generated for each category and thus only accessible using the
$FIXTURES environment variable. This makes it harder for students to access the path, but not impossible.
A way to further limit student permissions in the
$FIXTURES folder is to execute student code with the
become_nobody command. When executed in this mode, students will have no permissions to read from the
$FIXTURES folder. They will have permissions in the
$STUDENT folder, which is the current directory in which student submission files are accessible, to read and execute.
A setup script can be specified which runs prior to the tests to customize the initial environment. Any script can be uploaded as fixture and subsequently run with the command given in the Global setup script to run input field.
This can be, for example, a bash script that installs software using apt and extracts archives, or compiles unit tests. Of course, one or multiple commands can also be given in the Global setup script to run directly.
If you need to setup or compile software for each student specifically and not globally, use the Per student setup script for this. Install any packages using the Global setup script as this will greatly increase the speed of AutoTest Runs
Use the per student setup script to compile, for example, each submission's code. This part is executed when setting up a student's individual environment and the current directory already holds the student's submission.