Levels & Categories
Levels and Categories are the structure and hierarchy of AutoTest, which allow you to group tests that are similar or of equal difficulty.


Every AutoTest contains of one or multiple levels, which all hold one or multiple categories. These Categories are connected to a Rubric Category and contain the actual tests (or steps). Levels are executed in order and can be used to display increased difficulty of tests.
The Categories within a Level are executed independent of each other and could be executed concurrently. This means that there is no order between Categories within a level (the steps within a Category are executed in order).
For most use cases one level is enough to do all your testing with.

Advanced use cases

For most use cases, one level is enough. There is the possibility, however, to create multiple levels. This can be done, for example, if you only want to run certain categories if other categories (in another level) get a certain percentage of points. After adding a new level you can set this percentage in the footer of the previous level above. As levels are executed in order, and not concurrently, using multiple levels can show advancement in the tests.


Within a level you can create one or multiple categories. Each Category has to be connected to a Rubric Category, the steps added to a category will fill in this selected rubric category. Categories run independently of each other and always start with a fresh setup, steps within a category are executed in order and keep state.
By connecting rubric and Autotest, we encourage teachers to create meaningful rubric categories and think about how to group their tests accordingly. By grouping steps into autoTest categories that are connected to a rubric category, this also means that students know where tests belong and how they got their grade for an assignment.
Each Category starts with a fresh snapshot of the environment setup. This means that Categories run independently of each other.


Categories hold one or multiple steps, the scores of these steps fill in the rubric category. Steps within a category are executed in order and keep state between each other (e.g. if you compile code in one step, it will still be available in the next).
Each step has a weight, which is used to calculate the final percentage that is the score of the category. This final score is simply the percentage of points (weight) that was gotten divided by the total combination of weights.
Finally, you can also toggle the visibility of all tests. By default, all steps are visible. When you hide a step, the name is still visible, but its details aren’t. Also, when you’re using Continuous Feedback, hidden steps won’t be run before the deadline. AutoTest will automatically run all hidden steps within 30 minutes after the deadline.
It is good practice to hide performance heavy tests, so that these are not run in the continuous feedback and thus do not slow down the instant feedback to students.

Advanced options

Each category has a few settings that will be active for all steps within the category.
  • Custom description: Under the Description Template you can replace the default, automatically generated description (which describes what command(s) are executed in the test), with your own custom description. Multiple substitution variables are available to add variable information about the test, commands and points in your description.
  • Timeout per step: When a single step in the category is taking longer than this amount of seconds, the process is killed, the status of the test will be set to timeout and no points are given for this step.
It is preferred to keep this value low - while giving the student's code enough time to run - so that code that contains an accidental infinite loop will not fill up the AutoTest queue.
  • Network disabled: Disable the network while the steps in this category are running.
It is good practice to disable network access for assignments or categories in which students do not need network access.
  • Submission information environment variables: Include information about the submission being tested in the $CG_INFO environment variable. The value is a JSON object with the following keys:
    • deadline The deadline of this assignment, in ISO 8601 format.
    • submitted_at The date and time the student submitted their
      work, in ISO 8601 format.
    • result_id An identifier unique to this AutoTest result. This
      value changes every time the AutoTest is run, even if it is run
      multiple times for the same submission of the same student.
    • student_id An identifier unique to the student for which the
      AutoTest is run. This value stays constant between runs of
      different submissions by the same student.
Submission info allows you to automatically subtract points for late days or to use as seed data when randomising input (either to keep this stable over multiple submissions of the same student or to always change this).
Last modified 10mo ago
Copy link