Automatically grading assignments with random outputs

Learn how you can seed or replace the random variable assignments in your students' code.
It can be tricky to carry out input/output testing with random variables, (a common requirement for beginner Python assignments!) but we have two simple solutions for this common dilemma!
This is a summary of a guide posted on our blog, check out the full blog post here:

Setting up an example assignment

First of all, we need to design an assignment which requires our students to generate a random integer. The assignment I have created asks students to make a simple dice roll simulator. All that is required is that they:
  • Import random. For instance:from random import randint or randrange).
  • Generate two random integers in the range from 1-6.
  • Print out “You rolled a <number>” for each die.
  • Ask the user whether they would like to roll again.
  • Terminate the program when the user does not want to roll again.
Following these requirements we end up with the following code:
import random
# Constants for the minimum and maximum values of the random variables
MIN = 1
MAX = 6
def main():
again = ‘y’
While again = ‘y’ or again = ‘Y’:
print (‘rolling the dice…’)
print (‘their values are:)
print (random.randint(MIN,MAX))
print (random.randint(MIN,MAX))
again = input (‘Would you like to roll again? (y = yes):)
if __name__ == '__main__':
‍ A simple assignment like this is best tested with an Input/Output test in CodeGrade. However, because of the random numbers generated in the program, the output is unpredictable and we cannot expect a consistent output.

Seeding random numbers

One simple work-around is to use the seed function in Python’s built-in module, random. The seed function allows you to generate a consistent set of pseudo-random numbers. As long as the seed string and the commands used to generate the numbers after seeding are always the same, the numbers generated will also always be the same. The set of numbers generated will also be the same across all devices so you can expect the output you generate on your local machine to be the same as the output generated in CodeGrade.
We can apply this to our CodeGrade assignment by running the following command in the Run program field of an IO Test step in AutoTest:
python3 -ic ‘import random; random.seed(<seed string>); import; dice.main()’
Because we know, for instance, that the seed string ‘100’ will produce the same sequence of numbers when random.randint(1,6) is called, we can then write this sequence in the expected output. The first four numbers produced with the seed string ‘100’ are 2, 4, 4 and 2.
In our case, we can make two tests:
  1. 1.
    A simple test to check that the program produces two random numbers
  2. 2.
    A more complex test to see if we can make multiple rolls.
We expect our student’s program to automatically roll the dice when the program is run (without providing any input). Therefore, in our first test we don’t need to enter any inputs and we just enter the first two numbers of the seed sequence in the expected output field. For our second test, we need to tell the program that we want to roll again. Therefore, we enter y in the input field and in the expected output field we can enter the first 4 numbers in the seed sequence.
Note: Because we added the -i flag in our Program to the run command, we can interact with the Input field as we would with the Python interpreter.

Create substitute ‘random’ module

Seeding is a simple and effective solution in most cases, but occasionally it may be necessary to have more control over the numbers generated by the student’s script. Say for instance we need our students’ scripts to produce the same number multiple times.
For this situation, we can employ a simple hack where we create a python file called which will act as a stand-in for the built-in random python module. In our python file we can define the two functions randint and randrange which always return a predefined number ensuring that the output of our students’ programs is always the same.
The file should look something like this:
constant = 4
def randint(x, y):
return constant
def randrange(x, y):
return constant
Now that we have our substitute ‘random’ module, we need to upload it as a fixture in the setup portion of CodeGrade’s AutoTest. To make our lives easier when creating our tests we can then move this file from the $FIXTURES directory to the $STUDENT directory where our student’s tests will be run. We can do this easily in the Per-student setup script with the command:
Next, we can begin setting up our tests. Create an IO Test step and, in the Program to run field, run your students code with python3 Since we already know that the outcome of the dice simulator will provide two rolls each with the value of 4, we can then enter this into the Expected outcome field.
Note: We can constrain our students’ submissions using the Hand-in requirements in the General tab of the assignment management menu. There we can specify that students may only submit the file and nothing else.

Streamlining the workflow by using environment variables

An assignment as simple as the example provided above may not need more than one number to be produced but, In certain cases, it may be desirable to test the outcome of a program when the random numbers produced are different. In that instance, Linux environment variables can be used in combination with the surrogate ‘random’ module so that multiple IO tests can be conducted without needing to upload a different file for each desired output. We should, firstly, adapt our script accordingly:
import os
num = int(os.getenv(‘RANDNUM’))
def randint(x, y):
return num
def randrange(x, y):
return num

Next steps

IO tests are great for short, simple assignments but more complex assignments with multiple processes or functions become harder to test this way. CodeGrade also offers a wide variety of ways to evaluate your students’ python code including unit testing, code structure testing and code quality testing. Check out our other guides to see how you can assess python code with these methods in CodeGrade: