CodeGrade AutoTest runs on Ubuntu (18.04.2 LTS) machines which you can configure in any way that you want.
In the setup section of your AutoTest, you can upload any files you might need for testing. These files are called Fixtures and will be placed in the
$FIXTURESdirectory on the Virtual Server.
The file structure of the server is like this:
All of your uploaded fixtures will be here.
This is where the submission of student is placed.
Tests are executed from this directory.
After uploading any files you might need, you can run some setup to install any packages you might need.
- Global setup script: this is where you install your additional pip packages you want to be available on the server, this is where you can install any R packages. The global setup only runs once and is then cached, so the student submission won't be available here yet.
- Per-student setup script: this will run once when running the autograding for a student and will run before all tests. This is a good place to move
$STUDENTdirectory in case the student solution needs some input files to run correctly.
Use any command in the Global Setup Script field to install software or run your setup script
- 1.Create the following script in a file called
setup.Rand upload this as a fixture.UserLib=Sys.getenv("R_LIBS_USER")dir.create(UserLib, recursive=TRUE)install.packages(c("tidyr"), lib=UserLib)
- 2.In the global setup script, run the following command:
Now that you have created the setup, it's time to create the actual tests. Do this by pressing the "Add Level" button and then the "Add Category" button.
All tests in AutoTest fill in a specific rubric category that you select. After selecting the category, you can start creating tests that will fill it in. Multiple test types are available in CodeGrade, which can be used together depending on your needs and wishes.
We will use CodeGrade's I/O test step to test the
fibonaccifunction in our students' code. The steps to set this up are as follows:
- Create an I/O test and Run Rscript We can create an I/O test and run Rscript with some arguments. With the
-eflag we can provide Rscript expressions to execute. In the first instance we provide the source for our students' script,
fibonacci.R. In the second instance it is used to append the input argument of our I/O test:
Rscript -e "source(fibonacci.R')" -e
- Enter the Input Enter
fibonacci(5)which will tell Rscript that we want the students' function to return the first 5 numbers of the fibonacci sequence.
- Enter the output We can then compare the output of this function with our expected output which we set to  1 1 2 3 5.
CodeGrade I/O automatic R script function test
It is also possible to autograde entire R scripts using CodeGrade, either using standard input or by specifying exactly what the scripts should produce (without input). In this example, we tell the students that their
fibonacci.Rscript should always print the first 10 numbers of the fibonacci sequence.
This can again be tested with a very simple I/O Test in CodeGrade. This time, we just have to run
Rscript fibonacci.R. This runs the R script the student has uploaded. Since we do not need to give any input, we can leave the input arguments and input fields empty and just define an expected output, which is  1 1 2 3 5 8 13 21 34 55 in our case.
CodeGrade I/O automatic script test
- Run Program Tests: Run a command and if the command runs successfully this test will pass. Great for custom tests and compile commands.
- Unit Tests: Upload a unit test file as a
$FIXTUREand automatically run unit tests on the code. We have several built-in options, follow one of the automatic grading guides above to find out more!
- Code Quality Tests: Automatic run static code analysis on the student's code. We have several built-in options, follow one of the automatic grading guides above to find out more!
- Capture Points Tests: Use your own custom grading script, upload it as a
$FIXTURE, execute it in this test and make sure it outputs a float between 0.0 and 1.0, this score will then be the number of points a student receives.
- Checkpoints: Only execute tests below the checkpoint if the tests above the checkpoint reach a certain score. Great for separating simple and advanced tests.
Once you have created a configuration, press start and your AutoTest will start running. In the General Settings, you should have already uploaded a Test Submission, so you will see the results of this straight away.