R

Overview
An AutoTest V2 configuration for a R assignment has two stages:
The Setup stage: this is where you setup the auto grading environment. You can install packages and upload files. This builds into an image on which all tests are executed. All the configuration here only runs once for all submissions;
The Tests stage: this is what students will see and where you can configure all your tests that you want to run on every student’s code. Everything that you specify here is executed in the working directory of the student.
Steps are created in a “Scratch-like” block layout and can be dragged and dropped, reordered and nested to create an AutoTest program. We are continuously adding new blocks to AutoTest V2 to make it more powerful and easier to use.
AutoTest V2 is still available next to our original AutoTest, to start setting up an AutoTest V2 select AutoTest V2 when creating a new AutoTest.

Already have an original AutoTest? You can switch to AutoTest V2 by deleting it and pressing "Select another version of AutoTest" to finally select AutoTest V2. Please note that your original AutoTest will be deleted when pressing the "Delete" button!
Developing, snapshots and publishing to students
When developing your AutoTest V2 configuration, you can continuously test your configuration on the "Test Submission".
After configuring something, you press the “Build Snapshot” button in the bottom right corner of the screen. This will Build your AutoTest into a Snapshot (a history of your snapshots are available via the Snapshots tab).
After pressing "Build Snapshot", you can:
Test the configuration by seeing the results in seconds.
If you are ready to publish your AutoTest to your students press the big green Publish button.
If you make any changes, you build again and if you are satisfied, you can republish them to your students.
If you want to unpublish your snapshot, you can simply go to it in the green bar and press the red “Unpublish” button.
Step 1: Setup
CodeGrade AutoTest V2 runs on Ubuntu (20.04 LTS) machines which you can configure in any way that you want. Common software is pre-installed, and you can very easily install any version of R yourself.
In the setup section of your AutoTest, you can install software or packages and upload any files you might need for testing. The setup section will build an image and is only run once for all submissions. 
You can upload files using the "Upload Files" block, if you want to for instance include a csv dataset. These files will be placed in the $UPLOADED_FILES directory on the Virtual Server.
You can install software and packages (or configure the build image in any other way) using the "Script" block. You have network access by default in the Setup tab, so you may also download libraries you want to use.
Installing R
1.In the setup step, create a new "Script" block. In this script block, use the following commands to install R:
sudo apt-get update
sudo apt-get install r-base r-base-dev
2.The above installs R 3.X by default, consult the official R website for more info on R versions. At the time of writing, adding these lines at the start of your "Script" block will add the correct repository for R 4.X:
wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc | sudo tee -a /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc
sudo add-apt-repository "deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/"
Installing R packages
1.Create the following script in a file called setup.R and upload this using a "Upload Files" block:
UserLib=Sys.getenv("R_LIBS_USER")
dir.create(UserLib, recursive=TRUE)
install.packages(c("tidyr"), lib=UserLib)
2.In a "Script" block run the following command: Rscript $UPLOADED_FILES/setup.R.
Basic set up for an autograded R assignment in AutoTest V2
Step 2: Create your tests
Now that you have created the setup, it's time to create the actual tests. This is done on the Tests tab. Select one of the many test-blocks to configure a AutoTest V2 procedure in a "scratch"-like format. Some tests can be nested and you can chose to connect tests to rubric categories with a certain weight, you can also hide aspects of tests and enable internet access for specific tests.
Autograding specific functions in R
We will use CodeGrade's "IO Test" block to test the fibonacci function in our students' code. The steps to set this up are as follows:
1.Create an IO test and Run Rscript: We can create an IO Test block and run Rscript with some arguments. With the -e flag we can provide Rscript expressions to execute. In the first instance we provide the source for our students' script, fibonacci.R . In the second instance it is used to call the function and provide an input argument.

Rscript -e "source('fibonacci.R')" -e "fibonacci(5)"
2.Enter the output: We can then compare the output of this function with our expected output. Add a "Substring Match" or "Full Match" block, leave the input empty and simply set the expected output to [1] 0 1 1 2 3.
3.Optionally add a "Connect to Rubric" block to connect your test to a rubric and use to grade the student.
CodeGrade I/O automatic R script function test
Autograding R scripts
It is also possible to autograde entire R scripts using CodeGrade, either using standard input or by specifying exactly what the scripts should produce (without input). In this example, we tell the students that their fibonacci.R script should always print the first 10 numbers of the fibonacci sequence.
This can again be tested with a very simple IO Test block in CodeGrade. This time, we just have to run Rscript fibonacci.R. This runs the R script the student has uploaded. Since we do not need to give any input, we can leave the input arguments and input fields empty and just define an expected output, which is 0 1 1 2 3 5 8 13 21 34 in our case. Again, both a "Substring Match" or "Full Match" block can be used inside the IO Test block. 

Optionally add a "Connect to Rubric" block to connect your test to a rubric and use to grade the student.
CodeGrade IO automatic script test
Set up a Code Structure test
CodeGrade integrates a tool called Semgrep to make it easy to perform more complex code analysis by allowing you to write rules in a human readable format. You can provide generic or language specific patterns, which are then found in the code. With its pattern syntax, you can find:
Equivalences: Matching code that means the same thing even though it looks different.
Wildcards / ellipsis (...): Matching any statement, expression or variable.
Metavariables ($X): Matching unknown expressions that you do not yet know what they will exactly look like, but want to be the same variable in multiple parts of your pattern.
Writing the tests
Semgrep can be installed in the Setup step using pip, use: pip3 install semgrep. The easiest way to use Semgrep in AutoTest V2 is via our structure.py script (this will be built-in in a later version). With this script, you will write your Semgrep patterns directly in the Input field in an IO Test. Try out Semgrep patterns using their online editor here:
Semgrep
For instance, this is a ruleset to detect if a student has a fibonacci function.
pattern: |
    fibonacci <- function(n) {
        ...
    }
We use the ellipses (the … in the pattern) here, so that any code can be in the function.
Creating the test in CodeGrade
structure.py
2KB
Text
Download the structure.py script to run Semgrep in AutoTest V2.
1.Under the Setup tab, upload the structure.py file you can download above. Also, in a "Script" block, install Semgrep via pip3 install semgrep.
2.Continue in the Tests tab: first drag the "IO Test" block to your Test Procedure. Execute the script in the code field, follow: python3 $UPLOADED_FILES/structure.py <STUDENTFILE>. For a Fibonacci assignment, this could be: python3 $UPLOADED_FILES/structure.py fibonacci.R.
3.Create and drag one or more "Simple Test" block(s) inside your "IO Test" block. Each block is one Semgrep structure to check. As Input write the Semgrep pattern (including pattern: or similar like pattern-either:).
4.As Expected Output write "Found!" if the student code should match or "Not found!" if the student code should not contain the pattern.
5.Optionally: Use a "Connect Rubric" block to connect your structure check to a rubric category.
6.Optionally: Use a "Hide" block to hide the "config" (i.e. input / pattern) of your tests so that students cannot see the pattern you are checking for.
A Semgrep structure check in an IO Test in AutoTest V2 (with optional Hide block to hide the pattern).
CodeGrade's AutoTest V2 supports many more types of tests, from linters to unit testing to custom test scripts. Further more, you can use specific blocks to hide, weight and grade tests. Want to set up a more advanced R autograder? Reach out to us via [email protected]. 
Step 3: Start the AutoTest
After you have created a configuration, create a Snapshot to test it on a "Test Submission". In the General Settings, you should have already uploaded a Test Submission, so you will see the results of this straight away.
Once everything works as expected, press the green "Publish to students" button to publish your AutoTest V2 to students. If they use the editor they can run it instantly, and if they hand in in any other way they will get their AutoTest results instantly after any submission.
Automatic Grading Guides - Previous
Javascript
Autograding R using AutoTest V1
Last modified 3mo ago