Writing Code Quality wrapper scripts
CodeGrade offers an ever expanding list of code analysis tools that are built in to our Code Quality step. In this guide we explain how you can add your own by writing a very simple wrapper script.
CodeGrade's Code Quality step can be used to automatically assess code structure, code style or perform any other static code analysis, and give this feedback right in the lines of code. For most use cases, our built in code analysis tools will be more than sufficient. Some advanced assignments however ask for specific code analysis tools. How to integrate these tools with CodeGrade by writing a custom wrapper script will be explained in this guide. Want to learn more about our Code Quality step? Read the guide below!
pageCreating Code Quality TestsIn this guide we will go over how we created the pylint
wrapper script for the Code Quality step in AutoTest to check Python code for common programming mistakes and conventions. We will write the script in Python because it's a bit more flexible and powerful than a shell script, and also because AutoTest runners have the cg_at_utils
library installed, which is a Python package that provides utility functions for writing wrapper scripts like this one.
The skeleton of a wrapper script
Code Quality wrapper scripts receive the command line arguments you specify in the Custom program field when creating the AutoTest step. In this script we will accept a list of arguments that we will pass on to pylint
.
When a wrapper script exits with an error code (anything not equal to 0
) the step will be marked as "failed" and no points will be given to the student, regardless of the number of comments that were placed.
We use the typer
library to help defining the command line arguments to your script, which are automatically inferred from the types of the arguments of the functions you define.
We use Python for our script so that we can use the CodeGrade library called cg_at_utils.
This library is available automatically for all Python scripts in CodeGrade AutoTest and offers many functions to make your life easier.
Running pylint
on the given files
pylint
on the given filesWe use Python's subprocess
module to run pylint
with the arguments we were given and capture its output. We also pass --output-format json
so that pylint
will output a JSON list containing an object for each comment it produced. Most code analysis tools have flags to format the output in an easy to handle format, you can find these in the documentation of the tool you choose to implement.
Once pylint
is finished, we check its exit code. pylint
will exit with code 32
if it was not able to run, for example due to a configuration error or invalid command line arguments being passed. In this case we print "Pylint crashed:" followed by the output produced by pylint
and exit the program with code 32
. This printed output will then be made visible in the Error Output Tab of the step in CodeGrade.
pylint
will exit with a 1
if it could not process the files it was given, which can happen when the code to check is not a valid Python module because it does not contain an __init__.py
file. This error is not specific to any file in particular, so we cannot show it inline in the code viewer. If this happens we print a message stating that pylint
could not run, and exit with a 1
.
The above edge case is specific to the pylint
code analysis tool and can be different for the tool of your choice. It is a good practice to add conditions for these edge cases upfront, but by thoroughly testing your AutoTest you will be able to catch them and fix them on the go too.
We can now process pylint
's output by parsing the output as JSON and looping through the list of comments, passing each of them through a handle_comment
function that we will define later on.
Finally, to post the comments back to CodeGrade so they can be shown in the Code Viewer, we use the put_comment
function provided by the cg_at_utils
library. It expects a dict with the following keys:
op
: The operation to perform. Right now the only supported operation isput_comments
.comments
: The list of processed comments. We will discuss the format of acomment in the next section.
ignore_files_not_found
: Whether to ignore comments on files that do notexist in the submission of the student or produce an error. These comments can
occur, for example, when you have copied some test files for a "Unit Test"
step in your AutoTest setup script. In most cases setting this to
True
isrecommended.
Handling the output of the code analysis tool
Code Quality comments for CodeGrade (the comments
key for the put_comment
function) must adhere to the following format:
Luckily, this is already very similar to pylint
's output, so our customhandle_comment
function to translate the comments from pylint
to CodeGrade's format above is fairly straightforward. Firstly, we get the severity and convert it into one that CodeGrade understands, because pylint
's severity levels do not completely correspond: it can output convention
or refactor
, both of which we choose to map to info
(but of course, this mapping is up to your preferences). All other severities, warning
, error
and fatal
, already map correctly.
Then we return a dict
containing all the necessary information that we copy over from the input dict:
Most of these fields correspond very easily with the output of our linter. Two things to note:
We use the same
comment['line']
for both the start and the end line of the message. For most tools,pylint
included, this is sufficient as the output messages are for singular lines of code. If your tool produces messages for blocks of code, you should take that into account here.We split the file path into a list with
utils.path.split
from thecg_at_utils
library. This function takes double path separators in consideration, for example, and is more likely in general to produce the correct splitting of the path.
The entire wrapper script
After combining all snippets discussed above, we get to our final wrapper script. We can now upload this as a fixture and run it in our Code Quality step by using the custom script option.
Last updated