Writing Code Quality wrapper scripts
CodeGrade offers an ever expanding list of code analysis tools that are built in to our Code Quality step. In this guide we explain how you can add your own by writing a very simple wrapper script.
CodeGrade's Code Quality step can be used to automatically assess code structure, code style or perform any other static code analysis, and give this feedback right in the lines of code. For most use cases, our built in code analysis tools will be more than sufficient. Some advanced assignments however ask for specific code analysis tools. How to integrate these tools with CodeGrade by writing a custom wrapper script will be explained in this guide. Want to learn more about our Code Quality step? Read the guide below!
In this guide we will go over how we created the
pylint
wrapper script for the Code Quality step in AutoTest to check Python code for common programming mistakes and conventions. We will write the script in Python because it's a bit more flexible and powerful than a shell script, and also because AutoTest runners have the cg_at_utils
library installed, which is a Python package that provides utility functions for writing wrapper scripts like this one.Code Quality wrapper scripts receive the command line arguments you specify in the Custom program field when creating the AutoTest step. In this script we will accept a list of arguments that we will pass on to
pylint
.When a wrapper script exits with an error code (anything not equal to
0
) the step will be marked as "failed" and no points will be given to the student, regardless of the number of comments that were placed.We use the
typer
library to help defining the command line arguments to your script, which are automatically inferred from the types of the arguments of the functions you define.We use Python for our script so that we can use the CodeGrade library called
cg_at_utils.
This library is available automatically for all Python scripts in CodeGrade AutoTest and offers many functions to make your life easier.import sys
import json
import typing as t
import subprocess
â
import typer
import cg_at_utils.utils as utils
from cg_at_utils.comments import put_comment
â
â
@app.command()
def main(
pylint_args: t.List[str] = typer.Argument(
None, help='Arguments to pass to pylint'
),
): """Run pylint with the given arguments."""
# TODO
â
â
if __name__ == '__main__':
app()
We use Python's
subprocess
module to run pylint
with the arguments we were given and capture its output. We also pass --output-format json
so that pylint
will output a JSON list containing an object for each comment it produced. Most code analysis tools have flags to format the output in an easy to handle format, you can find these in the documentation of the tool you choose to implement.proc = subprocess.run(
['pylint', '--output-format', 'json', *pylint_args],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
encoding='utf8',
)
Once
pylint
is finished, we check its exit code. pylint
will exit with code 32
if it was not able to run, for example due to a configuration error or invalid command line arguments being passed. In this case we print "Pylint crashed:" followed by the output produced by pylint
and exit the program with code 32
. This printed output will then be made visible in the Error Output Tab of the step in CodeGrade.if proc.returncode == 32:
print('PyLint crashed:\n', proc.stdout, file=sys.stderr)
utils.exit(32)
pylint
will exit with a 1
if it could not process the files it was given, which can happen when the code to check is not a valid Python module because it does not contain an __init__.py
file. This error is not specific to any file in particular, so we cannot show it inline in the code viewer. If this happens we print a message stating that pylint
could not run, and exit with a 1
.The above edge case is specific to the
pylint
code analysis tool and can be different for the tool of your choice. It is a good practice to add conditions for these edge cases upfront, but by thoroughly testing your AutoTest you will be able to catch them and fix them on the go too.if proc.returncode == 1:
print(
'The submission is not a valid python module, it probably lacks'
' an `__init__` file.',
file=sys.stderr,
)
utils.exit(1)
We can now process
pylint
's output by parsing the output as JSON and looping through the list of comments, passing each of them through a handle_comment
function that we will define later on.comments = [handle_comment(comm) for comm in json.loads(proc.stdout)]
Finally, to post the comments back to CodeGrade so they can be shown in the Code Viewer, we use the
put_comment
function provided by the cg_at_utils
library. It expects a dict with the following keys:op
: The operation to perform. Right now the only supported operation isput_comments
.comments
: The list of processed comments. We will discuss the format of acomment in the next section.ignore_files_not_found
: Whether to ignore comments on files that do notexist in the submission of the student or produce an error. These comments canoccur, for example, when you have copied some test files for a "Unit Test"step in your AutoTest setup script. In most cases setting this toTrue
isrecommended.
put_comment({
'op': 'put_comments',
'comments': comments,
'ignore_files_not_found': True,
})
Code Quality comments for CodeGrade (the
comments
key for the put_comment
function) must adhere to the following format:{
# The linter that produced this comment
'origin': str,
â
# An identifier for this error message defined by the linter
# (can be None or str)
'code': Optional[str],
â
# The severity of this comment, one of
# 'info', 'warning', 'error', or 'fatal'
'severity': CommentSeverity,
â
# A helpful message describing the issue with the code
'msg': str,
â
# The line(s) on which the comment should be placed, where
# 1 indicates the first line of the file
'line': {
'start': int,
'end': int,
},
â
# The column(s) on which the comment should be placed, where
# 1 indicates the first column in a line (column information
# is not yet used by CodeGrade, but may be in the future)
# end can be None or int
'column': {
'start': int,
'end': Optional[int],
},
â
# The path of the file on which this comment was placed
'path': List[str],
}
Luckily, this is already very similar to
pylint
's output, so our customhandle_comment
function to translate the comments from pylint
to CodeGrade's format above is fairly straightforward. Firstly, we get the severity and convert it into one that CodeGrade understands, because pylint
's severity levels do not completely correspond: it can output convention
or refactor
, both of which we choose to map to info
(but of course, this mapping is up to your preferences). All other severities, warning
, error
and fatal
, already map correctly.def handle_comment(comment):
severity = comment['type']
if severity in ('convention', 'refactor'):
severity = 'info'
Then we return a
dict
containing all the necessary information that we copy over from the input dict: return {
'origin': 'PyLint',
'msg': comment['message'],
'code': comment['symbol'],
'severity': severity,
'line': {
'start': comment['line'],
'end': comment['line'],
},
'column': {
# Pylint 0-indexes columns
'start': comment['column'] + 1,
'end': None,
},
'path': utils.path.split(comment['path']),
}
Most of these fields correspond very easily with the output of our linter. Two things to note:
- We use the same
comment['line']
for both the start and the end line of the message. For most tools,pylint
included, this is sufficient as the output messages are for singular lines of code. If your tool produces messages for blocks of code, you should take that into account here. - We split the file path into a list with
utils.path.split
from thecg_at_utils
library. This function takes double path separators in consideration, for example, and is more likely in general to produce the correct splitting of the path.
After combining all snippets discussed above, we get to our final wrapper script. We can now upload this as a fixture and run it in our Code Quality step by using the custom script option.â
#!/usr/bin/env python3
â
import sys
import json
import typing as t
import subprocess
â
import typer
import cg_at_utils.utils as utils
from cg_at_utils.comments import put_comment
â
â
def handle_comment(comment):
"""Convert a pylint message to a message that can be understood by
CodeGrade.
"""
â
severity = comment['type']
â
# Pylint has 4 message types: 'error', 'warning', 'convention', and
# 'refactor'.
if severity in ('convention', 'refactor'):
severity = 'info'
â
return {
'origin': 'PyLint',
'msg': comment['message'],
'code': comment['symbol'],
'severity': severity,
'line': {
'start': comment['line'],
'end': comment['line'],
},
'column': {
# Pylint 0-indexes columns
'start': comment['column'] + 1,
'end': None,
},
'path': utils.path.split(comment['path']),
}
â
â
app = utils.cli.make_typer(name='my-pylint')
â
â
@app.command()
def main(
pylint_args: t.List[str] = typer.Argument(
None, help='Arguments to pass to pylint'
),
):
"""Run pylint on the given files."""
â
proc = subprocess.run(
['pylint', '--output-format', 'json', *pylint_args],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
encoding='utf8',
)
â
if proc.returncode == 32:
print('PyLint crashed:\n', proc.stdout, file=sys.stderr)
utils.exit(32)
â
if proc.returncode == 1:
print(
'The submission is not a valid python module, it probably lacks'
' an `__init__` file.',
file=sys.stderr,
)
utils.exit(1)
â
comments = [handle_comment(comm) for comm in json.loads(proc.stdout)]
â
put_comment({
'op': 'put_comments',
'comments': comments,
'ignore_files_not_found': True,
})
â
â
if __name__ == '__main__':
app()
Last modified 9mo ago