Writing Code Quality wrapper scripts

CodeGrade offers an ever expanding list of code analysis tools that are built in to our Code Quality step. In this guide we explain how you can add your own by writing a very simple wrapper script.

CodeGrade's Code Quality step can be used to automatically assess code structure, code style or perform any other static code analysis, and give this feedback right in the lines of code. For most use cases, our built in code analysis tools will be more than sufficient. Some advanced assignments however ask for specific code analysis tools. How to integrate these tools with CodeGrade by writing a custom wrapper script will be explained in this guide. Want to learn more about our Code Quality step? Read the guide below!

pageCreating Code Quality Tests

In this guide we will go over how we created the pylint wrapper script for the Code Quality step in AutoTest to check Python code for common programming mistakes and conventions. We will write the script in Python because it's a bit more flexible and powerful than a shell script, and also because AutoTest runners have the cg_at_utils library installed, which is a Python package that provides utility functions for writing wrapper scripts like this one.

The skeleton of a wrapper script

Code Quality wrapper scripts receive the command line arguments you specify in the Custom program field when creating the AutoTest step. In this script we will accept a list of arguments that we will pass on to pylint.

When a wrapper script exits with an error code (anything not equal to 0) the step will be marked as "failed" and no points will be given to the student, regardless of the number of comments that were placed.

We use the typer library to help defining the command line arguments to your script, which are automatically inferred from the types of the arguments of the functions you define.

We use Python for our script so that we can use the CodeGrade library called cg_at_utils. This library is available automatically for all Python scripts in CodeGrade AutoTest and offers many functions to make your life easier.

import sys
import json
import typing as t
import subprocess
​
import typer
import cg_at_utils.utils as utils
from cg_at_utils.comments import put_comment


@app.command()
def main(
    pylint_args: t.List[str] = typer.Argument(
        None, help='Arguments to pass to pylint'
    ),
):    """Run pylint with the given arguments."""
    # TODO


if __name__ == '__main__':
    app()

Running pylint on the given files

We use Python's subprocess module to run pylint with the arguments we were given and capture its output. We also pass --output-format json so that pylint will output a JSON list containing an object for each comment it produced. Most code analysis tools have flags to format the output in an easy to handle format, you can find these in the documentation of the tool you choose to implement.

proc = subprocess.run(
    ['pylint', '--output-format', 'json', *pylint_args],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    encoding='utf8',
)

Once pylint is finished, we check its exit code. pylint will exit with code 32 if it was not able to run, for example due to a configuration error or invalid command line arguments being passed. In this case we print "Pylint crashed:" followed by the output produced by pylint and exit the program with code 32. This printed output will then be made visible in the Error Output Tab of the step in CodeGrade.

if proc.returncode == 32:
    print('PyLint crashed:\n', proc.stdout, file=sys.stderr)
    utils.exit(32)

pylint will exit with a 1 if it could not process the files it was given, which can happen when the code to check is not a valid Python module because it does not contain an __init__.py file. This error is not specific to any file in particular, so we cannot show it inline in the code viewer. If this happens we print a message stating that pylint could not run, and exit with a 1.

The above edge case is specific to the pylint code analysis tool and can be different for the tool of your choice. It is a good practice to add conditions for these edge cases upfront, but by thoroughly testing your AutoTest you will be able to catch them and fix them on the go too.

if proc.returncode == 1:
    print(
        'The submission is not a valid python module, it probably lacks'
        ' an `__init__` file.',
        file=sys.stderr,
    )
    utils.exit(1)

We can now process pylint's output by parsing the output as JSON and looping through the list of comments, passing each of them through a handle_comment function that we will define later on.

comments = [handle_comment(comm) for comm in json.loads(proc.stdout)]

Finally, to post the comments back to CodeGrade so they can be shown in the Code Viewer, we use the put_comment function provided by the cg_at_utils library. It expects a dict with the following keys:

• op: The operation to perform. Right now the only supported operation is

  put_comments.

• comments: The list of processed comments. We will discuss the format of a

  comment in the next section.

• ignore_files_not_found: Whether to ignore comments on files that do not

  exist in the submission of the student or produce an error. These comments can

  occur, for example, when you have copied some test files for a "Unit Test"

  step in your AutoTest setup script. In most cases setting this to True is

  recommended.

put_comment({
    'op': 'put_comments',
    'comments': comments,
    'ignore_files_not_found': True,
})

Handling the output of the code analysis tool

Code Quality comments for CodeGrade (the comments key for the put_comment function) must adhere to the following format:

{
    # The linter that produced this comment
    'origin': str,

    # An identifier for this error message defined by the linter 
    # (can be None or str)
    'code': Optional[str],

    # The severity of this comment, one of 
    # 'info', 'warning', 'error', or 'fatal'
    'severity': CommentSeverity,

    # A helpful message describing the issue with the code
    'msg': str,

    # The line(s) on which the comment should be placed, where
    # 1 indicates the first line of the file
    'line': {
        'start': int,
        'end': int,
    },

    # The column(s) on which the comment should be placed, where
    # 1 indicates the first column in a line (column information
    # is not yet used by CodeGrade, but may be in the future)
    # end can be None or int
    'column': {
        'start': int,
        'end': Optional[int],
    },

    # The path of the file on which this comment was placed
    'path': List[str],
}

Luckily, this is already very similar to pylint's output, so our customhandle_comment function to translate the comments from pylint to CodeGrade's format above is fairly straightforward. Firstly, we get the severity and convert it into one that CodeGrade understands, because pylint's severity levels do not completely correspond: it can output convention or refactor, both of which we choose to map to info (but of course, this mapping is up to your preferences). All other severities, warning, error and fatal , already map correctly.

def handle_comment(comment):
    severity = comment['type']
    if severity in ('convention', 'refactor'):
        severity = 'info'

Then we return a dict containing all the necessary information that we copy over from the input dict:

    return {
        'origin': 'PyLint',
        'msg': comment['message'],
        'code': comment['symbol'],
        'severity': severity,
        'line': {
            'start': comment['line'],
            'end': comment['line'],
        },
        'column': {
            # Pylint 0-indexes columns
            'start': comment['column'] + 1,
            'end': None,
        },
        'path': utils.path.split(comment['path']),
    }

Most of these fields correspond very easily with the output of our linter. Two things to note:

• We use the same comment['line'] for both the start and the end line of the message. For most tools, pylint included, this is sufficient as the output messages are for singular lines of code. If your tool produces messages for blocks of code, you should take that into account here.

• We split the file path into a list with utils.path.split from the cg_at_utils library. This function takes double path separators in consideration, for example, and is more likely in general to produce the correct splitting of the path.

The entire wrapper script

After combining all snippets discussed above, we get to our final wrapper script. We can now upload this as a fixture and run it in our Code Quality step by using the custom script option.

#!/usr/bin/env python3
​
import sys
import json
import typing as t
import subprocess
​
import typer
import cg_at_utils.utils as utils
from cg_at_utils.comments import put_comment
​
​
def handle_comment(comment):
    """Convert a pylint message to a message that can be understood by
    CodeGrade.
    """
​
    severity = comment['type']
​
    # Pylint has 4 message types: 'error', 'warning', 'convention', and
    # 'refactor'.
    if severity in ('convention', 'refactor'):
        severity = 'info'
​
    return {
        'origin': 'PyLint',
        'msg': comment['message'],
        'code': comment['symbol'],
        'severity': severity,
        'line': {
            'start': comment['line'],
            'end': comment['line'],
        },
        'column': {
            # Pylint 0-indexes columns
            'start': comment['column'] + 1,
            'end': None,
        },
        'path': utils.path.split(comment['path']),
    }
​
​
app = utils.cli.make_typer(name='my-pylint')
​
​
@app.command()
def main(
    pylint_args: t.List[str] = typer.Argument(
        None, help='Arguments to pass to pylint'
    ),
):
    """Run pylint on the given files."""
​
    proc = subprocess.run(
        ['pylint', '--output-format', 'json', *pylint_args],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        encoding='utf8',
    )
​
    if proc.returncode == 32:
        print('PyLint crashed:\n', proc.stdout, file=sys.stderr)
        utils.exit(32)
​
    if proc.returncode == 1:
        print(
            'The submission is not a valid python module, it probably lacks'
            ' an `__init__` file.',
            file=sys.stderr,
        )
        utils.exit(1)
​
    comments = [handle_comment(comm) for comm in json.loads(proc.stdout)]
​
    put_comment({
        'op': 'put_comments',
        'comments': comments,
        'ignore_files_not_found': True,
    })
​
​
if __name__ == '__main__':
    app()