JSON object for automatic comments

We now have a generic cg command for parsing custom code quality tool outputs and placing automatically generated comments on students' code. Using that method you will not need to write your own parser to write to the JSON object. Check this page for more information:

pagecg comments generic

Should you choose to write your own parser, you can use this guide to write the parsed comments on your students code:

1. Run your test so that it outputs the required information for creating comments. The test could be a code quality test for instance.

2. Using an uploaded script or by coding directly in the editor of your test block, parse the test output into the required JSON object format as denoted below.

3. Write the JSON object to the 3 file. Here is a code snippet that illustrates how to write the JSON object for generating comments to the 3 file using Python:

Make sure the message isn't too long. Messages that are larger than 250kb will be silently dropped.

import json

with open(3, 'w') as f:
    f.write(json.dumps({
            'tag': 'comments',
            'comments': [{
                    'filename': '/home/codegrade/student/Calculator.java',
                    'origin': 'checkstyle',
                    'items': [{
                        'loc': {'start': {'line': 1, 'column': 1}, 'end': {'line': 1, 'column': 1}},
                        'severity': 'error',
                        'msg': 'Missing package-info.java file.'
                    }]
            }]
    }))

Comments JSON object format

• Tag (String) - This should always be "comments".

• Comments (list) - A list of comment objects for each file that should contain:

  • Filename (string) - the path to the file on which the comment should be placed. The directory where the students submissions are found by default is /home/codegrade/student/.

  • Origin (string) - This can be any string but usually indicates the type of test that was run to generate the comments. For example, "flake8".

  • Items (list) - A list of item objects that are the actual comments placed in the code. These items should contain:

    • Severity (string) - This must be on of four options: "error", "warning", "info", or "fatal".

    • Msg (string) - This contains the message in the comment.

    • Loc (obj) - An object containing the location of the message in the format:

      • Start (obj) - An object containing the start location of the message.

      • End (obj) - An object containing the end location of the message.

        • Line (int) - The line number of the start or end location of the comment.

        • Column (int) - The character location of the start or end of the comment.

When automatically generating comments on students' code, points may be awarded (or rather deducted) for each comment placed on their code. This can be parsed from a combination of the number of comments and their level of severity. The final score must then be written to a separate JSON object that is then read by the test step from the 3 file. You can read more about how to score the tests in our Create a Custom Test guide:

pageCreate a Custom Test

Comments can automatically be generated via any AutoTest v2 test block including the Script block, the IO Test block and the Custom Test block.