🟨Advanced JavaScript autograding

Discover the advanced grading options available for JavaScript assignments

In this guide, we explore the advanced grading options available for JavaScript assignments. For more information about setting up a JavaScript assignment from scratch, see:

🟨Create your first JavaScript assignment

Jest

Jest is an industry-standard unit testing framework for JavaScript. It is great for grading assignments in which students create their own functions. Consider the example submission below:

calculator.js

// Function to add two numbers
function add(a, b) {
    return a + b;
}

// Function to subtract the second number from the first
function subtract(a, b) {
    return a - b;
}

// Function to multiply two numbers
function multiply(a, b) {
    return a * b;
}

// Function to divide the first number by the second
// Throws an exception if the second number is 0
function divide(a, b) {
    if (b === 0) {
        throw new Error("Division by zero is not allowed.");
    }
    return a / b;
}

// Export the functions for use in other modules
module.exports = {
    add,
    subtract,
    multiply,
    divide
};

Using Jest, we can easily create multiple test cases to assess the functionality of the functions.

calculator.test.js

const { add, subtract, multiply, divide } = require('./calculator');

describe('Calculator Functions', () => {
    test('add function adds two numbers', () => {
        expect(add(1, 2)).toBe(3);
        expect(add(-1, -1)).toBe(-2);
        expect(add(1.5, 2.5)).toBe(4);
    });

    test('subtract function subtracts the second number from the first', () => {
        expect(subtract(5, 3)).toBe(2);
        expect(subtract(0, 5)).toBe(-5);
        expect(subtract(5.5, 2.5)).toBe(3);
    });

    test('multiply function multiplies two numbers', () => {
        expect(multiply(2, 3)).toBe(6);
        expect(multiply(-2, -3)).toBe(6);
        expect(multiply(2.5, 2)).toBe(5);
    });

    test('divide function divides the first number by the second', () => {
        expect(divide(6, 3)).toBe(2);
        expect(divide(-6, -3)).toBe(2);
        expect(divide(5.5, 2)).toBe(2.75);
    });

    test('divide function throws an error when dividing by zero', () => {
        expect(() => divide(1, 0)).toThrow('Division by zero is not allowed.');
    });
});

"cg junitxml" command

We could simply compile and run the unit test to show students a simple pass or fail, depending on the exit code of the tests. However, we wouldn't be able to award partial marks for each test case unless we parsed the results of the unit tests. To accomplish that, we can use the cg junitxml command. This command allows us to parse the number of test cases passed and the feedback from failed test cases from any unit test report written in the JUnit XML format.

Running the command cg junitxml --help shows the following information:

This is a parser for generic unit test coverage reports in JUnit XML format.

The parser can take in input multiple JUnit XML files. The final reported score
is an aggregate of all the reports. Skipped test cases do not count towards the
score.

Example use:
    Input to the command (in a file called coverage.xml):
<?xml version="1.0" encoding="UTF-8" ?>
<testsuites id="Calculator suite" name="empty_name" tests="2" failures="1" time="0.001">
  <testsuite id="uuid1" name="Addition and Multiplication" tests="2" failures="1" time="0.001">
    <testcase id="uuid2" name="[1] Check whether the addition function returns the expected result." time="0.001"></testcase>
    <testcase id="uuid2" name="[2] Check whether the multiply function returns the expected result." time="0.001">
      <failure message="expected: 6.0 but was: 5.0" type="ERROR">
ERROR: Expected: 6.0 but was: 5.0
Category: Checking returns - Multiplication
File: /home/codegrade/student/calculator.py
Line: 2
      </failure>
    </testcase>
  </testsuite>
</testsuites>

    You would then call:
cg junitxml coverage.xml

Usage:
  cg junitxml XMLS... [flags]

Flags:
  -h, --help              help for junitxml
      --no-parse-weight   Don't try to parse the weight from titles.
      --no-score          Don't calculate and output the final score based on the results of the input data.
  -o, --output string     The output file to to use. If this is a number it will use that file descriptor. (default "3")

The cg junitxml command works well in combination with the Custom Test block by displaying the results of the parsed JUnit XML report beautifully, making it easy to read and interpret, as shown in the image below.

Fortunately, we can use the "jest-junit" reporter option to automatically output a JUnit XML report from our unit test. We can set the reporter in our package.json file:

package.json

{
  "dependencies": {
    "jest": "^29.7.0",
    "jest-junit": "^16.0.0"
  },
  "scripts": {
    "test": "jest"
  },
  "jest": {
    "reporters": [
      "jest-junit"
    ]
  }
}

Instructions

1. In the AutoTest settings, navigate to the Setup tab.

2. Add an Install Node block and select the latest version.

3. Add an Upload Files block. Upload calculator.test.js and package.json.

4. Add a Script block. Use the following bash commands to install the node modules for jest and the jest-junit reporter:

   cd $UPLOADED_FILES
   npm install

5. Navigate to the Tests tab.

6. Add a Script block. Use the following bash commands to make the unit tests, node modules, and package.json available to the students' submissions:

   mv $UPLOADED_FILES/* .

7. Add a Connect Rubric block and a Custom Test block. Nest the Custom Test block within in the Connect Rubric block. Use the following command to run Jest and parse the resulting JUnit XML report:

   npm test calculator.test.js --silent
   cg junitxml junit.xml

8. Build and publish your snapshot.

ESLint

ESLint is an industry-standard static code analysis tool for JavaScript that allows you to identify styling and implementation issues in your code. It is a useful tool for enforcing code styling best practices for beginner programmers.

"cg comments" command

Simply running ESLint would not be particularly useful for students as they would have to spend time interpreting the command line output and would have to switch back and forth between the AutoTest output and their code. Instead, we can use the cg comments command to parse the output of ESLint and write the comments directly onto our students' code.

Running the command cg comments --help shows the following information:

This is a parser for generic linters to CodeGrade ATv2 comments.

The regex should be written in Perl5 flavor and should have at least 3 match groups:
    file:       The filename of the comment.
    line:       The line of the comment, by default this will be the start and endline.
    message:    The message of the comment.

There are 5 optional match groups:
    column:     The (begin) column of the error, defaults to 0.
    lineEnd:    The end line of the error, defaults to the begin line.
    columnEnd:  The end column of the error, defaults to the end of the line.
    code:       The error code, any string is accepted.
    severity:   The severity, valid values are "fatal","error","warning", and "info".
                Will default to not set. If the parsed severity is not valid,
                it will also default to not set.

Example use:
    Input to the command:
filename:1:Some error that has occurred on line 1
different_filename:25:An error on line 25 of a different file

    Regex to pass to the command to capture filename, line, and message:
^(?P<file>[^:]*):(?P<line>\d+):(?P<message>.*)$

    Resulting in the following invocation piping the input to cg comments:
input_to_command | cg comments \
    '^(?P<file>[^:]*):(?P<line>\d+):(?P<message>.*)$' \
    --origin "the-linter"

Usage:
  cg comments REGEX [flags]
  cg comments [command]

Available Commands:
  generic     A generic reporter that uses regex to parse log messages.

Flags:
      --base-path string              The base path of the reported files.
                                      Files that are not children of this path will not be reported.
                                      Pass the empty string to disable this feature. (default "/home/codegrade/student")
      --buffer-time duration          The time comments are buffered to be grouped together into a single message. (default 2s)
      --deduction-error Deduction     The percentage deduction for comments with *error* severity.
                                      It should be an integer value. (default 20)
      --deduction-fatal Deduction     The percentage deduction for comments with *fatal* severity.
                                      It should be an integer value. (default 100)
      --deduction-info Deduction      The percentage deduction for comments with *info* severity.
                                      It should be an integer value. (default 5)
      --deduction-unknown Deduction   The percentage deduction for comments with no set severity.
                                      It should be an integer value. (default 0)
      --deduction-warning Deduction   The percentage deduction for comments with *warning* severity.
                                      It should be an integer value. (default 10)
  -h, --help                          help for comments
      --ignore-parser-errors          Should we ignore it if we encounter a line in the input that we cannot parse using the regex provided.
      --ignore-regex string           Lines matching this regex will be ignored.
      --no-score                      If enabled, the score will not be generated based on the comments according to the set
                                      deduction for each severity level.
                                      To customize these values when disabled, add a --deduction-<severity> [percentage] flag for each
                                      severity you wish to override. To apply a deduction to comments with no severity set, set the
                                      --deduction-unknown value to a number greater than 0.
      --origin string                 The origin to use for the linter.
  -o, --output string                 The output file to to use. If this is a number it will use that file descriptor. (default "3")
  -p, --print                         Print comments after parsing them.
      --severities string             A lookup mapping for parsed severities.
                                      Should be of the format: "parsed1:severity,parsed2:severity",
                                      for example "note:info,remark:info".

Use "cg comments [command] --help" for more information about a command.

The cg comments command will highlight each target line according to the severity of the comment and can be read by hovering over the line number with the mouse cursor. The comments are also placed on students' code in the editor, making it a powerful combination.

Requirements

ESLint requires a configuration file and a package.json file to be run without errors. Here you can find an example of a package.json file and a eslint.config.js file.

package.json

{
  "dependencies": {
    "eslint": "^9.3.0",
    "eslint-plugin-jsdoc": "^48.2.6",
    "eslint-formatter-compact": "^8.40.0"
  },
  "type": "module"
}

Here, we require the jsdoc plugin and the compact formatter for eslint. The jsdoc plugin provides extra rules related to code documentation and the compatc formatter allows us to format the output of ESLint. Additionally, we indicate "type" as "module" so that the config file is treated as a module.

eslint.config.js

import jsdoc from "eslint-plugin-jsdoc";

export default [
    {
        files: ["**/*.js"],
        plugins: {
            jsdoc: jsdoc
        },
        rules: {
            "jsdoc/require-description": "error",
            "jsdoc/check-values": "error",

            // Indentation
            "indent": ["warn", 2],

            // Semicolons
            "semi": ['warn', 'always'],
            "semi-style": ['warn', 'last'],

            // Empty lines
            "no-multiple-empty-lines": ["warn", { "max": 1, "maxBOF": 0, "maxEOF": 0 }],

            // Spaces
            "no-multi-spaces": "warn",
            "space-infix-ops": ["warn", { "int32Hint": false }],
            "space-in-parens": ["warn", "never"],
            "no-trailing-spaces": "warn",
            "semi-spacing": ["warn", {"before": false, "after": true}],

            // Variable naming
            // WARNING: does not check if PascalCase is used instead of camelCase
            "camelcase": ["warn", { "properties": "always"}],
            "id-length": ["warn", { "min": 2, 'exceptions': ['i', 'j', 'n', 'm', 'PI', 'Pi', 'pi'] }],
            "no-underscore-dangle": ["warn"],


            // Other
            "no-var": "warn",
            "prefer-const": "warn",
        }
    }
];

For more information about ESLint rules and configuration, please see their documentation.

Instructions

1. In the AutoTest settings, navigate to the Setup tab.

2. Add an Install Node block and select the most recent version of Node to install.

3. Add an Upload Files block and upload both your package.json file and your eslint.config.js file.

4. Add a Script block and use the following commands to install eslint and the dependencies listed in the package.json file.

   cd $UPLOADED_FILES && npm install

5. Navigate to the Tests tab.

6. Add a Connect Rubric block and a Custom Test block. Nest the Custom Test block in the Connect Rubric block. Use the following commands to move the uploaded files and dependencies you installed to the student directory, run ESLint, and parse the result using the cg comments command.

   mv $UPLOADED_FILES/* .
   
   npx eslint --format compact calculator.js | cg comments \
       '(?P<file>/[^:]*): line (?P<line>[\d]+), col (?P<column>[\d]+), (?P<severity>[\S]+) - (?P<message>.*) \((?P<code>[^\)]*)\)$' \
        --origin eslint \
        --severities 'Error:error,Warning:warning' \ # Map unknown severities
        --ignore-regex '^\s*$|^[^/].*$' \ # Ignore this pattern in the ESLint output

7. Build and publish your snapshot

Conclusion

These advanced JavaScript testing options open the door to more rigorous testing methods and better student feedback. However, these are just the most commonly used frameworks used for testing JavaScript. There are many more testing tools and packages available that you can use with some simple setup. For more information, contact us at [email protected].