🌐
Websites

This guide uses some information provided by Johan Holmberg from Malmö University, who has shared his steps to set up Selenium in CodeGrade in a public GitHub repository. You can find that in the link below.
GitHub - koddas/selenium-with-jest-on-codegrade: A simple guide on how to set up Selenium unit testing on CodeGrade
GitHub
Step 1: Setup
CodeGrade AutoTest runs on Ubuntu (18.04.2 LTS) machines which you can configure in any way that you want.
In the setup section of your AutoTest, you can upload any files you might need for testing. These files are called Fixtures and will be placed in the $FIXTURES directory on the Virtual Server.
The file structure of the server is like this:
$FIXTURES/
    All of your uploaded fixtures will be here.
$STUDENT/
    This is where the submission of student is placed. 
    Tests are executed from this directory.
After uploading any files you might need, you can run some setup to install any packages you might need.
Global setup script: this is where you install your additional pip packages you want to be available on the server. The global setup only runs once and is then cached, so the student submission won't be available here yet.
Per-student setup script: this will run once when running the autograding for a student and will run before all tests. This is a good place to move $FIXTURES to the $STUDENT directory in case the student solution needs some input files to run correctly.
Use any command in the Global Setup Script field to install software or run your setup script
Installing Jest and Selenium
1.Create a setup.sh scrip and upload it as a fixture to CodeGrade and  The full script would be (from Johan Holmberg’s GitHub):

#!/bin/sh
​
# Installs Jest as per https://help.codegrade.com/user-reference/autotest-general/unit-test
cg-jest install
​
# Updates the Ubuntu repo settings
sudo apt update
​
# Installs Firefox and a headless X window manager
sudo apt install firefox xvfb
​
# Installs the neccessary javascript libraries
npm install -g selenium-webdriver geckodriver
​
# Makes the run script executable
chmod +x $FIXTURES/run.sh
2.Run that in our Global Setup Script with bash $FIXTURES/setup.sh.
Step 2: Create your tests
Now that you have created the setup, it's time to create the actual tests. Do this by pressing the "Add Level" button and then the "Add Category" button.
All tests in AutoTest fill in a specific rubric category that you select. After selecting the category, you can start creating tests that will fill it in. Multiple test types are available in CodeGrade, which can be used together depending on your needs and wishes.
Simple Web Development HTML and CSS example assignment
For the purpose of this guide, we have designed a very simple example assignment that uses HTML, CSS and some inline JavaScript. We will use Jest and Selenium to assess two parts of the user interface of this website:
Clicking on one button with the ID email should link to a page that has the title “Email sent!”.
Clicking on another button with the ID help should open the CodeGrade Help Center (help.codegrade.com) in a new pop-up window.
The student should upload this index.html file, next to that they may upload email.html and style.css.
Example webpage submission rendered in CodeGrade
Designing Selenium and Jest tests for our example assignment
Now we can begin writing our Selenium and Jest tests. For this example we will be using a template provided by Johan Holmberg. His template (find it here) starts with the following base code:
const { Builder, By, until } = require('selenium-webdriver');
require('geckodriver');
​
// This is the directory where the student's submission ends up.
// It can easily be changed to any URL.
const baseURL = 'file:///home/codegrade/student/';
// Change this to select another file to test.
const fileUnderTest = 'index.html';
const defaultTimeout = 10000;
let driver;
jasmine.DEFAULT_TIMEOUT_INTERVAL = 1000 * 60 * 5;
​
// Three simple functions to ease DOM navigation
const getElementByName = async (driver, name, timeout = defaultTimeout) => {
    const element = await driver.wait(until.elementLocated(By.name(name)), timeout);
    return await driver.wait(until.elementIsVisible(element), timeout);
};
​
const getElementById = async (driver, id, timeout = defaultTimeout) => {
    const element = await driver.wait(until.elementLocated(By.id(id)), timeout);
    return await driver.wait(until.elementIsVisible(element), timeout);
};
​
const getElementByTag = async (driver, tag, timeout = defaultTimeout) => {
    const element = await driver.wait(until.elementLocated(By.tagName(tag)), timeout);
    return await driver.wait(until.elementIsVisible(element), timeout);
};
​
// This is run before any test.
beforeAll(async () => {
    driver = await new Builder().forBrowser('firefox').build();
    // This could be done elsewhere if you want to test multiple pages
    await driver.get(baseURL + fileUnderTest);
});
​
// This is run when all tests are done. If this isn't run, the Firefox session
// lingers, so make sure it actually runs.
afterAll(async () => {
    await driver.quit();
}, defaultTimeout);
baseURL constant points to the $STUDENT directory where our students submission files can be found.
fileUnderTest is the name of the student's html file to be tested. Change this accordingly.
getElementbyName, getElementById and getElementByTag are functions that will help us navigate in the browser.
We can now start to write our tests, following Selenium’s documentation.
Autograding email button
We want to test our student's email button. In order to do this we can write a simple Selenium test that simulates clicking on the "email" button, waits for the new page to load and the verifies that the new page's title matches our expectation. We use the following test suite for that:
// This is a test suite. All tests in the suite share resources and the tests
// are run in sequence. You can create as may suites as you want.
describe('Check "Email" button.', () => {
‍
    // This is how to describe a test within a test suite. You may define as
    // many tests as you want in a suite. You may even put other suites within
    // a suite.
    // Please note that you have to prepend each call to an asychronous
    // function with 'await', or the assertations won't work.
    test('Did you link to the "Email Sent!" page?', async () => {
        const submitButton = await getElementById(driver, 'email');
        await submitButton.click();
        const title = await driver.getTitle();
        expect(title).toBe('Email sent!');
    });
}, defaultTimeout);
We do a number of things in this test suite:
1.Give the test suite a description that will show up in CodeGrade. In this case we chose 'Did you link to "Email Sent!" page after clicking button?'.
2.Use the helper function getElementById to find an element with the ID email.
3.Simulate a click() action on the button
4.Get the title of the page we are directed to.
5.Compare the expected page title, Email sent!, to the actual title and capture the result.
Autograding a pop up window
Now we want to test the "Help Center" button. To do this we must again simulate a button click and then compare the titles of all open windows to an expected value, 'help.codegrade.com'. The test suite is written as follows:
// This is a test suite. All tests in the suite share resources and the tests
// are run in sequence. You can create as may suites as you want.
describe('Check "Help" button.', () => {
    // This is how to describe a test within a test suite. You may define as
    // many tests as you want in a suite. You may even put other suites within
    // a suite.
    // Please note that you have to prepend each call to an asychronous
    // function with 'await', or the assertations won't work.
    test('Was the Help Center was opened in new tab?', async () => {
        await driver.get(baseURL + fileUnderTest);
        const submitButton = await getElementById(driver, 'help');
        await submitButton.click();
        const windowHandles = await driver.getAllWindowHandles();
        expect(windowHandles).toHaveLength(2);
​
        // You can only get the title of the currently active window, so we
        // loop through all the window handles to see if the confirmation
        // popup is open.
        let foundPopup = false;
        for (const win of windowHandles) {
            await driver.switchTo().window(win);
            const windowUrl = await driver.getCurrentUrl();
            if (windowUrl.includes('help.codegrade.com')) {
                foundPopup = true;
                break;
            }
        }
        expect(foundPopup).toBe(true);
    });
}, defaultTimeout);
Respectively, in this test suite we:
1.Navigate back to the main page of our submission: index.html.
2.Use the getElementById function to look for a button with the ID 'help'.
3.Use the click() function to simulate a click on the button.
4.Get all WindowHandles since the pop-up is considered a new window.
5.Expect all WindowHandles to have a length of 2.
6.Loop through all windows to see if one of them has help.codegrade.com in the URL. If so, the test passes.
We simply append these two test suites to the initial base code to form our test file selenium.test.js, which we will upload as a fixture to CodeGrade. In this instance, we nested the two tests in a test suite, describe('Testing the functionality of your buttons'), which appears as a label in the CodeGrade UI (see screenshot later in this blog).
Setting up the student folder in CodeGrade
Now that we have uploaded our tests and global setup script as a fixtures (these will be available in the $FIXTURES folder) and have run setup.sh in the Global Setup Script to install and configure the virtual environment, it is now time to configure student environment for our Jest and Selenium testing.
Jest requires us to add a configuration file, named jest.config.js. This mandatory configuration file can be left empty if wished for. Our simple configuration file is found below and can be tweaked with the help of Jest’s configuration documentation here. We upload this jest.config.js file as a fixture.
module.exports = {
  // Indicates whether each individual test should be reported during the run
  verbose: true,
};
Now, all configuration that is left is moving all our fixtures to the right folders. For this we have created a little script called setup_student.sh, which moves our configuration file to the current student directory and moves our tests to a new directory called tests:
#!/bin/sh
​
# Copies the Jest config file to the student's directory
cp $FIXTURES/jest.config.js .
​
# Copies the test files to the student's directory
mkdir tests
cp $FIXTURES/*test.js tests
We run this script to configure our student environment by running ​​$FIXTURES/setup_student.sh in the Per-student setup script.
Running the tests
It is now time to create our actual Selenium tests in AutoTest to actually assess our students’ code. For this we can add a Unit Test step to our AutoTest category and select custom from the drop-down menu of unit testing frameworks. Because we need to run Firefox ‘headless’ (i.e. without a graphical display), we need to run our tests using the xvbf tool that we installed earlier. In total, our Program to run will become: xvfb-run cg-jest run tests/selenium.test.js (or xvfb-run cg-jest run tests/ if we want to run all tests in the tests directory).
Selenium unit test output in CodeGrade (describe() statements are stacked to fill the different labels)
Other test types
Run Program Tests: Run a command and if the command runs successfully this test will pass. Great for custom tests and compile commands.
Unit Tests: Upload a unit test file as a $FIXTURE and automatically run unit tests on the code. We have several built-in options, follow one of the automatic grading guides above to find out more!
Code Quality Tests: Automatic run static code analysis on the student's code. We have several built-in options, follow one of the automatic grading guides above to find out more!
Capture Points Tests: Use your own custom grading script, upload it as a $FIXTURE, execute it in this test and make sure it outputs a float between 0.0 and 1.0, this score will then be the number of points a student receives.
Checkpoints: Only execute tests below the checkpoint if the tests above the checkpoint reach a certain score. Great for separating simple and advanced tests.
Step 3: Start the AutoTest
Once you have created a configuration, press start and your AutoTest will start running. In the General Settings, you should have already uploaded a Test Submission, so you will see the results of this straight away.
Once it's started and your assignment is set to Open or Done, students can hand in and get immediate feedback!
Autograding SQLite in AutoTest V1
Next - Automatic Grading Guides
Javascript
Last modified 9mo ago