ð
Websites
This guide uses some information provided by Johan Holmberg from Malmö University, who has shared his steps to set up Selenium in CodeGrade in a public GitHub repository. You can find that in the link below.
CodeGrade AutoTest runs on Ubuntu (18.04.2 LTS) machines which you can configure in any way that you want.
In the setup section of your AutoTest, you can upload any files you might need for testing. These files are called Fixtures and will be placed in the
$FIXTURES
directory on the Virtual Server.The file structure of the server is like this:
$FIXTURES/
All of your uploaded fixtures will be here.
$STUDENT/
This is where the submission of student is placed.
Tests are executed from this directory.
After uploading any files you might need, you can run some setup to install any packages you might need.
- Global setup script: this is where you install your additional pip packages you want to be available on the server. The global setup only runs once and is then cached, so the student submission won't be available here yet.
- Per-student setup script: this will run once when running the autograding for a student and will run before all tests. This is a good place to move
$FIXTURES
to the$STUDENT
directory in case the student solution needs some input files to run correctly.

Use any command in the Global Setup Script field to install software or run your setup script
- 1.Create a
setup.sh
scrip and upload it as a fixture to CodeGrade and The full script would be (from Johan Holmbergâs GitHub):#!/bin/shâ# Installs Jest as per https://help.codegrade.com/user-reference/autotest-general/unit-testcg-jest installâ# Updates the Ubuntu repo settingssudo apt updateâ# Installs Firefox and a headless X window managersudo apt install firefox xvfbâ# Installs the neccessary javascript librariesnpm install -g selenium-webdriver geckodriverâ# Makes the run script executablechmod +x $FIXTURES/run.sh - 2.Run that in our Global Setup Script with
bash $FIXTURES/setup.sh
.
Now that you have created the setup, it's time to create the actual tests. Do this by pressing the "Add Level" button and then the "Add Category" button.
All tests in AutoTest fill in a specific rubric category that you select. After selecting the category, you can start creating tests that will fill it in. Multiple test types are available in CodeGrade, which can be used together depending on your needs and wishes.
For the purpose of this guide, we have designed a very simple example assignment that uses HTML, CSS and some inline JavaScript. We will use Jest and Selenium to assess two parts of the user interface of this website:
- Clicking on one button with the ID
email
should link to a page that has the title âEmail sent!â. - Clicking on another button with the ID
help
should open the CodeGrade Help Center (help.codegrade.com) in a new pop-up window.
The student should upload this
index.html
file, next to that they may upload email.html
and style.css
.
Example webpage submission rendered in CodeGrade
Now we can begin writing our Selenium and Jest tests. For this example we will be using a template provided by Johan Holmberg. His template (find it here) starts with the following base code:
const { Builder, By, until } = require('selenium-webdriver');
require('geckodriver');
â
// This is the directory where the student's submission ends up.
// It can easily be changed to any URL.
const baseURL = 'file:///home/codegrade/student/';
// Change this to select another file to test.
const fileUnderTest = 'index.html';
const defaultTimeout = 10000;
let driver;
jasmine.DEFAULT_TIMEOUT_INTERVAL = 1000 * 60 * 5;
â
// Three simple functions to ease DOM navigation
const getElementByName = async (driver, name, timeout = defaultTimeout) => {
const element = await driver.wait(until.elementLocated(By.name(name)), timeout);
return await driver.wait(until.elementIsVisible(element), timeout);
};
â
const getElementById = async (driver, id, timeout = defaultTimeout) => {
const element = await driver.wait(until.elementLocated(By.id(id)), timeout);
return await driver.wait(until.elementIsVisible(element), timeout);
};
â
const getElementByTag = async (driver, tag, timeout = defaultTimeout) => {
const element = await driver.wait(until.elementLocated(By.tagName(tag)), timeout);
return await driver.wait(until.elementIsVisible(element), timeout);
};
â
// This is run before any test.
beforeAll(async () => {
driver = await new Builder().forBrowser('firefox').build();
// This could be done elsewhere if you want to test multiple pages
await driver.get(baseURL + fileUnderTest);
});
â
// This is run when all tests are done. If this isn't run, the Firefox session
// lingers, so make sure it actually runs.
afterAll(async () => {
await driver.quit();
}, defaultTimeout);
baseURL
constant points to the $STUDENT directory where our students submission files can be found.fileUnderTest
is the name of the student's html file to be tested. Change this accordingly.getElementbyName
,getElementById
andgetElementByTag
are functions that will help us navigate in the browser.
Autograding email button
We want to test our student's email button. In order to do this we can write a simple Selenium test that simulates clicking on the "email" button, waits for the new page to load and the verifies that the new page's title matches our expectation. We use the following test suite for that:
// This is a test suite. All tests in the suite share resources and the tests
// are run in sequence. You can create as may suites as you want.
describe('Check "Email" button.', () => {
â
// This is how to describe a test within a test suite. You may define as
// many tests as you want in a suite. You may even put other suites within
// a suite.
// Please note that you have to prepend each call to an asychronous
// function with 'await', or the assertations won't work.
test('Did you link to the "Email Sent!" page?', async () => {
const submitButton = await getElementById(driver, 'email');
await submitButton.click();
const title = await driver.getTitle();
expect(title).toBe('Email sent!');
});
}, defaultTimeout);
We do a number of things in this test suite:
- 1.Give the test suite a description that will show up in CodeGrade. In this case we chose 'Did you link to "Email Sent!" page after clicking button?'.
- 2.Use the helper function
getElementById
to find an element with the IDemail
. - 3.Simulate a
click()
action on the button - 4.Get the title of the page we are directed to.
- 5.Compare the expected page title,
Email sent!
, to the actual title and capture the result.
Autograding a pop up window
Now we want to test the "Help Center" button. To do this we must again simulate a button click and then compare the titles of all open windows to an expected value,
'help.codegrade.com'
. The test suite is written as follows:// This is a test suite. All tests in the suite share resources and the tests
// are run in sequence. You can create as may suites as you want.
describe('Check "Help" button.', () => {
// This is how to describe a test within a test suite. You may define as
// many tests as you want in a suite. You may even put other suites within
// a suite.
// Please note that you have to prepend each call to an asychronous
// function with 'await', or the assertations won't work.
test('Was the Help Center was opened in new tab?', async () => {
await driver.get(baseURL + fileUnderTest);
const submitButton = await getElementById(driver, 'help');
await submitButton.click();
const windowHandles = await driver.getAllWindowHandles();
expect(windowHandles).toHaveLength(2);
â
// You can only get the title of the currently active window, so we
// loop through all the window handles to see if the confirmation
// popup is open.
let foundPopup = false;
for (const win of windowHandles) {
await driver.switchTo().window(win);
const windowUrl = await driver.getCurrentUrl();
if (windowUrl.includes('help.codegrade.com')) {
foundPopup = true;
break;
}
}
expect(foundPopup).toBe(true);
});
}, defaultTimeout);
Respectively, in this test suite we:
- 1.Navigate back to the main page of our submission:
index.html
. - 2.Use the
getElementById
function to look for a button with the ID'help'
. - 3.Use the
click()
function to simulate a click on the button. - 4.Get all
WindowHandles
since the pop-up is considered a new window. - 5.Expect all
WindowHandles
to have a length of 2. - 6.Loop through all windows to see if one of them has
help.codegrade.com
in the URL. If so, the test passes.
We simply append these two test suites to the initial base code to form our test file
selenium.test.js
, which we will upload as a fixture to CodeGrade. In this instance, we nested the two tests in a test suite, describe('Testing the functionality of your buttons')
, which appears as a label in the CodeGrade UI (see screenshot later in this blog).Now that we have uploaded our tests and global setup script as a fixtures (these will be available in the
$FIXTURES
folder) and have run setup.sh
in the Global Setup Script to install and configure the virtual environment, it is now time to configure student environment for our Jest and Selenium testing.Jest requires us to add a configuration file, named
jest.config.js
. This mandatory configuration file can be left empty if wished for. Our simple configuration file is found below and can be tweaked with the help of Jestâs configuration documentation here. We upload this jest.config.js
file as a fixture.module.exports = {
// Indicates whether each individual test should be reported during the run
verbose: true,
};
Now, all configuration that is left is moving all our fixtures to the right folders. For this we have created a little script called
setup_student.sh
, which moves our configuration file to the current student directory and moves our tests to a new directory called tests
:#!/bin/sh
â
# Copies the Jest config file to the student's directory
cp $FIXTURES/jest.config.js .
â
# Copies the test files to the student's directory
mkdir tests
cp $FIXTURES/*test.js tests
We run this script to configure our student environment by running
ââ$FIXTURES/setup_student.sh
in the Per-student setup script.It is now time to create our actual Selenium tests in AutoTest to actually assess our studentsâ code. For this we can add a Unit Test step to our AutoTest category and select
custom
from the drop-down menu of unit testing frameworks. Because we need to run Firefox âheadlessâ (i.e. without a graphical display), we need to run our tests using the xvbf
tool that we installed earlier. In total, our Program to run will become: xvfb-run cg-jest run tests/selenium.test.js
(or xvfb-run cg-jest run tests/
if we want to run all tests in the tests
directory).
Selenium unit test output in CodeGrade (
describe()
statements are stacked to fill the different labels)- Run Program Tests: Run a command and if the command runs successfully this test will pass. Great for custom tests and compile commands.
- Unit Tests: Upload a unit test file as a
$FIXTURE
and automatically run unit tests on the code. We have several built-in options, follow one of the automatic grading guides above to find out more! - Code Quality Tests: Automatic run static code analysis on the student's code. We have several built-in options, follow one of the automatic grading guides above to find out more!
- Capture Points Tests: Use your own custom grading script, upload it as a
$FIXTURE
, execute it in this test and make sure it outputs a float between 0.0 and 1.0, this score will then be the number of points a student receives. - Checkpoints: Only execute tests below the checkpoint if the tests above the checkpoint reach a certain score. Great for separating simple and advanced tests.
Once you have created a configuration, press start and your AutoTest will start running. In the General Settings, you should have already uploaded a Test Submission, so you will see the results of this straight away.
Once it's started and your assignment is set to Open or Done, students can hand in and get immediate feedback!
Last modified 9mo ago