End-to-End Testing With Puppeteer
A sound application architecture doesn’t resist to changes, but welcomes them. Yet it still doesn’t guarantee that the code is unbroken after we implement new features, making fixes or refactoring. Here we run automated tests to ensure that the app integrity didn’t suffer. So we write unit-tests to check if separate objects, methods, functions work property independently. With integration tests we ensure they play as designed together. Eventually we create system tests to find out if the entire system meets our business requirements. The last one is also known as E2E testing and covers different aspects such functionality, GUI/Usability, security performance. As for functional testing I have already published a few articles sharing my experience with such tools as Nightmare and Zombie.js. They both provide nice programming experience, but still have their drawbacks. Then I asked myself what could serve me better? What I need is an execution environment accessible from both command-line interface and in a browser. Thus I can run the tests during CI (e.g. by Jenkins), but still use interactive mode while debugging the tests. Besides I prefer to have access to the latest features emerging with evergreen browsers. That makes me think of Headless Chrome. Does it have a Node.js API? It turned out it does. The library is called Puppeteer and it’s truly amazing. Below we are going to examine it by writing a test suite for an RWD demo app with a form.
Welcome Puppeteer
So Puppeteer allows to access different browser contexts, pages, frames and workers running in Headless Chrome (Chromium) or Chrome over the DevTools Protocol. It’s a Node.js library, so we can install it with NPM:
npm i -D puppeteer
Typical flow with Puppeteer may look like that:
puppeteer-demo-1.js
const puppeteer = require( "puppeteer" );
async function main() {
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto( "https://css-tricks.com" );
// Do something...
await browser.close()
};
main();
Puppeteer examples can be executed with Node.js or online at https://puppeteersandbox.com
We call puppeteer.launch to connect to Chromium and receive an object representing browser. Next we open a tab of the browser and get the reference to the corresponding object (page). Using the API by the reference we can, for example, load a URL. We can access DOM nodes, emulate user behavior like mouse/keyboard events and much more. See here all the events and methods available on Page instance. When we done with the session we close the browser.
As we run the script (puppeteer-demo-1.js) with Node.js it connects Chromium behind the scene, so we have no feedback. However we can make a screenshot of the open page:
await page.screenshot({ path: "page.png" });
What is more, we can launch Puppeteer in Chrome browser and observe the script working by passing the corresponding options to launch method:
const browser = await puppeteer.launch({
headless: false
});
We can also specify that we want DevTools panel open and “user” interactions slowed down by 40ms
const browser = await puppeteer.launch({
headless: false,
slowMo: 40,
devtools: true
});
What really excites me, we can emulate mobile devices:
const devices = require( "puppeteer/DeviceDescriptors" );
...
await page.emulate( devices[ "iPhone X" ] );
await page.goto( "https://css-tricks.com" );
...
The full list of device descriptions is available here.
What else? We can intercept HTTP(S) requests. We can handle custom events. We can save page as PDF and much more. But it’s enough with the introduction, let’s do now some testing.
Puppeteer meets Jest
Puppeteer is a Node.js library, so we can use it together with a wide range of testing frameworks. I would prefer to go with Jest. So we install it:
npm i -D jest
We can provide Jest configuration in the project manifest file:
package.json
...
"jest": {
"testMatch": [
"**/*.spec.js"
]
},
...
Here I state that Jest shell consider any files with ending .spec.js as test specifications.
As we already on the manifest, we can add a script to run tests:
package.json
...
"scripts": {
"test": "jest"
}
...
So it seems like everything is ready, we can write our first test:
landing-page.spec.js
const puppeteer = require( "puppeteer" );
let browser, page;
describe( "css-tricks.com", () => {
beforeEach( async () => {
browser = await puppeteer.launch();
page = await browser.newPage();
});
afterEach( async () => {
browser.close();
});
describe( "Landing page", () => {
beforeEach( async () => {
await page.goto( "https://css-tricks.com", { waitUntil: "networkidle2" } );
});
describe( "Page title", () => {
it( "equals CSS-Tricks", async () => {
const title = await page.title();
expect( title ).toEqual( "CSS-Tricks" );
});
});
});
});
Jest supports BDD style. So we can use describe function to define a test suite (a group of tests related to the given subject). In other words here I state that we test “css-tricks.com” site. Among others for the site we test “Landing page”, specifically the “Page title”. With function it we define a test. For example we assert the page title corresponds to a specified value. By using expect function we make an assertion (See the full list of Jest assertions).
You likely noted before/after functions. With beforeEach we define setup functionality for every test in the scope. Respectively afterEach does tear-down. Thus we launch the browser and create a brand-new tab prior to every single test and close the browser after. For the landing page tests we load “https://css-tricks.com” and wait until page fully loaded.
Let’s rock it:
npm test
We have got the following output:
Now we change the expectations to make deliberately the test to fail. On running the output changes:
What can be improved regarding the code above. Considering the fact that we are going to have many tests, it feels like top level beforeEach/afterEach functionality we rather encapsulate:
shared/BrowserSession.js
const puppeteer = require( "puppeteer" );
class BrowserSession {
async setup() {
this.browser = await puppeteer.launch(
process.env.DEBUG
? {
headless: false,
slowMo: 40,
devtools: false
}
: {}
);
this.page = await this.browser.newPage();
}
async teardown() {
this.browser.close();
}
}
module.exports = new BrowserSession();
As you can see I slightly extended the functionality. I am using the environment variable DEBUG to launch Chrome in interactive more for debugging. So we can now modify the manifest file for a new script:
package.json
...
"scripts": {
"test": "jest",
"test:debug": "DEBUG=true jest"
}
...
If we run it like npm run test:debug, Chrome browser launches and we can see all the interactions with page.
Alternatively you can configure your testing environment following Facebook recommendations.
Test application
In order to show Puppeteer / Jest in action I prepared a demo app. It’s a simple form performing field validation on submit. Besides the page is RWD-compliant. When you open it on a mobile device it hides long description and reformats the table with form fields. That what we are about to test.
Let’s start by defining global constants:
shared/constants.js
const { join } = require( "path" );
exports.SEL_FORM = `#myform`;
exports.SEL_SUBMIT = `button[type=submit]`;
exports.SEL_EMAIL = `[name=email]`;
exports.SEL_FNAME = `[name=firstName]`;
exports.SEL_VATID = `[name=vatId]`;
exports.SEL_DAY = `[name=day]`;
exports.SEL_MONTH = `[name=month]`;
exports.SEL_FORM_ERROR = `.alert-danger`;
exports.SEL_JUMBOTRON_DESC = `.jumbotron p.gt-small`;
exports.BASE_URL = "https://dsheiko.github.io/react-html5-form/";
exports.PATH_SCREENSHOTS = join( __dirname, "/../", "/screenshots/" );
exports.ASYNC_TRANSITION_TIMEOUT = 200;
exports.NETWORK_TIMEOUT = 5000;
Basically we exported from the module all the selectors that we will need for testing. We defined the page URL and path to store screenshots. Besides, we set two timeouts: NETWORK_TIMEOUT - time we allow per test given that it performs HTTPS requests and network conditions may vary, ASYNC_TRANSITION_TIMEOUT - time between we request setState() and it gets updated asynchronously.
Also as shared functionality I would have screenshot options builder function:
shared/helpers.js
const { PATH_SCREENSHOTS } = require( "./constants" );
exports.png = ( name ) => ({ path: `${PATH_SCREENSHOTS}/${name}.png` });
Well, the first requirements we are going to cover is following. The form submit button is blocked with property disabled until the first input. So start the test spec:
specs/form.spec.js
const bs = require( "../shared/BrowserSession" ),
{ png } = require( "../shared/helpers" ),
{ BASE_URL, SEL_FORM, SEL_SUBMIT, SEL_EMAIL, SEL_FNAME, SEL_VATID,
SEL_DAY, SEL_MONTH, SEL_FORM_ERROR,
ASYNC_TRANSITION_TIMEOUT, NETWORK_TIMEOUT } = require( "../shared/constants" );
jest.setTimeout( NETWORK_TIMEOUT );
describe( "Boostrap Form Demo", () => {
beforeEach(async () => {
await bs.setup();
});
afterEach(async () => {
await bs.teardown();
});
describe( "Form", () => {
beforeEach(async () => {
// set viewport width
await bs.page.setViewport({ width: 1280, height: 1024 });
await bs.page.goto( BASE_URL, { waitUntil: "networkidle2" } );
});
describe( "Submit button", () => {
it( "is disabled before any input", async () => {
// Obtain handlers for form and submit button elements
const form = await bs.page.$( SEL_FORM ),
submitBtn = await form.$( SEL_SUBMIT );
// Make screenshot of the submit button
await submitBtn.screenshot( png( `form-submit-before-input` ) );
// Obtain property disable value from submit button element
const isDisabled = await form.$eval( `${SEL_SUBMIT}`, el => el.disabled );
// Make assertion
expect( isDisabled ).toBeTruthy();
});
});
});
});
What it does? We using $ method of page object to get handlers for form and submit button elements, pretty much like we find elements in DOM by using document.querySelector:
const form = await bs.page.$( SEL_FORM ),
submitBtn = await form.$( SEL_SUBMIT );
Talking of screenshot, we are not that interested in the entire page, but need to see how submit button look. In disabled state according to Bootstrap styles it’s slightly lighter. So we screenshot the button:
await submitBtn.screenshot( png( `form-submit-before-input` ) );
Next we need the actual value of disabled property on the button element. I prefer to do it with the only line of code and that is achievable with $eval method. by some reason the method on element handler still requires selector, so I reckon I cannot call directly on submitBtn, but shall do it on upper container e.g. form. the second parameter is callback, which receives DOM element. We return the desired property value and it gets assigned to isDisabled. Now we just assert the property is truthy.
Alternatively we could obtain the property value like: const handle = await submitBtn.getProperty( “disabled” );
const isDisabled = await handle.jsonValue();
When testing form opposite condition we obtain the handler for email input and type in a text:
const email = await form.$( SEL_EMAIL );
await email.type( `anything` );
Further requirement is to ensure the email input gets validated on submit:
describe( "Email field", () => {
it( "gets invalid state when invalid email typed in", async () => {
const form = await bs.page.$( SEL_FORM ),
submitBtn = await form.$( SEL_SUBMIT ),
email = await form.$( SEL_EMAIL );
await email.type( `invalid-email` );
await submitBtn.click();
await bs.page.waitFor( ASYNC_TRANSITION_TIMEOUT );
const isInvalid = await form.$eval( `${SEL_EMAIL}`, el => el.matches( `:invalid` ) );
await form.screenshot( png( `form-email-invalid-state` ) );
expect( isInvalid ).toBeTruthy();
});
});
We type in an invalid email address into the input and click on the submit button:
await submitBtn.click();
The demo app is written in React.js. On form submit it calls checkValidity of HTML Form Validation API, what toogles :invalid pseudo-selector on inputs in invalid state. So we check if the input is invalid by:
el.matches( `:invalid` )
As the last functional requirement for the form we check that after submission form renders server error:
describe( "Submission", () => {
it( "gets invalid state when invalid emailed typed in", async () => {
const form = await bs.page.$( SEL_FORM ),
submitBtn = await form.$( SEL_SUBMIT );
await bs.page.type( `${SEL_FORM} ${SEL_EMAIL}`, `[email protected]` );
await bs.page.type( `${SEL_FORM} ${SEL_FNAME}`, `Jon Snow` );
await bs.page.type( `${SEL_FORM} ${SEL_VATID}`, `DE000` );
await bs.page.select( `${SEL_FORM} ${SEL_DAY}`, `...` );
await bs.page.select( `${SEL_FORM} ${SEL_MONTH}`, `...` );
await submitBtn.click();
await bs.page.waitForSelector( `${SEL_FORM}[data-submitted=true]` );
await form.screenshot( png( `form-submitted` ) );
const errorMsg = await form.$eval( `${SEL_FORM_ERROR}`, el => el.innerText );
expect( errorMsg ).toEqual( `Oh snap! Opps, a server error` );
});
});
Here we fill in all the fields, click submit. Now we need to wait until HTTPS request completes. When form reloads the page we can simply use:
await bs.page.waitForNavigation();
The demo app sends XHR, waits for the response and updates the view. Fortunately the app updates form attribute data-submitted when submission completes and we can rely on it:
await bs.page.waitForSelector( `${SEL_FORM}[data-submitted=true]` );
So now we are just to check the content of error message container:
const errorMsg = await form.$eval( `${SEL_FORM_ERROR}`, el => el.innerText );
Running the test in debug mode
Testing RWD
How do we test responsive web design requirements? Well, let’s see. We have a long description in Jumbotron, which appears on wide screens and hides on mobile devices. Besides, we have email and first name inputs lined up in a row on wide screens, but restructured one under another on mobile devices. To test it we create the following template:
specs/rwd.spec.js
const devices = require( "puppeteer/DeviceDescriptors" ),
bs = require( "../shared/BrowserSession" ),
{ png } = require( "../shared/helpers" ),
{ BASE_URL, SEL_FORM, SEL_SUBMIT, SEL_EMAIL, SEL_FNAME,
SEL_JUMBOTRON_DESC, NETWORK_TIMEOUT } = require( "../shared/constants" );
jest.setTimeout( NETWORK_TIMEOUT );
describe( "Boostrap Form Demo", () => {
beforeEach(async () => {
await bs.setup();
});
afterEach(async () => {
await bs.teardown();
});
describe( "Page", () => {
describe( "on PC/Notebook 1280x1024", () => {
beforeEach(async () => {
await bs.page.setViewport({ width: 1280, height: 1024 });
await bs.page.goto( BASE_URL, { waitUntil: "networkidle2" } );
});
// tests
});
describe( "on iPhone X", () => {
beforeEach(async () => {
await bs.page.emulate( devices[ "iPhone X" ] );
await bs.page.goto( BASE_URL, { waitUntil: "networkidle2" } );
});
// tests
});
});
});
Here we define two scopes: one for 1280x1024 screen and the second for iPhone X emulator. We can test the visibility of long description as follows:
describe( "on PC/Notebook 1280x1024", () => {
//...
it( "has Jumbotron description", async () => {
await bs.page.screenshot( png( `rwd-jumbotron-on-1280x1024` ) );
const el = await bs.page.$( SEL_JUMBOTRON_DESC );
const isVisible = ( await el.boundingBox() !== null );
expect( isVisible ).toBeTruthy();
});
});
We assume that element matching description container is available in the browser DOM. For any displayed element el.boundingBox() returns an object with element coordinates and size. Otherwise we get null.
We can copy/pase this test for iPhone X test scope, but change the assertion to opposite one.
As for input fields location, we can rely again on el.boundingBox():
describe( "on PC/Notebook 1280x1024", () => {
//...
it( "keeps Email/First Name inputs on the same line", async () => {
const form = await bs.page.$( SEL_FORM ),
email = await bs.page.$( SEL_EMAIL ),
firstName = await bs.page.$( SEL_FNAME );
await form.screenshot( png( `rwd-email-fname-on-1280x1024` ) );
const emailBox = await email.boundingBox(),
firstNameBox = await firstName.boundingBox();
expect( emailBox.y ).toEqual( firstNameBox.y );
});
});
We just just assert both fields boxes have the same y coordinate. For iPhone X test scope we assert the opposite.
The form on a wide screen and on iPhone X
Running the enite test suite in debug mode
Extending Puppeteer
And one more thing about Puppeteer. It’s really easy to extend. For example remember the tricks I was doing to obtain a property of an element and to find out if it’s visible? But actually we can add our own custom methods to close these gaps. Consider this extension:
./shared/extendPuppeteer.js
const ElementHandle = require( "puppeteer/lib/ElementHandle" );
ElementHandle.prototype.isVisible = async function(){
return (await this.boundingBox() !== null);
};
ElementHandle.prototype.getAttr = async function( attr ){
const handle = await this._page.evaluateHandle( ( el, attr ) => el.getAttribute( attr ), this, attr );
return await handle.jsonValue();
};
ElementHandle.prototype.getProp = async function( prop ){
const handle = await this._page.evaluateHandle( ( el, prop ) => el[ prop ], this, prop );
return await handle.jsonValue();
};
As soon as we import this module once we can do element assertions as easy as that:
require( "../shared/extendPuppeteer" );
const elh = await bs.page.$( `#testTarget` );
console.log( await elh.isVisible() );
console.log( await elh.getAttr( "class" ) );
console.log( await elh.getProp( "innerHTML" ) );
Recap
So Puppeteer is a Node.js library providing access to high-level API to Chromium and Chrome. We can use it to crawl web-pages including SPA, to simulate user-behaviour (keyboard/mouse events, form submission and so on). We can run it in both command-line with Headless Chrome and in real browser taking advantage of the latest implemented features. We can even access goodness of DevTool such as timeline trace.
Jest is an advanced testing framework for Noide.js. When combine the power of both we can bring the automating testing to a truly new level.