CSS Visual Regression Testing with Grunt-BackstopJS

by: Joe Watkins

  • 21 March 2015

These days as a front-end web developer you can find yourself drowning in new tools to try out. You can spend hours on end trying to keep up with all of these new tools to add to your workflow. I find myself down this rabbit hole quite often. (I secretly like it)

##CSS Visual Regression CSS visual regression testing is the process of running automated visual test comparisons on pages or elements in your projects. Generally these tests utilize tools consisting of PhantomCSS, CasperJS, and Resemble.js. These tools can literally visit specified web pages, take snapshots at different screen sizes and then compare those images to each other. These comparisons can uncover bugs you’d otherwise not learn about until it’s too late.

After being introduced to CSS visual regression testing by my colleague Greg, I took on the challenge of integrating a testing system into our team’s workflow.

I’ve tested a few CSS regression tools out there and my favorite has been BackstopJS. This tool is fairly easy to configure which allows for you to get up and rolling quickly. Backstop.js loads your tests in a fancy webpage that gives you the visual feedback needed for targeting bugs caused from CSS related issues. There are some other great posts out there on BackstopJS to get you familiar with the workflow.

Basic CSS Regression Workflow

From a high level CSS visual regression testing consists of a few simple steps in your workflow.

  • Create reference files/images to test against. Style Guides make for awesome testing grounds.
  • Make normal changes to your project’s CSS.
  • Run CSS visual regression test.
  • If the tests fail you will see what is broken in results to fix.
  • Fix issues, pass the test, and then deploy.

Step One - Required Packages

Be sure to have Node.js, Grunt, Gulp, and Bower Package Manager setup before moving forward. This tooling actually uses both Grunt and Gulp so you will need them both :)

Start things off by installing PhatomJS and Capser.js

$ sudo npm install -g phantomjs

$ sudo npm install -g casperjs

Step Two

Create a project folder and setup a Bower config file in the root. bower init Now you will install Backstop.js using Bower.

$ bower install --save-dev backstopjs

This will install Backstop.js inside the /bower_components/ folder.

Step Three - grunt-backstopjs

This is where we will deviate from other BackstopJS tutorials out there and we will setup grunt-backstop. This awesome grunt plugin makes using BackstopJS a cinch.

I challenged my colleague Dan at the office, who is comfortable in Node.js, to author the plugin. Together we came up with a solution that we use today on projects. While the plugin is still in its infancy its super easy to setup. BackstopJS, in it’s current state, forces you to work within the /bower_components/ folder. This conflicted with our team’s workflow as we do not check that folder into our version control. We needed to work outside of that folder. Here’s the issue that started it all :)

$ npm install grunt-backstop --save-dev

Once the plugin has been installed to the /node_modules/ folder add the task to your Gruntfile.js

grunt.loadNpmTasks(‘grunt-backstop’);

Basic Configuration

backstop: {
		your_target: {
			options: {
				backstop_path: ‘./bower_components/backstopjs’,
				test_path: ‘./tests’,
				setup: true,
				configure: true,
				create_references: true,
				run_tests: true
			}
		},
	},

Advanced Configuration

Because the standard workflow for BackstopJS is a series of steps to setup I like to break up my Grunt config so that I can have each of these commands at my disposal. e.g. Setup, Create reference files, Run test etc. It is a bit verbose but gives you more control:

backstop: {
	setup: {
		options : {
			backstop_path: ‘./bower_components/backstopjs’,
			test_path: ‘./tests’,
			setup: false,
			configure: true
		}
	},
	test: {
		options : {
			backstop_path: ‘./bower_components/backstopjs’,
			test_path: ‘./tests’,
			create_references: false,
			run_tests: true
		}
	},
	reference: {
		options : {
			backstop_path: ‘./bower_components/backstopjs’,
			test_path: ‘./tests’,
			create_references: true,
			run_tests: false
		}
	}
}

Step Four

Once you have your Gruntfile setup you will run the BackstopJS setup. This will trigger BackstopJS’s usual setup.

$ grunt backstop:setup

Step Five

Now if you don’t already have a backstop.json file in your project root let’s create one. We will do this manually. From the project root use these commands:

$ cd bower_components/backstopjs/ then $ gulp genConfig

This will place a backstop.json file in your project root. It has all the settings for screen sizes, target testing urls, and components to test. The default file that BackstopJS creates should work for a quick test.

Backstop.json example

Step Six

Everything should be good to go for us to create our first reference files. We will use the default configuration in the generated backstop.json file. This reaches out to Bootstrap’s site and check a couple components. From the project root run:

$ grunt backstop:reference

This will create a bunch of reference .png files and place them in the /tests folder. These are the images the tests will compare against. When you create new components to test you will update the backstop.json file and then run the reference command again and it will add the new component to the reference files.

Step Seven

Finally we are ready for our first test! Now from the project root run:

$ grunt backstopjs:test

A web browser will open up and give you your test results.

Test results
Visual Feedback

Wrapping it up

So now you are setup for Visual Regression Testing. You can configure the backstop.json file to work with your project either locally or remotely. I highly suggest testing a style guide or component library as part of your deployment strategy.

I have created a Github repo with demo files if you’d like to see this in action.