Optimizing Mocha's Builds with Travis CI's Build Stages

A little less than a year ago, Travis CI introduced a beta feature, Build Stages.
Could it be maintainers in my world (that world being “userland tooling and libraries for Node.js”) don’t even know Build Stages are a Thing? Despite the overwhelming popularity of Travis CI amongst OSS Node.js projects, I haven’t seen a lot of adoption.

Mocha, the JavaScript testing framework, has been a happy user of Travis CI for over six years. This last week, Mocha’s team modified its build matrix to leverage Build Stages (thanks @Outsideris!). I'll share what I've learned.

I have written this article (which gets kind of dry) for users of Travis CI or those otherwise experienced in continuous integration software.

Why Build Stages?

If a project’s build consists of running npm test against a handful of Node.js versions, that project probably doesn’t need Build Stages. While Mocha’s build isn’t as complex as some—like those projects which need to compile or deploy—its build is non-trivial.

Foremost and first, Build Stages allow a build to “fail fast.” Before Build Stages, all jobs in the matrix would run concurrently (with an optional concurrency limit). A project’s build could not run, for example, an initial smoke test to determine if it’s practical to run a more expensive test suite. Build Stages eliminate extra work.

Build Stages can enable better dependency caching. As mentioned, without Build Stages, all jobs would run concurrently, and each cache-compatible configuration would “miss” the cache on its first try. For Node.js projects, this means repeated installation of dependencies via npm install, and ain’t nobody got time for that. By “warming up” Travis CI’s cache in preparation, we can prepare npm to work quickly in subsequent cache-compatible Build Stages.

By leveraging Build Stages—and npm’s new features—Mocha can run more tests in less time. Join me in the Bonesea Skullenger, and we’ll dive deep.

Swimming elasipod sea cucumber. Photo from Vailulu'u 2005 Exploration, NOAA-OE / NOAA

About Mocha’s Old Build

Before the changes, this is what Mocha’s .travis.yml looked like (with irrelevant sections removed):

language: node_js

matrix:
  fast_finish: true
  include:
    - node_js: '9'
      env: TARGET=test.node COVERAGE=true
    - node_js: '8'
      env: TARGET=test.node
    - node_js: '6'
      env: TARGET=test.node
    - node_js: '4'
      env: TARGET=test.node
    - node_js: '8'
      env: TARGET=lint
    - node_js: '8'
      env: TARGET=test.browser

before_install: scripts/travis-before-install.sh
before_script: scripts/travis-before-script.sh
script: npm start $TARGET
after_success: npm start coveralls

addons:
  artifacts:
    paths:
      - .karma/
      - ./mocha.js
  sauce_connect: true
  chrome: stable
cache:
  directories:
    - ~/.npm

Much pain!

1. fast_finish does nothing unless you have jobs within an allowed_failures mapping; Mocha does not.
2. We can optimize caching and installation of dependencies:
   1. Each job has its own cache due to the use of environment variables, but (most) every job sharing a Node.js version should share a cache.
   2. node_modules isn’t cached in the cases where it should be.
   3. npm ci is now a thing
   4. Mocha’s dev dependencies include a fair number of native modules, which we don’t always need
3. Browser tests create artifacts (essentially bundles created by karma-browserify and the bundled mocha.js—we use these for debugging esoteric failures on SauceLabs’ browsers) so the upload paths contain a steaming pile of nothing after most jobs
4. Likewise, we’re starting Sauce Connect and installing headless Chrome for jobs that won’t use it
5. One (1) job generates any coverage at all, yet every job attempts to send a report to Coveralls
6. The before_install script runs some smoke tests, but duplicates effort by running three (3) times in Node.js 8
7. The before_script script creates .karma/, but Travis CI’s cache creates it first.

Many of these issues are due to my own ignorance, as git blame will tell you. Those problems can be solved by actually reading Travis CI’s documentation like I should have; we can solve the rest with Build Stages.

Mocha’s New Build using Build Stages

I’ll analyze the new .travis.yml in parts for better context (you can see it in its entirety, if you wish).

Defining the Build Stage Order

It’s optional, but a project will usually want to run stages in order (to enable fast failure, and tasks like conditional deploys).

stages:
  - smoke # this ensures a "user" install works properly
  - precache # warm up cache for default Node.js version
  - lint # lint code and docs
  - test # all tests

In the smoke stage, Mocha runs its smoke (or “sanity”) tests. We want to do this stage first, because if it fails, somebody screwed up big. Doing further work would be a waste.

The precache stage, then, installs dependencies which multiple jobs in subsequent Build Stages will reuse. The single job in the lint stage will hit this cache, as well as two other jobs in the test stage.

Default Job Configuration

At the top level of .travis.yml, we can establish some defaults. Jobs in Build Stages will use these unless they override the options.

language: node_js
node_js: '9'

Jobs won’t change the project’s language, which is node_js.

Mocha runs its complete test suite against all maintained LTS versions of Node.js, in addition to the “current” release. At the time of this writing, those versions are 4.x, 6.x, 8.x and 9.x—these are the only versions of Node.js which Mocha supports! The Build Stages contain jobs which don’t depend on any specific version (lint checks and browser tests), so we may as well use the latest—and often, fastest—Node.js version.

Warning: rabbit hole ahead.

Photo by Tatiana Gettelman / Flickr

Efficient Installation of npm v5.8.0

At the time of this writing, the version of default npm that ships with 8.x and 9.x is v5.6.0. npm ci wasn’t available before v5.8.0, which is the latest version.

Travis CI manages its Node.js versions with nvm. Travis CI also runs nvm install <version> before it reaches into its cache. That means, to use v5.8.0 of npm, a build would naively do this:

before_script: npm install -g npm@5.8.0

That’ll run for every job, which is slow. A build could attempt to cache some subset of ~/.nvm itself (where nvm keeps its installed Node.js versions and globally-installed packages, including npm), but that’s going to contain the node executable and other cruft. Anyway, trying to cache whatever npm install -g npm installed is a dead-end, at the time of this writing. But there’s another way.

If Travis CI’s caching worked with individual files—or supported exclusion—this solution would be more viable. Its granularity stops at the directory level.

Here’s what we do (and these are job defaults, remember):

before_install: |
  [[ ! -x ~/npm/node_modules/.bin/npm ]] && {
    cd ~/npm && npm install npm
    cd -
  } || true
env: PATH=~/npm/node_modules/.bin:$PATH
cache:
  directories:
    - ~/.npm # cache npm's cache
    - ~/npm # cache latest npm

1. Our before_install Bash script checks for the executability (executableness?) of a file, ~/npm/node_modules/.bin/npm. Note that ~/.npm is not ~/npm.
2. If ~/npm/node_modules/.bin/npm is not executable, we assume this was a cache miss. Navigate into ~/npm and use npm to install the latest version of npm local to this directory. After this step, ~/npm will contain node_modules and package-lock.json.
3. We must navigate back to our working copy after navigating away, which is what cd - does.
4. If a job hits the cache, our npm executable is ready, and the script ends with great success (true).
5. We set the PATH to look in ~/npm/node_modules/.bin/ before anything else, so it finds our custom npm executable instead of the one nvm installed.
6. We cache ~/.npm, which is npm’s cache. Right? Right.
7. We cache ~/npm, which contains our custom-installed npm.

The point of this madness is to avoid an npm self-upgrade on every job. Not pretty, but it works.

Using npm ci

By keeping the installation separate from the working copy, we avoid extraneous dependencies. Why is this important? Because:

install: npm ci --ignore-scripts

Armed with npm v5.8.0, we can use npm ci instead of npm install (in most cases; I’d wager the majority of projects looking to use npm ci should always use it). One reason npm cioffers better consistency is that it blasts the local node_modules and re-creates it from scratch. That means we can’t throw anything in there (npm is not a direct dependency of Mocha) that doesn’t belong.

Mocha uses the --ignore-scripts flag in most of its jobs. npm’s lifecycle scripts invoke native module compilation; Mocha consumes native modules when building docs or running browser tests. We don’t test the doc-building scripts themselves, so that leaves a single case where Mocha needs a native module to run a test suite.

This situation isn’t unique to Mocha, but neither is it ubiquitous. Any given dependency of a project may use an install, postinstall, or an infamous prepublish script. Because most of Mocha’s dependencies don’t do this (thanks, dependencies!) , we can get away with it.

I don’t know of any way to tell npm to only avoid compilation of native modules; it’s either “ignore all scripts” or “run all scripts.”

After we have established the default behavior, we can define our jobs. Let’s analyze each stage.

Build Stage 1: Smoke

As noted above, our stages run in order (but the jobs within them run concurrently), and the smoke stage is the first. Here’s its definition:

- &smoke
  stage: smoke
  install: npm install --production --no-shrinkwrap
  script: >
    ./bin/mocha --opts /dev/null --reporter spec 
    test/sanity/sanity.spec.js
  cache:
    directories:
      - ~/.npm
      - node_modules

- <<: *smoke
  node_js: '8'

- <<: *smoke
  node_js: '6'

- <<: *smoke
  node_js: '4'

If you’re unfamiliar with anchors and aliases in YAML (like I was), well… there it is.

An Aside: YAML Anchors & Aliases

It defines an anchor, &smoke. We can then refer to this anchor using an alias, *smoke. The <<: syntax means the mapping will be merged with the mapping from the anchor. JavaScripters could think of it like this:

const baseSmoke = {
	stage: 'smoke',
	install: 'npm install --production --no-shrinkwrap'
	// etc
};

const smokeStage = [
	baseSmoke, 
  Object.assign({}, baseSmoke, {node_js: '8'}),
  Object.assign({}, baseSmoke, {node_js: '6'}),
  Object.assign({}, baseSmoke, {node_js: '4'})
];

If we serialize smokeStage into JSON, this is the result (I apologize for the verbosity, but this is why a hypothetical .travis.json would suck):

[
  {
    "stage": "smoke",
    "install": "npm install --production --no-shrinkwrap"
  },
  {
    "stage": "smoke",
    "install": "npm install --production --no-shrinkwrap",
    "node_js": "8"
  },
  {
    "stage": "smoke",
    "install": "npm install --production --no-shrinkwrap",
    "node_js": "6"
  },
  {
    "stage": "smoke",
    "install": "npm install --production --no-shrinkwrap",
    "node_js": "4"
  }
]

As you can see, we have our top-level defaults, but we can also define defaults within individual stages via YAML voodoo.

Why Smoke Test, Anyway?

Photo by Marcus Kauffman / Unsplash

The goal is to establish a baseline of functionality in a minimal amount of time. This baseline will be different for every project! In Mocha’s case, we want to minimize the likelihood a npm install mocha somehow misses a dependency.

If you’re wondering, yes, this has happened—a dependency was living in package.json’s devDependencies when it should have been in dependencies. Raise your hand if you’ve done that before.

Since we can’t npm install mocha, the next best thing is running npm install --production --no-shrinkwrap in the working copy. This mimics the result a user would get; we don’t install Mocha’s development dependencies, and we ignore package-lock.json (read more about package-lock.json; npm never publishes it to the registry).

Another way to do this could be to npm install the current changeset directly from GitHub, but we already have the working copy cloned. And as of npm v5.x, running npm install /path/to/working/copy results in a symlink, so that’s not workable.

Once npm has installed Mocha’s production dependencies, the bin/mocha executable should run a simple test like:

describe('a production installation of Mocha', function () {
  it('should be able to execute a test', function () {
    assert.ok(true);
  });
}); 

Because Mocha’s own mocha.opts contains references to development dependencies, we must ignore it; the flag --opts /dev/null is a hacky workaround which effectively disables mocha.opts. Normally, I’d put this kind of nonsense in package-scripts.js and let nps run it, but we don’t have nps at our disposal here. Though npx could…

If that simple test runs OK—for each supported version of Node.js—Mocha has passed its smoke tests, and we can move on to the precache stage.

Build Stage 2: Pre-cache

What is it? It is this:

- stage: precache
  script: true

We run everything in the default configuration except an actual build script; true is POSIX shell for “it worked.” That’s enough to create a “warmed-up” cache of our development dependencies for Node.js v9.x, as well as a cache of the latest npm version.

Build Stage 3: Lint

The lint stage is the first to hit our pre-cached dependencies. The latest npm is already present, and all the dependencies live in npm’s cache. We get an even faster install (npm ci, if you recall) since we can ignore scripts.

Since running linter(s) is far less time-consuming than running tests, we may as well run these before Real Tests.

Like precache, this Build Stage has one job:

- stage: lint
  script: npm start lint

npm start lint kicks off our linters (ESLint and markdownlint-cli).

Some months ago, Mocha dropped the Makefile we were using for Kent C. Dodds’ wonderful nps; npm start calls nps. Formerly known as p-s, nps is an elegant task runner (no plugins necessary, unlike Grunt or Gulp). I highly recommend it if your scripts in package.json has become unwieldy.

Like with smoke tests, if the lint checks fail, then we abort the build.

Build Stage 4: Test

Mocha runs its main test suites concurrently in the fourth stage.

You’ll notice there’s no stage mapping in any of the items below. If you look at the entire file, you’ll see that these are the first items in the jobs.include mapping; we must also keep them together. Jobs lacking a stage will use the same stage as the previous job in the list; if there is no previous job, the default stage is test. Ahh, the wonders of convention…

Like jobs in previous stages, these inherit from the default job configuration.

Node.js Tests

The first is our test against the default Node.js version (9.x), which also computes coverage:

- script: COVERAGE=1 npm start test.node
  after_success: npm start coveralls

You’ll note that the environment variable COVERAGE is not defined in an env mapping. This is because variables defined in env create a unique cache configuration—it’d bust the pre-cache and we’d miss everything in it! This environment variable causes our test.node script (found in package-scripts.js) to invoke mocha via nyc.

The unique after_success script will fire coverage information generated by nyc to Coveralls using node-coveralls. If any tests fail, after_success does not run, as you might guess.

These next three (3) jobs are identical, except the Node.js version. These all miss the cache, because we pre-cache Node.js 9.x only (and we don’t use these configurations again). Since we’re already computing coverage in the previous job, we omit the COVERAGE variable.

- &node
  script: npm start test.node
  node_js: '8'

- <<: *node
  node_js: '6'

- <<: *node
  node_js: '4'

The fascinating and frightening YAML anchors (and associated aliases) also appear above; these entries expand exactly as explained earlier.

What’s left?

Browser Tests

- script: npm start test.bundle test.browser
  install: npm ci
  addons:
    artifacts:
      paths:
        - .karma/
        - ./mocha.js
    chrome: stable
    sauce_connect: true

Note the custom install script; every other install script uses —ignore-scripts. This one doesn’t, because it needs some compiled modules to bootstrap headless Chrome. The chrome addon provides a headless Chrome executable to the job.

This is actually two suites; test.bundle is a test launched via the Node.js mocha executable which ensures the bundle we build (via browserify) retains compatibility with RequireJS. Then, Karma handles the test.browser suite.

We abuse NODE_PATH to trick karma-mocha into running our tests with our own mocha.js bundle. Don't do this.

When the browser tests run in a local development environment, they run in headless Chrome by default. On Travis CI, we add handful of “real” browsers, by the grace of SauceLabs.

I recommend using sauce_connect instead of giving the wheel to karma-sauce-launcher; I’ve found Travis CI’s addon considerably more reliable.

You might wonder why we are using Karma instead of WebDriver.

The answer: these are mainly unit tests. Mocha’s HTML reporter—which is what you get when you run Mocha in a browser—doesn’t have much of a UI to run functional tests against. We’re not checking DOM nodes for attributes, so we don’t script a browser. Though it couldn’t hurt!

What are the artifacts for? While SauceLabs provides tooling to debug a test manually, sometimes all you need is the bundle and whatever Karma was running to make sense of a stack trace (these files are manually dumped into .karma/ by hooking into karma-browserify).

The files get tossed into a public Amazon S3 bucket, though Travis CI does its best to redact the URLs. I should mention: ever since Mocha dropped IE8 support, we haven’t had any failures so weird we needed to look at them. Funny about that.

The Aftermath

Can a good thing even have an aftermath?

I haven’t crunched the numbers—these changes are super new—but it’s obvious that our builds now do more in less time. Typically, the first push to a branch (or PR) will be the slowest to build, and caching will kick in for the next pushes. We’ll see the greatest performance gain on failed builds.

So please send broken PRs to Mocha, so I can pad my numbers.

I don't think I mean that.

After we’ve used the configuration for a three-to-four weeks, I’ll gather up some data, and update this post with an addendum which will delight the reader with fancy charts and graphs and crap like that.

I’ll shut up, so you can start hacking at your .travis.yml. You’re welcome.