Recursive directory removal has landed in Node.js v12.10.0!

This has been a long-standing feature request.  New Node.js developers often express incredulity  when they discover this particular “battery” isn’t included in Node.js.

Over the years, userland modules (rimraf, rmdir, del, fs-extra, etc.) have heroically provided what core did not. Thanks to the superbad maintainers-of and contributors-to these packages!

Here's a little story about how it came to pass—and why something so seemingly simple as rm -rf isn’t necessarily so.

About Node.js’ Filesystem Operations

First, I want to explain a bit about how Node.js works under the hood with regards to filesystem operations.

libuv provides filesystem operations to Node.js.  Node.js’ fs module is just a JavaScript file which provides the fs.* APIs; those APIs call into an internal C++ binding (you could think of this as a “native module”). That binding is glue between libuv and the JavaScript engine (V8).

Here’s an example.  At the lowest level, libuv provides a C API (uv_fs_rmdir) to make the a system call to remove a directory.

const fs = require('fs');

// `rmdir` is just a function which calls into a C++ binding.
// The binding asks libuv to remove the "/tmp/foo" directory.
// Once libuv returns a result, the binding calls `callback`
fs.rmdir('/tmp/foo', function callback(err) {
  if (err) {
    // handle error
  }
});

Importantly, Node.js makes only a single call to libuv above.

In fact, until recently, Node.js’ fs bindings follow a pattern: single calls into libuv. fs.readFile, fs.stat, fs.unlink; these are all just one call.

Oh, that recent change? It was recursive fs.mkdir.  I’ll explain what makes it different.

Shell Operations vs. System Operations

Developers may not think about this much because it’s so well-abstracted by our tools.  Take mkdir, for example:

$ mkdir ./foo

mkdir is a command-line utility (which flavor, exactly, depends on your operating system).  It’s not a system call.  The above command may only execute a single system call, but the following may execute several:

# creates dirs foo, then bar, then baz, ignoring dirs that already exist
$ mkdir -p ./foo/bar/baz

Unless our tools have transactional behavior—they can “commit” or “roll back” operations—it’s possible for this command to partially succeed (though maybe not obvious in this case, but trust me).

What happens if mkdir -p fails halfway through?  It depends. You get zero or more new directories.  Yikes!

If that seems weird, consider that the user may want to keep the directories it did create.  It’s tough to make assumptions about this sort of thing; cleanup is best left to the user, who can deal with the result as they see fit.

How does this relate to Node.js?  When a developer supplies the recursive: true option to fs.mkdir, Node.js will potentially ask libuv to make several system calls—all, some, or none of which may succeed.

Previous to the addition of recursive fs.mkdir, Node.js had no precedent for this behavior.  Still, its implementation is relatively straightforward; when creating directories, the operations must happen both in order and  sequentially—we can’t create bar/baz/ before we create bar/!

It may be surprising, then, that a recursive rmdir implementation is another beast entirely.

There Was An Attempt

I was likely not the first to attempt to implement a recursive rmdir in Node.js at the C++ level, but I did try, and I’ll explain why it didn’t work.

The idea was that a C++ implementation could be more performant than a JavaScript implementation—that’s probably true!

Using mkdir as a template, I began coding.  My algorithm would perform a depth-first traversal of the directory tree using libuv’s uv_fs_readdir; when it found no more directories to descend into, it would call uv_fs_unlink on each file therein. Once the directory was clear of files, it would ascend to the parent, and finally remove the now-empty directory.

It worked!  I was very proud of myself.  Then I decided to run some benchmarks against rimraf. Maybe I shouldn't have!

I found out that my implementation was faster for a very small N, where N is the number of files and directories to remove.  But N didn’t have to grow very large for userland's rimraf to overtake my implementation.

Why was mine slower?  Besides using an unoptimized algorithm, I used recursive mkdir as a template, and mkdir works in serial (as I mentioned above).  So, my algorithm only removed one file at a time.  rimraf, on the other hand, queued up many calls to fs.unlink and fs.rmdir.  Because libuv has a thread pool for filesystem operations, it could speedily blast a directory full of files, only limited by its number of threads!

Note that when we say Node.js is single-threaded, we're talking about the programming model.  Under the hood, I/O operations are multithreaded.

At this point, I realized that if it was going to be “worth it” to implement at the C++ layer—meaning a significant performance advantage which outweighs-the- maintenance-costs-of-more-C++-code—I’d have to rewrite the implementation to manage its own thread pool.  Of course, there’s no great precedent for that in Node.js either.  It’d be possible, but very tricky, and best left to somebody with a better handle on C++ and multithreaded programming.

I went back to the Node.js tooling group and explained the situation.  We decided that the most feasible way forward would be a pure-JavaScript implementation of recursive directory removal.

Let’s Write It In JavaScript!

Well, that was the idea, but we didn’t get very far.  We took a look at the source of rimraf, which is the most popular userland implementation. It’s not as straightforward as you’d expect!  It covers many edge cases and peculiarities (and all of those hacks would need to be present in a Node.js core implementation; it needs to work like a consumer would expect).

Furthermore, rimraf is stable, and these workarounds have proven themselves to be robust over the years that it’s been consumed by the ecosystem.

I won’t attempt to explain what rimraf must do to achieve decent performance in a portable manner—but rest assured it’s sufficiently non-trivial.  So non-trivial, in fact, that it made more sense to just pull rimraf into Node.js core instead of trying to code it again from scratch.

So that’s what we did.

It’s Just rimraf

Ian Sutherland extracted the needed code from rimraf.  In particular, rimraf supplies a command-line interface, and we didn’t need that.  For simplicity (and to eliminate dependencies) glob support (e.g., foo/**/*.js) was also dropped (though it may still have a future).  After this, it was a matter of integrating it into a Node.js-style API, and the needed docs and tests.

To be clear, recursive directory removal in Node.js does not make rimraf obsolete. It does mean that for many use cases, Node.js’ fs.rmdir can get the job done.  Stick with rimraf if you need globs or a portable command-line utility.

Thanks to Isaac Schlueter for rimraf—and to bless Node.js’ copy-and-paste efforts.

In Conclusion

That’s the story of Node.js’ recursive rmdir thus far. Want to help write the rest? Come participate in the Node.js Tooling Group, where we’re looking to make Node.js the best platform it can be for building CLI apps.

Acknowledgements to Isaac Schlueter and Ian Sutherland for reviewing this post.

Node.js will ship a few features in the current release line (v10.x; see release schedule) which may be of interest to those writing command-line applications. Let's take a closer look-see.

fs.readdir() optionally outputs file types

fs.readdir() is simple; it outputs a list of filenames.  Of course, those filenames may represent directories, symbolic links, sockets, devices, hot dogs, etc.  

If you want to know, say, which of those filenames represent directories, you will have to call fs.stat().  This is ultimately kind of silly, because the underlying method in libuv actually provides this information; Node.js simply discards it.

Silliness is forbidden, so Bryan English created nodejs/node#22020 to address this malfeasance.  fs.readdir(), fs.readdirSync(), and fs.promises.readdir() (experimental) will now accept a new option, withFileTypes.  Use it like this:

// Reads contents of directory `/some/dir`, providing an `Array` of
// `fs.Dirent` objects (`entries`)
fs.readdir('/some/dir', { withFileTypes: true }, (err, entries) => {
  if (err) throw err;
  entries.filter((entry) => entry.isDirectory())
    .forEach((entry) => {
      console.log(`${entry.name} is a directory`);
    });
});

// or
const entries = await fs.promises.readdir('/some/dir', {
  withFileTypes: true
});

// or
const entries = fs.readdirSync('/some/dir', { withFileTypes: true });

Above, entries is an array of fs.Dirent objects, which are similar to fs.Stats objects.  These objects contain the same methods as fs.Stats objects, but no other properties except name.  You cannot get information about file’s size from an fs.Dirent object, for example.

The withFileTypes feature provides a more performant and convenient API to work with if you need file type information after reading a directory.  Since it’s optional behavior, it introduces no breaking changes.

fs.mkdir() optionally creates directories recursively

This is mkdir -p.  Over the years of Node.js’ existence, the mkdirp module and its ilk have courageously provided this functionality from userland.  In fact, mkdirp has become ubiquitous, leading some to wonder why it’s not part of the core API.  This author wondered that as well!

The short answer is that Node.js has a “small core” philosophy.  Whether you agree with that philosophy or not, we can all agree mkdirp is a wildly popular module which provides a common filesystem operation.  It’s proven its necessity, and that’s why Node.js merged the Benjamin Coe-created PR nodejs/node#21875.

fs.mkdir(), fs.mkdirSync and fs.promises.mkdir() will now support the recursive option.  Use it like this:

// Creates /tmp/a/apple, regardless of whether `/tmp` 
// and /tmp/a exist.
fs.mkdir('/tmp/a/apple', { recursive: true }, (err) => {
  if (err) throw err;
});

// or
await fs.promises.mkdir('/tmp/a/apple', { recursive: true });

// or
fs.mkdirSync('/tmp/a/apple', { recursive: true });

Note: A recursive fs.mkdir() is not an atomic operation.  It’s…unlikely…to fail halfway through, but it could.

Another Note: There's an open PR concerning how to support feature detection here.

process.allowedNodeEnvironmentFlags: Queryable & Iterable Flags

NODE_OPTIONS is an environment variable supported as of Node.js v8.0.0. If present, it works just like flags passed to the node executable.  The flags allowed in NODE_OPTIONS have a unique property: they do not fundamentally alter the default behavior of the node executable.  What does that mean, exactly?

Flags which don’t fundamentally alter node’s behavior retain two properties:

• Executing node --some-flag will open a REPL
• Executing  node --some-flag file.js will run file.js

A handful of flags are excluded from NODE_OPTIONS, such as --preserve-symlinks, due to security concerns.

This means flags like --help, --version, --check, etc., aren’t supported by NODE_OPTIONS.

Now, if you’re using node with flags in production, you’re probably using flags supported by NODE_OPTIONS.  And you probably have some tests.  Assuming you do have tests, if you’re using a test runner which wraps the node executable (such as Mocha), you will need to pass those same flags to the test runner’s executable.  For example:

# production app
$ node --experimental-modules ./app.js
# test runner
$ mocha --experimental-modules "test/**/*.spec.js"

That’s a nice user experience, and it’s fine and good as long as the test runner supports the flags you want to use.  But it also means  the test runner must add support for any given flag in NODE_OPTIONS and pass it along to Node.js.  This is more manual labor to maintain than you’d expect; many flags are considered “temporary,” and only some flags support swapping any dash (-) for an underscore (_) or vice-versa.

This author created PR nodejs/node#19335 which adds   process.allowedNodeEnvironmentFlags. Using this, test runners and other CLI apps needing to wrap the node executable won’t need to manually track new flags as they are added to (and removed from) Node.js.

To detect whether or not a flag as provided in process.argv is a NODE_OPTIONS flag—and thus a useful one, for our purposes—we can do this:

// cli.js
const command = ['cli2']; 
process.argv.slice(2).forEach((arg) => {
  if (process.allowedNodeEnvironmentFlags.has(arg)) {
    command.unshift(arg);
  } else {
    command.push(arg);
  }
});
command.unshift(process.execPath);
// `command` looks like:
// ['node', '--node-flag', 'cli2', '--cli2-flag']

process.allowedNodeEnvironmentFlags is a Set-like object.  It can't be mutated—add(), delete() and clear() operations will silently fail—and its has() method will return true for any allowed permutation of a NODE_OPTIONS flag.  For example:

process.allowedNodeEnvironmentFlags.has('--stack-trace-limit') // true

process.allowedNodeEnvironmentFlags.has('--stack_trace_limit') // true

process.allowedNodeEnvironmentFlags.has('--stack_trace-limit') // true

has() will also return true for a disallowed (but convenient) permutation: omitted leading dashes.  This means:

process.allowedNodeEnvironmentFlags.has('--experimental-modules') // true

process.allowedNodeEnvironmentFlags.has('experimental-modules') // true

// careful with your underscores!
process.allowedNodeEnvironmentFlags.has('--experimental_modules') // false

Since it’s a Set-like object, you can iterate over it using forEach and other typical methods:

process.allowedNodeEnvironmentFlags.forEach((flag) => {
  // --enable-fips
  // --experimental-modules
  // --experimental-repl-await
  // etc.
});

Only the canonical format (as shown in node --help) of each flag will appear when iterated over; it won’t contain any duplicates.

This addition enables tooling authors to more easily wrap the node executable, and provides runtime insight into the nature of flags in process.argv.

Horn-Tooting

If you’re interested in helping make Node.js a better experience for authors of command-line tools, click on these links, dammit:

• Join our new tooling group!
• Join this Slack for Node.js tooling authors!!!
• Participate in a Node.js user feedback tooling group session!!!!!!!

I wrote this short tutorial for a PDXNode “Nodebots” hackathon. It should work on Windows, Mac, and Linux boxes. I'm reposting it here, since it may be useful.

Prerequisite Software

• Node.js v8.x
• Arduino IDE v1.8.5
• CP2104 driver (found linked here; if your dev board needs a different driver, install that instead)

A toolchain to build native modules may be required. windows-build-tools may be the quickest way to get setup on Windows. macOS users will need to install XCode; Linux users will need to install build-essential, python, and likely some other stuff.

You're welcome to try a newer (or older version) of Node.js--likewise Arduino IDE--but YMMV. The firmware can also be flashed via means other than Arduino IDE, if you are so inclined.

@christianmello writes that macOS users may be able to install the drivers via Homebrew:

$ brew tap homebrew/cask-drivers
$ brew cask install silicon-labs-vcp-driver

Add ESP8266 Board Support to Arduino IDE

1. Launch Arduino IDE
2. Open Preferences
3. Add http://arduino.esp8266.com/stable/package_esp8266com_index.json to the Additional Boards Manager URLs input.
   If you have something in this field already, you can add multiple URLs by delimiting with commas.
4. Click OK
5. Navigate to menu Tools > Board.. > Boards Manager
6. Find esp8266 by ESP8266 Community in the list. Click Install.
7. Once this is complete, click OK.

Flash Dev Board

1. Plug in dev board via USB to your computer.

2. Back in Arduino IDE, in menu Tools > Board, select “Adafruit Feather Huzzah ESP8266” (or your appropriate dev board) from the list.

3. In menu Tools > Port, select the proper port.

   • On Windows, this will be COMx.
   • On Linux, this will likely look like /dev/ttyUSBx.
   • On Mac, this will likely look like /dev/tty.xxxxx.
   • If you only see Bluetooth-related stuff, or otherwise can't find an appropriate port, ensure your driver is working, check your USB cable, check your dev board, etc.

4. In menu File > Examples … Examples for any board > Firmata choose StandardFirmataWifi.

5. Modify this sketch by uncommenting line 85: #define SERIAL_DEBUG. This will allow you to view debug output in Arduino IDE's Serial Monitor.

6. There will be a tab for a file wifiConfig.h. Click this tab to open wifiConfig.h.

7. On line 119, enter the name of your WiFi network (in double quotes), e.g., char ssid[] = "foobar"; where foobar is your WiFi network name.

8. If using a secured network (requiring a password), on line 151, enter the WiFi network password in double quotes, e.g. char wpa_passphrase[] = "foobar-password";. Otherwise, uncomment line 183 for an unsecured network.

9. Click the Upload button (icon is a “right arrow”) in the toolbar. This should flash your board with the firmware.

10. To confirm things are working, click the Serial Monitor button in the toolbar (icon is a magnifying glass).

11. Change the baud rate to 9600

12. Assert that you see something like:

    connected with SON OF ZOLTAR, channel 11
    dhcp client start...
    ip:10.0.0.49,mask:255.255.255.0,gw:10.0.0.1
    IP will be requested from DHCP ...
    Attempting to connect to WPA SSID: SON OF ZOLTAR
    WiFi setup done
    scandone
    .SSID: SON OF ZOLTAR
    IP Address: 10.0.0.49
    signal strength (RSSI): -39 dBm
    

    …where SON OF ZOLTAR is your WiFi network’s name.

    Note and/or copy the IP address. You'll need it later. This is important!

13. Close the serial monitor window.

Install Johnny-Five & Node.js Modules

1. Create a new directory.
2. Run npm init -y to generate an empty package.json.
3. Execute npm install johnny-five etherport-client.
4. Create blink.js:

'use strict';

const {
  EtherPortClient
} = require('etherport-client');
const five = require('johnny-five');
const board = new five.Board({
  port: new EtherPortClient({
    host: '10.0.0.49',
    port: 3030
  }),
  repl: false
});

const LED_PIN = 2;

board.on('ready', () => {
  board.pinMode(LED_PIN, five.Pin.OUTPUT);
  // the Led class was acting hinky, so just using Pin here
  const pin = five.Pin(LED_PIN);
  let value = 0;
  setInterval(() => {
    if (value) {
      pin.high();
      value = 0;
    } else {
      pin.low();
      value = 1;
    }
  }, 500);
});

Replace 10.0.0.49 with the IP address you noted from the serial monitor.

You may need to change LED_PIN to 13. I forget what the number is for the builtin LED on Huzzah (this guide was tested on a Wemos D1 Mini, which is functionally equivalent in most ways which matter).

4. Save this file, and execute it. You should see something like this:

$ node hello.js
1532643089553 SerialPort Connecting to host:port: 10.0.0.49:3030
1532643089556 Connected Connecting to host:port: 10.0.0.49:3030

You should also see the onboard LED blink, toggling on and off every half-second. When satisfied, hit Ctrl-C to quit.

Any corrections or improvements appreciated!

A little less than a year ago, Travis CI introduced a beta feature, Build Stages.
Could it be maintainers in my world (that world being “userland tooling and libraries for Node.js”) don’t even know Build Stages are a Thing? Despite the overwhelming popularity of Travis CI amongst OSS Node.js projects, I haven’t seen a lot of adoption.

Mocha, the JavaScript testing framework, has been a happy user of Travis CI for over six years. This last week, Mocha’s team modified its build matrix to leverage Build Stages (thanks @Outsideris!). I'll share what I've learned.

I have written this article (which gets kind of dry) for users of Travis CI or those otherwise experienced in continuous integration software.

Why Build Stages?

If a project’s build consists of running npm test against a handful of Node.js versions, that project probably doesn’t need Build Stages. While Mocha’s build isn’t as complex as some—like those projects which need to compile or deploy—its build is non-trivial.

Foremost and first, Build Stages allow a build to “fail fast.” Before Build Stages, all jobs in the matrix would run concurrently (with an optional concurrency limit). A project’s build could not run, for example, an initial smoke test to determine if it’s practical to run a more expensive test suite. Build Stages eliminate extra work.

Build Stages can enable better dependency caching. As mentioned, without Build Stages, all jobs would run concurrently, and each cache-compatible configuration would “miss” the cache on its first try. For Node.js projects, this means repeated installation of dependencies via npm install, and ain’t nobody got time for that. By “warming up” Travis CI’s cache in preparation, we can prepare npm to work quickly in subsequent cache-compatible Build Stages.

By leveraging Build Stages—and npm’s new features—Mocha can run more tests in less time. Join me in the Bonesea Skullenger, and we’ll dive deep.

Swimming elasipod sea cucumber. Photo from Vailulu'u 2005 Exploration, NOAA-OE / NOAA

About Mocha’s Old Build

Before the changes, this is what Mocha’s .travis.yml looked like (with irrelevant sections removed):

language: node_js

matrix:
  fast_finish: true
  include:
    - node_js: '9'
      env: TARGET=test.node COVERAGE=true
    - node_js: '8'
      env: TARGET=test.node
    - node_js: '6'
      env: TARGET=test.node
    - node_js: '4'
      env: TARGET=test.node
    - node_js: '8'
      env: TARGET=lint
    - node_js: '8'
      env: TARGET=test.browser

before_install: scripts/travis-before-install.sh
before_script: scripts/travis-before-script.sh
script: npm start $TARGET
after_success: npm start coveralls

addons:
  artifacts:
    paths:
      - .karma/
      - ./mocha.js
  sauce_connect: true
  chrome: stable
cache:
  directories:
    - ~/.npm

Much pain!

1. fast_finish does nothing unless you have jobs within an allowed_failures mapping; Mocha does not.
2. We can optimize caching and installation of dependencies:
   1. Each job has its own cache due to the use of environment variables, but (most) every job sharing a Node.js version should share a cache.
   2. node_modules isn’t cached in the cases where it should be.
   3. npm ci is now a thing
   4. Mocha’s dev dependencies include a fair number of native modules, which we don’t always need
3. Browser tests create artifacts (essentially bundles created by karma-browserify and the bundled mocha.js—we use these for debugging esoteric failures on SauceLabs’ browsers) so the upload paths contain a steaming pile of nothing after most jobs
4. Likewise, we’re starting Sauce Connect and installing headless Chrome for jobs that won’t use it
5. One (1) job generates any coverage at all, yet every job attempts to send a report to Coveralls
6. The before_install script runs some smoke tests, but duplicates effort by running three (3) times in Node.js 8
7. The before_script script creates .karma/, but Travis CI’s cache creates it first.

Many of these issues are due to my own ignorance, as git blame will tell you. Those problems can be solved by actually reading Travis CI’s documentation like I should have; we can solve the rest with Build Stages.

Mocha’s New Build using Build Stages

I’ll analyze the new .travis.yml in parts for better context (you can see it in its entirety, if you wish).

Defining the Build Stage Order

It’s optional, but a project will usually want to run stages in order (to enable fast failure, and tasks like conditional deploys).

stages:
  - smoke # this ensures a "user" install works properly
  - precache # warm up cache for default Node.js version
  - lint # lint code and docs
  - test # all tests

In the smoke stage, Mocha runs its smoke (or “sanity”) tests. We want to do this stage first, because if it fails, somebody screwed up big. Doing further work would be a waste.

The precache stage, then, installs dependencies which multiple jobs in subsequent Build Stages will reuse. The single job in the lint stage will hit this cache, as well as two other jobs in the test stage.

Default Job Configuration

At the top level of .travis.yml, we can establish some defaults. Jobs in Build Stages will use these unless they override the options.

language: node_js
node_js: '9'

Jobs won’t change the project’s language, which is node_js.

Mocha runs its complete test suite against all maintained LTS versions of Node.js, in addition to the “current” release. At the time of this writing, those versions are 4.x, 6.x, 8.x and 9.x—these are the only versions of Node.js which Mocha supports! The Build Stages contain jobs which don’t depend on any specific version (lint checks and browser tests), so we may as well use the latest—and often, fastest—Node.js version.

Warning: rabbit hole ahead.

Photo by Tatiana Gettelman / Flickr

Efficient Installation of npm v5.8.0

At the time of this writing, the version of default npm that ships with 8.x and 9.x is v5.6.0. npm ci wasn’t available before v5.8.0, which is the latest version.

Travis CI manages its Node.js versions with nvm. Travis CI also runs nvm install <version> before it reaches into its cache. That means, to use v5.8.0 of npm, a build would naively do this:

before_script: npm install -g npm@5.8.0

That’ll run for every job, which is slow. A build could attempt to cache some subset of ~/.nvm itself (where nvm keeps its installed Node.js versions and globally-installed packages, including npm), but that’s going to contain the node executable and other cruft. Anyway, trying to cache whatever npm install -g npm installed is a dead-end, at the time of this writing. But there’s another way.

If Travis CI’s caching worked with individual files—or supported exclusion—this solution would be more viable. Its granularity stops at the directory level.

Here’s what we do (and these are job defaults, remember):

before_install: |
  [[ ! -x ~/npm/node_modules/.bin/npm ]] && {
    cd ~/npm && npm install npm
    cd -
  } || true
env: PATH=~/npm/node_modules/.bin:$PATH
cache:
  directories:
    - ~/.npm # cache npm's cache
    - ~/npm # cache latest npm

1. Our before_install Bash script checks for the executability (executableness?) of a file, ~/npm/node_modules/.bin/npm. Note that ~/.npm is not ~/npm.
2. If ~/npm/node_modules/.bin/npm is not executable, we assume this was a cache miss. Navigate into ~/npm and use npm to install the latest version of npm local to this directory. After this step, ~/npm will contain node_modules and package-lock.json.
3. We must navigate back to our working copy after navigating away, which is what cd - does.
4. If a job hits the cache, our npm executable is ready, and the script ends with great success (true).
5. We set the PATH to look in ~/npm/node_modules/.bin/ before anything else, so it finds our custom npm executable instead of the one nvm installed.
6. We cache ~/.npm, which is npm’s cache. Right? Right.
7. We cache ~/npm, which contains our custom-installed npm.

The point of this madness is to avoid an npm self-upgrade on every job. Not pretty, but it works.

Using npm ci

By keeping the installation separate from the working copy, we avoid extraneous dependencies. Why is this important? Because:

install: npm ci --ignore-scripts

Armed with npm v5.8.0, we can use npm ci instead of npm install (in most cases; I’d wager the majority of projects looking to use npm ci should always use it). One reason npm cioffers better consistency is that it blasts the local node_modules and re-creates it from scratch. That means we can’t throw anything in there (npm is not a direct dependency of Mocha) that doesn’t belong.

Mocha uses the --ignore-scripts flag in most of its jobs. npm’s lifecycle scripts invoke native module compilation; Mocha consumes native modules when building docs or running browser tests. We don’t test the doc-building scripts themselves, so that leaves a single case where Mocha needs a native module to run a test suite.

This situation isn’t unique to Mocha, but neither is it ubiquitous. Any given dependency of a project may use an install, postinstall, or an infamous prepublish script. Because most of Mocha’s dependencies don’t do this (thanks, dependencies!) , we can get away with it.

I don’t know of any way to tell npm to only avoid compilation of native modules; it’s either “ignore all scripts” or “run all scripts.”

After we have established the default behavior, we can define our jobs. Let’s analyze each stage.

Build Stage 1: Smoke

As noted above, our stages run in order (but the jobs within them run concurrently), and the smoke stage is the first. Here’s its definition:

- &smoke
  stage: smoke
  install: npm install --production --no-shrinkwrap
  script: >
    ./bin/mocha --opts /dev/null --reporter spec 
    test/sanity/sanity.spec.js
  cache:
    directories:
      - ~/.npm
      - node_modules

- <<: *smoke
  node_js: '8'

- <<: *smoke
  node_js: '6'

- <<: *smoke
  node_js: '4'

If you’re unfamiliar with anchors and aliases in YAML (like I was), well… there it is.

An Aside: YAML Anchors & Aliases

It defines an anchor, &smoke. We can then refer to this anchor using an alias, *smoke. The <<: syntax means the mapping will be merged with the mapping from the anchor. JavaScripters could think of it like this:

const baseSmoke = {
	stage: 'smoke',
	install: 'npm install --production --no-shrinkwrap'
	// etc
};

const smokeStage = [
	baseSmoke, 
  Object.assign({}, baseSmoke, {node_js: '8'}),
  Object.assign({}, baseSmoke, {node_js: '6'}),
  Object.assign({}, baseSmoke, {node_js: '4'})
];

If we serialize smokeStage into JSON, this is the result (I apologize for the verbosity, but this is why a hypothetical .travis.json would suck):

[
  {
    "stage": "smoke",
    "install": "npm install --production --no-shrinkwrap"
  },
  {
    "stage": "smoke",
    "install": "npm install --production --no-shrinkwrap",
    "node_js": "8"
  },
  {
    "stage": "smoke",
    "install": "npm install --production --no-shrinkwrap",
    "node_js": "6"
  },
  {
    "stage": "smoke",
    "install": "npm install --production --no-shrinkwrap",
    "node_js": "4"
  }
]

As you can see, we have our top-level defaults, but we can also define defaults within individual stages via YAML voodoo.

Why Smoke Test, Anyway?

Photo by Marcus Kauffman / Unsplash

The goal is to establish a baseline of functionality in a minimal amount of time. This baseline will be different for every project! In Mocha’s case, we want to minimize the likelihood a npm install mocha somehow misses a dependency.

If you’re wondering, yes, this has happened—a dependency was living in package.json’s devDependencies when it should have been in dependencies. Raise your hand if you’ve done that before.

Since we can’t npm install mocha, the next best thing is running npm install --production --no-shrinkwrap in the working copy. This mimics the result a user would get; we don’t install Mocha’s development dependencies, and we ignore package-lock.json (read more about package-lock.json; npm never publishes it to the registry).

Another way to do this could be to npm install the current changeset directly from GitHub, but we already have the working copy cloned. And as of npm v5.x, running npm install /path/to/working/copy results in a symlink, so that’s not workable.

Once npm has installed Mocha’s production dependencies, the bin/mocha executable should run a simple test like:

describe('a production installation of Mocha', function () {
  it('should be able to execute a test', function () {
    assert.ok(true);
  });
}); 

Because Mocha’s own mocha.opts contains references to development dependencies, we must ignore it; the flag --opts /dev/null is a hacky workaround which effectively disables mocha.opts. Normally, I’d put this kind of nonsense in package-scripts.js and let nps run it, but we don’t have nps at our disposal here. Though npx could…

If that simple test runs OK—for each supported version of Node.js—Mocha has passed its smoke tests, and we can move on to the precache stage.

Build Stage 2: Pre-cache

What is it? It is this:

- stage: precache
  script: true

We run everything in the default configuration except an actual build script; true is POSIX shell for “it worked.” That’s enough to create a “warmed-up” cache of our development dependencies for Node.js v9.x, as well as a cache of the latest npm version.

Build Stage 3: Lint

The lint stage is the first to hit our pre-cached dependencies. The latest npm is already present, and all the dependencies live in npm’s cache. We get an even faster install (npm ci, if you recall) since we can ignore scripts.

Since running linter(s) is far less time-consuming than running tests, we may as well run these before Real Tests.

Like precache, this Build Stage has one job:

- stage: lint
  script: npm start lint

npm start lint kicks off our linters (ESLint and markdownlint-cli).

Some months ago, Mocha dropped the Makefile we were using for Kent C. Dodds’ wonderful nps; npm start calls nps. Formerly known as p-s, nps is an elegant task runner (no plugins necessary, unlike Grunt or Gulp). I highly recommend it if your scripts in package.json has become unwieldy.

Like with smoke tests, if the lint checks fail, then we abort the build.

Build Stage 4: Test

Mocha runs its main test suites concurrently in the fourth stage.

You’ll notice there’s no stage mapping in any of the items below. If you look at the entire file, you’ll see that these are the first items in the jobs.include mapping; we must also keep them together. Jobs lacking a stage will use the same stage as the previous job in the list; if there is no previous job, the default stage is test. Ahh, the wonders of convention…

Like jobs in previous stages, these inherit from the default job configuration.

Node.js Tests

The first is our test against the default Node.js version (9.x), which also computes coverage:

- script: COVERAGE=1 npm start test.node
  after_success: npm start coveralls

You’ll note that the environment variable COVERAGE is not defined in an env mapping. This is because variables defined in env create a unique cache configuration—it’d bust the pre-cache and we’d miss everything in it! This environment variable causes our test.node script (found in package-scripts.js) to invoke mocha via nyc.

The unique after_success script will fire coverage information generated by nyc to Coveralls using node-coveralls. If any tests fail, after_success does not run, as you might guess.

These next three (3) jobs are identical, except the Node.js version. These all miss the cache, because we pre-cache Node.js 9.x only (and we don’t use these configurations again). Since we’re already computing coverage in the previous job, we omit the COVERAGE variable.

- &node
  script: npm start test.node
  node_js: '8'

- <<: *node
  node_js: '6'

- <<: *node
  node_js: '4'

The fascinating and frightening YAML anchors (and associated aliases) also appear above; these entries expand exactly as explained earlier.

What’s left?

Browser Tests

- script: npm start test.bundle test.browser
  install: npm ci
  addons:
    artifacts:
      paths:
        - .karma/
        - ./mocha.js
    chrome: stable
    sauce_connect: true

Note the custom install script; every other install script uses —ignore-scripts. This one doesn’t, because it needs some compiled modules to bootstrap headless Chrome. The chrome addon provides a headless Chrome executable to the job.

This is actually two suites; test.bundle is a test launched via the Node.js mocha executable which ensures the bundle we build (via browserify) retains compatibility with RequireJS. Then, Karma handles the test.browser suite.

We abuse NODE_PATH to trick karma-mocha into running our tests with our own mocha.js bundle. Don't do this.

When the browser tests run in a local development environment, they run in headless Chrome by default. On Travis CI, we add handful of “real” browsers, by the grace of SauceLabs.

I recommend using sauce_connect instead of giving the wheel to karma-sauce-launcher; I’ve found Travis CI’s addon considerably more reliable.

You might wonder why we are using Karma instead of WebDriver.

The answer: these are mainly unit tests. Mocha’s HTML reporter—which is what you get when you run Mocha in a browser—doesn’t have much of a UI to run functional tests against. We’re not checking DOM nodes for attributes, so we don’t script a browser. Though it couldn’t hurt!

What are the artifacts for? While SauceLabs provides tooling to debug a test manually, sometimes all you need is the bundle and whatever Karma was running to make sense of a stack trace (these files are manually dumped into .karma/ by hooking into karma-browserify).

The files get tossed into a public Amazon S3 bucket, though Travis CI does its best to redact the URLs. I should mention: ever since Mocha dropped IE8 support, we haven’t had any failures so weird we needed to look at them. Funny about that.

The Aftermath

Can a good thing even have an aftermath?

I haven’t crunched the numbers—these changes are super new—but it’s obvious that our builds now do more in less time. Typically, the first push to a branch (or PR) will be the slowest to build, and caching will kick in for the next pushes. We’ll see the greatest performance gain on failed builds.

So please send broken PRs to Mocha, so I can pad my numbers.

I don't think I mean that.

After we’ve used the configuration for a three-to-four weeks, I’ll gather up some data, and update this post with an addendum which will delight the reader with fancy charts and graphs and crap like that.

I’ll shut up, so you can start hacking at your .travis.yml. You’re welcome.

Like other “serverless” platforms, an OpenWhisk JavaScript Action may be a single .js file. As functions grow beyond trivial—and begin to depend on third-party modules—that single, forlorn .js file can no longer shoulder the burden.

The OpenWhisk Documentation suggests simply throwing an entire project—node_modules and all—into a .zip file. Of course, that’s wasteful and silly—especially if most of node_modules is full of development dependencies.

Now, I’m not one to read documentation unless I get stuck, so I didn’t realize that the docs present an alternative: bundle an Action with webpack. Faced with the unpalatable task of zipping up my entire project dir, I knew I wanted to bundle, but I didn’t reach for webpack—I used Rollup.

Why Rollup?

In this article, Rich Harris (the author of Rollup) writes that webpack’s scope is to help bundle single-page applications (SPAs). Rollup, on the other hand:

“Rollup was created for a different reason: to build flat distributables of JavaScript libraries as efficiently as possible, taking advantage of the ingenious design of ES2015 modules.”

—Rich Harris

The key for us in the above quote is “flat distributable.” We want to upload our Action as a single .js file. That is precisely what Rollup provides—with little extra ornamentation.

Rollup doesn’t do stuff like code splitting nor hot module replacement. We don’t need these features anyway, since we’re not bundling SPAs. Heck, we don’t even need ES modules all the way down (though we won’t get all of the benefits of Rollup’s tree-shaking abilities)—its plugin ecosystem has us covered.

Read on for an example configuration.

An Example Action Using Rollup

Here’s the lovely example action which the OpenWhisk docs provide. It’s intended to be uploaded within .zip file which also includes node_modules and everything else in its project folder:

function myAction(args) {
  const leftPad = require("left-pad")
  const lines = args.lines || [];
  return { padded: lines.map(l => leftPad(l, 30, ".")) }
}

exports.main = myAction;

Similarly to OpenWhisk's webpack example, the above needs slight modification to work with Rollup. Let’s write a naïve conversion. Create index.js:

import leftPad from 'left-pad';

function myAction(args) {
  const lines = args.lines || [];
  return { padded: lines.map(l => leftPad(l, 30, ".")) }
}

export const main = myAction;

This may seem a little weird, but note that OpenWhisk executes our .js file as if it were at the top level (no, I’m not sure why). The webpack example in OpenWhisk’s docs make this explicit by assigning the function to global.main; const main = myAction is equivalent.

However, since Rollup aggressively tree-shakes, casually assigning myAction to an unused variable is verboten, and myAction would be trashed. This is also why we can’t just write export {myAction as main}; it doesn’t create the global variable converted to CommonJS module format by Rollup. To address this, just export main; we can use it later when we write our tests!

Even though our dependencies don’t need to use ES modules, our sources do, so we import left-pad at the top. Then, myAction will become the default export. We’ll see how this works in the Rollup config below.

Need a refresher on ES modules? I recommend the modules chapter of Dr. Axel Rauschmayer’s excellent book, Exploring ES6.

If we don’t yet have a package.json, we can create one via npm init -y or copy/paste:

{
  "name": "my-action"
}

Then, install rollup, and left-pad (assuming npm v5.0.0 or newer):

$ npm i rollup@^0.57.1 -D

+ rollup@0.57.1
added 56 packages from 109 contributors in 2.725s

$ npm i left-pad@^1.2.0

+ left-pad@1.2.0
added 1 package from 1 contributor in 1.222s

The versions of rollup and left-pad above, and any subsequent versions, are intended to future-proof this tutorial. We may be able to just use the latest versions of any of these; YMMV.

By default, Rollup looks for a config file in rollup.config.js, so let’s create that now:

// notice: this is an ES module
export default {
  input: 'index.js',
  output: {
    file: 'dist/my-action.js',
    format: 'cjs'
  }
};

This configuration declares we will use CommonJS (Node.js-style; require() and module.exports, exports, etc.).

CommonJS (cjs) format isn’t actually required by OpenWhisk, as an IIFE or UMD bundle would work, but actually to suppress annoying warnings; if we don’t use cjs, it will assume we are bundling for a browser and take exception to what we’re trying to do.

Invoke Rollup now, and see the warning it generates:

$ node_modules/.bin/rollup -c

index.js → dist/my-action.js...
(!) Unresolved dependencies
https://github.com/rollup/rollup/wiki/Troubleshooting#treating-module-as-external-dependency
left-pad (imported by index.js)
created dist/my-action.js in 19ms

In other words, Rollup doesn’t know what to do with left-pad. In fact, it’s just going to assume it can be retrieved via require()! If we dump the contents of dist/my-action.js, we see:

'use strict';

Object.defineProperty(exports, '__esModule', { value: true });

function _interopDefault (ex) { return (ex && (typeof ex === 'object') 
  && 'default' in ex) ? ex['default'] : ex; }

var leftPad = _interopDefault(require('left-pad'));

function myAction(args) {
  const lines = args.lines || [];
  return { padded: lines.map(l => leftPad(l, 30, ".")) }
}

const main = myAction;

exports.main = myAction;

Rollup has converted our ES module to Node-style, “CommonJS” exports, which is good. But…

We wanted to bundle left-pad, yet it didn’t happen; our bundle calls require('left-pad') like it’s in a node_modules/ somewhere. This won’t do; we only want to upload dist/my-action.js, and no part of node_modules/. Where’s the beef?

Other than the fact it broke the internet, left-pad has two problems:

1. left-pad was installed by npm and lives in node_modules/left-pad.
2. left-pad uses CommonJS exports.

Rollup makes few assumptions about our environment; indeed, both of the above cases necessitate a plugin. Let’s install them now:

$ npm i rollup-plugin-commonjs@^9.1.0 rollup-plugin-node-resolve@^3.3.0 -D

+ rollup-plugin-commonjs@9.1.0
+ rollup-plugin-node-resolve@3.3.0
added 9 packages from 6 contributors in 2.037s

Modify rollup.config.js as seen here:

import commonjs from 'rollup-plugin-commonjs';
import resolve from 'rollup-plugin-node-resolve';

export default {
  input: 'index.js',
  output: {
    file: 'dist/my-action.js',
    format: 'cjs'
  },
  plugins: [resolve(), commonjs()]
};

If we try bundling again, the warning is gone…

$ node_modules/.bin/rollup -c

index.js → dist/my-action.js...
created dist/my-action.js in 48ms

…and if we view dist/my-action.js, we will see that the entirety of left-pad is included. Neat! We can then deploy the action via:

$ wsk action create my-action dist/my-action.js

Or, if we’re using the IBM Cloud CLI (formerly Bluemix CLI):

$ bx wsk action create my-action dist/my-action.js

Next, I’ll discuss a few common recipes I've found useful.

OpenWhisk & Rollup Recipes

Photo by Pratiksha Mohanty / Unsplash

Here are some problems (and solutions, thankfully) I’ve encountered while experimenting with Rollup and OpenWhisk.

Using Built-In Node.js Modules

What if we wanted some extra logging? Node.js’ util.inspect() is super handy; we can use it to print only a few items from our lines array:

import leftPad from 'left-pad';
import {inspect} from 'util';

function myAction(args) {
  const lines = args.lines || [];
  console.log(inspect(lines, { maxArrayLength: 10 }));
  return { padded: lines.map(l => leftPad(l, 30, ".")) }
}

export const main = myAction;

Right?

Nope nope nope:

$ node_modules/.bin/rollup -c

index.js → dist/my-action.js...
(!) Unresolved dependencies
https://github.com/rollup/rollup/wiki/Troubleshooting#treating-module-as-external-dependency
util (imported by index.js)
created dist/my-action.js in 44ms

Fortunately, this is straightforward to fix. We add util to the external Array in rollup.config.js:

import commonjs from 'rollup-plugin-commonjs';
import resolve from 'rollup-plugin-node-resolve';

export default {
  input: 'index.js',
  output: {
    file: 'dist/my-action.js',
    format: 'cjs'
  },
  plugins: [resolve(), commonjs()],
  external: ['util']
};

If I end up using many builtin modules, I like to save myself some trouble and use rollup-plugin-auto-external.

Consuming JSON

What if we want to read a .json file? We might write this:

import leftPad from 'left-pad';
import {inspect} from 'util';
import {version} from './package.json';

function myAction(args) {
  const lines = args.lines || [];
  console.log(inspect(lines, { maxArrayLength: 10 }));
  return { padded: lines.map(l => leftPad(l, 30, '.')), version };
}

export const main = myAction;

But that would fail!

$ node_modules/.bin/rollup -c

index.js → dist/my-action.js...
[!] Error: Unexpected token
package.json (2:8)
1: {
2:   "name": "openwhisk-rollup",
           ^
3:   "version": "1.0.0",
4:   "description": "",

Shock! Rollup expects JavaScript files?! The solution is to pull in rollup-plugin-json:

$ npm install rollup-plugin-json@^2.3.0

+ rollup-plugin-json@2.3.0
added 1 package in 1.629s
$ node_modules/.bin/rollup -c

index.js → dist/my-action.js...
created dist/my-action.js in 41ms

Instead of just embedding the entire file, it will grab whatever portion of the .json file we actually use, effectively tree-shaking the JSON itself. In dist/my-action.js, we’ll see:

var version = "1.0.0";

Good work, rollup-plugin-json.

Third-Party Modules in The Environment

IBM provides many commonly used modules in its OpenWhisk service, IBM Cloud Functions. The list of these pre-installed modules which Actions can use is found in the documentation.

Practically speaking, this means we don’t need to bundle these modules. They can be ignored, just like a built-in, as shown above. We add any which we're using to the external Array in rollup.config.js.

It's still helpful to npm install any of these which we’re using (for code completion, testing, etc.). They must be added to the external Array, regardless.

Alternatively, use rollup-plugin-auto-external with option {dependencies: false}; then add the modules (as minimatch globs) which we do want to bundle to the Array include property of the commonjs plugin’s configuration object. Here’s an example of a Rollup config where we consume the pre-installed request-promise module, but exclude it from the bundle:

import resolve from 'rollup-plugin-node-resolve';
import commonjs from 'rollup-plugin-commonjs';

export default {
  // (imagine there is more configuration here)
  plugins: [
    resolve(),
    commonjs({
      // left-pad is the ONLY dependency which is bundled
      include: ['node_modules/left-pad/**']
    }),
    autoExternal({
      // continue to exclude "util", or any other built-in
      builtins: true,
      // default is true, but we still must bundle left-pad, so this is false.
      dependencies: false
    })
  ],
  // request-promise is a dependency in package.json
  external: ['request-promise']
}

Smooshing The Bundle

Photo by Sharon Drummond / Flickr

If we follow OpenWhisk's documentation on using webpack to the letter, our resulting bundle will be minified.

In smaller bundles, this ain’t matter. But with larger bundles, we should shave some milliseconds off of startup time.

To minify with Rollup, install rollup-plugin-uglify:

$ npm i rollup-plugin-uglify -D

Then:

import uglify from 'rollup-plugin-uglify';
import resolve from 'rollup-plugin-node-resolve';
import commonjs from 'rollup-plugin-commonjs';

export default {
  // (imagine there is more configuration here)
  plugins: [
    resolve(),
    commonjs(),
    uglify()
  ]
}

We can squeeze more bytes out of this by providing options to the uglify() function. By default, it doesn’t mangle top-level variable names. This is how we’d do that:

plugins: [
  resolve(),
	commonjs(),
  uglify({
    compress: {
      toplevel: true
    }
  })
]

See the UglifyJS docs for more options.

Obligatory Wrap-Up

Let me remind the reader what the reader read:

• We learned the difference between Rollup and webpack
• We learned how to use Rollup to bundle an OpenWhisk Action
• We learned how to
  • Use built-in Node.js modules
  • Bundle JSON files
  • Consume pre-installed third-party modules
  • Minify our bundle

So far, I’ve found Rollup works just as well as webpack for OpenWhisk Action deployment. I don’t see a clear winner, other than they both beat .zip files. Until I do, I’ll probably continue using Rollup, just ‘cause. What I’d really like to see is a zero-configuration, purpose-built bundler for Node.js OpenWhisk actions. Hmmm…

I love JetBrains’ IDEs. I’ve been a faithful user since PyCharm’s release, seven years ago.

As of late, if I’m watching a presentation, and someone is writing code in an editor, that editor is almost always VSCode.

Something’s up, and I’m going to get to the bottom of it. People rave about this thing.

I'll answer some questions for myself—and, with luck, maybe I can save the JetBrains-faithful some time and energy. I aim to discover:

• Does it support my key bindings, or will I need to relearn everything?
• What’s the analog of a “Run Configuration?”
• What’s debugging look like? How’s the source map support?
• How easy is it to configure?
• How’s the extension ecosystem?
• How does the VCS (Git) integration differ?
• What’s the story on inline errors or warnings?
• How smart is it about types and code completion?

I’ll be looking at this from the standpoint of a JavaScript developer, so I’ll write “WebStorm,” but I really mean “a JetBrains IDE.”

I’m certainly interested in how VSCode handles Python and C/C++, but I’m not going to explore it in this post.

First Impressions

I used Homebrew Cask to install it:

$ brew cask install visual-studio-code
==> Satisfying dependencies
==> Downloading https://az764295.vo.msecnd.net/stable/f88bbf9137d24d36d968ea6b2911786bfe103002/VSCode-darwin-stable.zip
==> Verifying checksum for Cask visual-studio-code
==> Installing Cask visual-studio-code
==> Moving App 'Visual Studio Code.app' to '/Users/boneskull/Applications/Visual Studio Code.app'.
==> Linking Binary 'code' to '/usr/local/bin/code'.
🍺  visual-studio-code was successfully installed!

This takes ~4s on my 2016 MBP—but I don’t have any extensions installed yet.

I’m greeted with this:

VSCode’s “Welcome Page”

It has also opened a web page in Chrome:

Online tutorials & such for VSCode

I ignore the web page (thanks, but no thanks) and click a few of the “Install support for…” links under Tools and languages to get some basic extensions installed. I want to avoid customizing too much at the outset. Since C/C++ isn’t listed on the “Welcome Page”, I dig into to the extensions and … can’t help myself from installing a bunch of extensions. Oops.

I am, however, elated to report that the JetBrains IDE Keymap is a thing, and it works great.

I go ahead and open up my Mocha working copy…

Summary

• Install “Extension Packs” to get started quickly. You will install many extensions.
• There’s plenty of tutorials.
• If you're using the default bindings in WebStorm, you likely want to install the JetBrains IDE Keymap extension.
• Key bindings are at once more powerful and complex than WebStorm. It’s difficult to discover what a particular keystroke does at any given time, but also supports conditionals, for a nearly absurd level of control.

Close Encounters with Version Control

OK, so I want to edit Mocha’s CHANGELOG.md. But I know origin has changes I need to pull.

Happily, VSCode understands this is, in fact, a working copy. To pull, I found a little “refresh” button in my status bar, and clicked it. I think it worked? It said "sync." What's a "sync"?

I’m unsure what just happened. Which changesets did Git pull? I want to look at the history. After searching in vain, I realize that there is no built-in support for Git history, and I’m going to need to grab an extension for this.

GitLens seems to solve this problem (and others). But there are many Git-related extensions for VSCode. This is a drawback of the “small core” philosophy of VSCode (this reminds me of the Node.js ecosystem). To VSCode’s credit, it aids discovery with tags, filters and sorting.

GitLens does some oddball things like “inline blame” and “code lens” (which is another view into “blame”? I don’t get it). I want to turn this noisy stuff off.

Hint: run GitLens: Toggle Code Lens and GitLens: Toggle Line Blame Annotations from the Command Palette.

GitLens then provides Git history in the left sidebar. The presentation of the repo is a little disorienting (there are so many trees, it's like a forest), but I do see the pulled changesets.

Whew.

I’ve made my changes to CHANGELOG.md, and it’s time to commit. VSCode helpfully marks the file with a big M in the file list. +1.

I find Git: Commit via the Command Palette, and realize I could have used my trusty ⌘-K. But I’m prompted that the stage is empty.

If you only use WebStorm’s built-in version control client (I don't), this will be culture shock. VSCode uses the stage, like literally every other Git client except WebStorm’s.

I go ahead and commit everything (including unstaged changes; this would be Git: Commit All if I wanted to avoid the prompt), and push.

Not the nicest initial experience, but I’m confident it’ll be smoother sailing from here.

Next, I’ll run Mocha’s test suites to prepare for publishing.

Summary

• You will likely want to install the Git extension pack, due to VSCode’s basic client implementation.
• VSCode uses the stage, unlike WebStorm.
• VSCode automatically enables Git support for working copies instead of prompting you into oblivion.

Tasks in VSCode

I think a “Task” is perhaps a “Run Configuration” or “External Tool”.

“Tasks” are not the same as “Tasks” in a WebStorm, which is WebStorm’s (leaky) abstraction around issues, staged changes and branching.

Let’s see what “Configure Tasks” does…

What's do you call a widget like this, anyway?

Ooook. that needs some further explanation, but sure. Is it trying to automatically detect my npm scripts? For reference, Mocha’s scripts in its package.json is literally just:

{
  "scripts": {
    "prepublishOnly": "nps test clean build",
    "start": "nps",
    "test": "nps test"
  }
}

I’ll roll the dice with npm: test.

VSCode creates and opens a .vscode/tasks.json file:

At least it's not XML, amirite?

Fascinating. I click the link and learn about this file. It’s unclear whether VSCode intends for the user to commit .vscode/ to VCS (I don’t do so; in fact, I add .vscode/ to my .gitignore post-haste).

VSCode likely didn’t need me to “configure” the Task—it discovered the script itself. I execute Run Task…, choose npm: test, and the output opens in a terminal, as you’d expect.

I’m now certain “Tasks” are analogous to “External Tools”. Like in WebStorm, the user (seemingly) cannot debug a Task, and there’s little integration. VSCode ships with some helpers for common build tools (unfortunately, nps is not one of them). Like WebStorm, the user has free rein to create a “shell”-based Task.

I’m still looking for the analog of a “Run Configuration,” which appears to be a “Debug Configuration,” though it’s just called “Configuration” under VSCode’s “Debug” menu. Next, I’ll take it for a test-drive. Whatever the hell it’s called.

Summary

• “Tasks” in VSCode are analogous to WebStorm’s “External Tools”.
• Configure tasks via JSON files. This isn’t too awesome, but VSCode provides validation/completion when editing, which is better than nothing.
• You must choose whether or not to commit .vscode/ to VCS. I’ve never had success committing any sliver of .idea/ to VCS, but maybe the story is different here. Don’t look at me; find some other guinea pig.

Trivia: VSCode recognizes the filetype of tasks.json as “JSON with Comments”, which, AFAIK, is unadulterated, imaginary nonsense. Is it JSON5 or not?

Debugging

Photo by Mink Mingle / Unsplash

First, I install the Node.js Extension Pack, since I figure I’ll need it to debug properly.

From VSCode’s menu, I open Debug > Open Configurations. I’m presented with a new file, .vscode/launch.json. Like tasks.json, this is my config file.

This menu item inexplicably corresponds to the command Debug: Open launch.json.

In Node.js

Do you want the good news or bad news first? I don't care.

The bad news is, I can’t just throw npm test in launch.json and expect my breakpoints to get hit. Why not? Because npm spawns nps, which spawns ten different mocha processes in series, each which spawn _mocha, and many of which spawn _mocha again.

This is by no means VSCode's fault, but the story around debugging subprocesses in Node.js is a sad one. I'd be foolish to expect a miracle.

The good news is that if I choose some subset of the tests to run with bin/_mocha, debugging works well (unless I want to debug more child processes). It’s a solid debugging experience, though lacking some of the bells & whistles of WebStorm’s debugger.

In a Browser

Debugging tests with Karma, awkward is.

In a WebStorm, you create Karma-based run configuration, point it to your config file, specify any particular browsers or other extra options, and push the button. It works well, even if you happen to be bundling your code with karma-browserify, which Mocha is.

This is the VSCode experience:

1. You need to install the Chrome Debugger extension.
2. Create a chrome debug configuration identical to:

{
  "type": "chrome",
  "request": "attach",
  "name": "Attach to Karma",
  "address": "localhost",
  "port": 9333,
  "pathMapping": {
    "/": "${workspaceRoot}/",
    "/base/": "${workspaceRoot}/"
  }
}

3. Modify your karma.conf.js (ugh, really?) to add a custom launcher to your setup object. The port below must be the same port as above. Hope it’s not in use!

{
  customLaunchers: {
    ChromeDebug: {
     base: 'Chrome',
      flags: ['--remote-debugging-port=9333']
    }
  }
}

4. Start Karma (you can create a Task to do this): karma start —-browsers ChromeDebug --auto-watch --no-single-run. Leave it running.
5. Run your “Attach to Karma” debug configuration; choose it from “Debug” > “Run Configuration…”.

At this point, you can set breakpoint(s) by clicking in the editor’s gutter, though they will not immediately be enabled.

There appears a small, odd, quasi-movable toolbar near the top of the window. Within this impish toolbar is a “refresh”-looking button; click it to re-run the tests. VSCode will then be able to discover which scripts/files Karma loaded. If you’re lucky, it’ll even hit your breakpoint!

The above took a solid hour to figure out, even with a few scattered examples out there.

The last thing I want to evaluate is VSCode’s “IntelliSense” capabilities.

Summary

• “Run Configurations” = “Debug Configurations”. I’m calling them “Debug Configurations”, so there.
• Configure “Debug Configurations” via JSON, like “Tasks”.
• VSCode has very basic breakpoints without support for features such as enabling/disabling based on previous breakpoints, or disabling once hit.
• Debugging in Karma is a poor experience. I couldn’t find a Karma-specific extension to help with this.

Code Completion, Inspections, & Intentions (Oh My)

Assuming you use ESLint, install the ESLint extension.

In terms of inspections, I just want ESLint to run on my JavaScript. I don’t need any other inspections, so I disable all the random crap WebStorm ships with.

The ESLint extension “just works,” and the user sees the same type of inline inspections as from ESLint within WebStorm, right down to the intentions, like “Fix file with ESLint”.

The Node.js Extension Pack provides some npm-related "IntelliSense," which knows important stuff about package.json, like if a package is missing or extraneous, will tell you so, and automatically fix it for you via an intention.

Is “IntelliSense” a trademark or something? It’s a Microsoft-ism, right? I am pretty sure I hate this word.

I believe VSCode uses TypeScript definitions under the hood (not just in TypeScript files) to inspect code, at least in part. This results in extremely accurate and nearly instantaneous code completion, jump-to-declaration, and the like. If there’s any “killer feature” for JavaScript developers, it’s this. You can coax such behavior out of WebStorm, but it will still be much more sluggish.

A “deep dive” into this would be an excellent source of information for WebStorm users (think: CSS, HTML, TypeScript, template languages, etc.), but is unfortunately out-of-scope. I’ll end with my final thoughts below.

Summary

• VSCode (and/or its extensions) provide accurate, zippy code-completion and inline docs for JavaScript.
• VSCode does not ship with a bunch of built-in inspections. If you use them, you’ll miss them, unless you find alternatives.
• The ESLint experience is strong.
• The npm-related experience is excellent.

VSCode: The Verdict

Photo by chuttersnap / Unsplash

Visual Studio Code is better than I expected. It’s comparable with WebStorm—by way of extensions—in terms of feature set. It has a couple notable advantages over WebStorm, however:

• VSCode is generally faster and more responsive than WebStorm
• VSCode’s JavaScript inspection/completion experience is just plain better
• Free as in beer
• Much exciting!

Notable disadvantages include:

• Poor browser-based debugging experience (for me, anyway)
• Piecemeal (though complete) Git support
• JSON-based configuration
• No customer support

As a developer, my code editor my most important tool. Having spent many years with JetBrains and WebStorm—and having very little dissatisfaction—a different tool must be incredibly compelling for me to want to pick up.

My advice for WebStorm users? Don’t try VSCode unless you are prepared to switch.

In the first part of this excruciating tutorial, I taught the reader how to begin with MicroPython on an ESP32-based development board. We:

1. Flashed the board
2. Frolicked in the REPL
3. Configured WiFi
4. Uploaded scripts
5. Build a circuit with a DS18B20 1-Wire temperature sensor
6. Used MicroPython to read the temperature

In this part of the tutorial, we’ll take the data we gather with the sensor and publish it over MQTT.

If you’re unfamiliar with the concept, I’ll try to explain MQTT in a nutshell.

MQTT in a Nutshell

MQTT is a machine-to-machine protocol for publishing and subscribing to messages. Importantly, MQTT imposes no constraints upon the content nor structure of those messages.

In a typical setup, you have a single MQTT broker and one-or-many MQTT clients. A client may publish messages, subscribe to messages, or both. A client needn’t be an IoT device, a web app, a desktop or mobile app, a microservice, or anything in particular, as long as it speaks MQTT.

All clients connect to the broker. The broker is responsible for receiving published messages and (possibly) delivering them to interested clients.

Each message has a “topic”. As is vital to the publish/subscribe pattern, a message’s publisher doesn’t necessarily care if anyone is listening. Interested clients will subscribe to this topic.

A MQTT Example

You have an MQTT client—perhaps a device with a temperature sensor—called bob which wants to publish temperature data. It may publish on a topic such as bob/sensor/temperature, and the message would be the data, e.g., 68.75.

Another MQTT client, ray, may want to listen for temperature data so we can display it as a time series graph on a dashboard; ray would tell the broker it wishes to subscribe to the bob/sensor/temperature topic. Finally, when bob publishes on this topic, the broker notifies ray, and ray receives the message. ray can then do whatever it needs with its data.

Wildcards

Subscriptions support wildcards. If client bob had another sensor which reports the relative humidity, it may publish this data under the topic bob/sensor/humidity. Client ray could use a single-level wildcard such as bob/sensor/+, which would receive messages published on bob/sensor/humidity and bob/sensor/temperature. Or perhaps a multi-level wildcard such as bob/#, which would subscribe to any topic beginning with bob/.

A “topic per client” is merely a convention for sake of our example. MQTT enforces no such constraint.

There’s certainly more to it than just the above—but that’s the nutshell, and I’m calling it good.

Photo by Caleb Martin / Unsplash

Why MQTT?

It’s just as important to understand why you’d want to use a technology over another (or none at all).

MQTT's designers had resource-constrained devices (such as sensors) in mind; it’s a “thin” protocol, and easier to implement compared to, say, HTTP. As such, you’ll find that MQTT is a core technology behind many cloud-based “IoT platforms”, including the offerings of IBM, Amazon, Microsoft, Adafruit, and many others.

You can directly access many of these services via RESTful APIs, but it will necessarily consume more of your devices’ resources to do so.

Using HTTP(S) instead of MQTT makes sense if you need to make a remote procedure call, or if a request/response model is more natural than MQTT's publish/subscribe model in your problem domain. Even then, protocols such as CoAP will demand fewer resources.

Now that we understand what MQTT is all (or more accurately, “partly”) about, let’s use it to spread the word about our ambient temperatures.

Boot Script and Temperature Module

We’ll use the code from the last tutorial to begin with. For reference, I’ll show them below.

You should have two (2) files, the first being our startup script, boot.py:

def connect():
    import network
    sta_if = network.WLAN(network.STA_IF)
    if not sta_if.isconnected():
        print('connecting to network...')
        sta_if.active(True)
        sta_if.connect('<YOUR SSID>', '<YOUR PASSWORD>')
        while not sta_if.isconnected():
            pass
    print('network config:', sta_if.ifconfig())

def no_debug():
    import esp
    # you can run this from the REPL as well
    esp.osdebug(None)

no_debug()
connect()

And the second is temperature.py, an abstraction around the temperature sensor:

import time
from machine import Pin
from onewire import OneWire
from ds18x20 import DS18X20


class TemperatureSensor:
    """
    Represents a Temperature sensor
    """
    def __init__(self, pin):
        """
        Finds address of single DS18B20 on bus specified by `pin`
        :param pin: 1-Wire bus pin
        :type pin: int
        """
        self.ds = DS18X20(OneWire(Pin(pin)))
        addrs = self.ds.scan()
        if not addrs:
            raise Exception('no DS18B20 found at bus on pin %d' % pin)
        # save what should be the only address found
        self.addr = addrs.pop()

    def read_temp(self, fahrenheit=True):
        """
        Reads temperature from a single DS18X20
        :param fahrenheit: Whether or not to return value in Fahrenheit
        :type fahrenheit: bool
        :return: Temperature
        :rtype: float
        """

        self.ds.convert_temp()
        time.sleep_ms(750)
        temp = self.ds.read_temp(self.addr)
        if fahrenheit:
            return self.c_to_f(temp)
        return temp

    @staticmethod
    def c_to_f(c):
        """
        Converts Celsius to Fahrenheit
        :param c: Temperature in Celsius
        :type c: float
        :return: Temperature in Fahrenheit
        :rtype: float
        """
        return (c * 1.8) + 32

Upload both of these files via ampy:

$ ampy --port /dev/tty.SLAB_USBtoUART put boot.py && \
  ampy --port /dev/tty.SLAB_USBtoUART put temperature.py

(Replace /dev/tty.SLAB_USBtoUART with your device path or COM port.)

In the first part of this tutorial, I told you to download (or clone) the micropython-lib project. This is not necessary! Read on.

Install the MQTT Modules via upip

Since your device should be online, we can use upip from the REPL. upip is a stripped-down package manager for MicroPython. It's built-in to the ESP32 port of MicroPython; you already have it. It downloads packages from PyPi, just like pip.

Open your REPL, and execute:

import upip
upip.install('micropython-umqtt.robust')

Sample output:

Installing to: /lib/
Warning: pypi.python.org SSL certificate is not validated
Installing micropython-umqtt.robust 1.0 from https://pypi.python.org/packages/31/02/7268a19a5054cff8ff4cbbb126f00f098848dbe8f402caf083295a3a6a11/micropython-umqtt.robust-1.0.tar.gz

Take note: if your device isn’t online, upip won’t work from the device’s REPL.

You also need to grab its dependency, micropython-umqtt.simple:

upip.install('micropython-umqtt.robust')

Sample output:

Installing to: /lib/
Installing micropython-umqtt.simple 1.3.4 from https://pypi.python.org/packages/bd/cf/697e3418b2f44222b3e848078b1e33ee76aedca9b6c2430ca1b1aec1ce1d/micropython-umqtt.simple-1.3.4.tar.gz

umqtt.simple is a barebones MQTT client. umqtt.robust depends on umqtt.simple; it’s an MQTT client which will automatically reconnect to the broker if a disconnection occurs.

To verify that this installed properly, you can execute from your REPL:

from umqtt.robust import MQTTClient

No errors? You’re good.

Get a MQTT Client App

Before we begin the next section, you might want another application handy—a standalone MQTT client. You could try:

• MQTT.fx (GUI; Windows/Mac)
• MQTTBox (GUI; Windows/Mac/Linux)
• mosquitto-clients from Mosquitto is available via package manager (CLI; Linux/Mac)
• Various free clients on app stores (iOS/Android)
• Node-RED can also connect to an MQTT broker (Web; Windows/Mac/Linux)

Using one isn’t strictly necessary, but will aid experimentation.

Experimenting with umqtt in the REPL

If you’ve been reading closely, you’ll understand that we need an MQTT broker (“server”); a MQTT client with no broker is useless.

It just so happens that public MQTT brokers exist; test.mosquitto.org by the Mosquitto project is one such broker. As a member of the public, you can use it! Just be aware: any data or information you publish on a public MQTT broker is also public. Don’t publish anything you wouldn’t want everyone to know about.

We’ll use this public broker for the purposes of the tutorial, but if you have a different one you wish to use, you go ahead and do that.

Now, let’s try to use our MQTT lib to publish a message on the broker.

Create a Unique “Client ID”

One caveat to note about MQTT: each MQTT client connected to a broker must have a unique identifier: a client ID. You’ll need to pick a phrase or generate something. I’ll just generate one on the command line:

$ python3 -c 'from uuid import uuid4; print(uuid4())'
52dc166c-2de7-43c1-88ff-f80211c7a8f6

Copy the resulting value to your clipboard; you’ll need it in a minute.

Connect to the REPL

Open up a serial connection to your ESP32. I’m going to use miniterm here, which Python 3 bundles:

$ python3 -m serial.tools.miniterm --raw /dev/tty.SLAB_USBtoUART 115200

The --raw flag avoids problems with special characters such as BS and DEL.

Connect to the Broker

As in the first tutorial, I’ll omit the prompt (>>>) when working with the REPL.

We should now be able to import MQTTClient:

from umqtt.simple import MQTTClient

The MQTTClient constructor accepts a client ID and a DNS or IP address of a MQTT broker. We’ll use our pseudorandom client ID from above, and test.mosquitto.org for the server, then call connect():

client = MQTTClient('52dc166c-2de7-43c1-88ff-f80211c7a8f6', 
		'test.mosquitto.org')
client.connect()

The output of this command, if all went well, should be 0; connect() will raise an exception it the connection failed.

Connect a Second Client

At this point, I’m going to fire up MQTT.fx; I’ll use it to subscribe to the messages which the ESP32 publishes.

I enter server test.mosquitto.org in the server input field, and leave the port field 1883, which is the default (insecure) MQTT port. I then click “Connect,” and wait for negotiation. Here’s a screenshot of my connected client:

MQTT.fx connected to test.mosquitto.org.

I’ll come back to MQTT.fx after we learn to publish from the REPL.

Publish an MQTT Message

Assuming the ESP32 is now connected to the broker, you can publish messages. First, I’ll emit a temperature in Fahrenheit, with the topic boneskull/test/temperature/fahrenheit:

client.publish('boneskull/test/temperature/fahrenheit', 72)

…but MicroPython complained:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "umqtt/simple.py", line 112, in publish
TypeError: object of type 'int' has no len()

What’s the problem here? Let me explain:

1. An MQTT message payload could be literally any data. MQTT has no notion of “data types”. It doesn’t know what a “number” or “integer” is. Your payload will always consist of raw bytes.
2. There’s no direct mapping of an integer to “bytes,” as there isn’t just one way to encode this number as binary data. We don’t know if this is a signed or unsigned integer, how many bits we should use, etc.
3. The problem could have been obvious (and we could have RTFM), but MicroPython shies away from overly “friendly” APIs due to resource constraints, so it’s not obvious what’s happening here.

The easiest solution? Publish a str instead:

client.publish('boneskull/test/temperature/fahrenheit', '72')

If this worked, there should be no output from the statement.

Hooray? I’m not convinced—are you? This just squirted the temperature into the ether! We should see where these messages are going. I can do that in my MQTT.fx client by subscribing to the topic. This is how:

Subscribing to a topic in MQTT.fx

1. Click on the “Subscribe” tab
2. Enter boneskull/test/temperature/fahrenheit in the input field
3. Click “Subscribe” button to the right of input field

After you’ve done this, MQTT.fx will contact the broker, and if successful, you will see the subscription appear beneath the input field:

An active subscription in MQTT.fx

Next time we (or any client attached to the broker) publishes on this topic, we will see it in the lower-right area of this window, where it is grey and empty.

Return to your serial terminal, and run the last command again (you can just hit “up-arrow” then “enter”):

client.publish('boneskull/test/temperature/fahrenheit', '72')

Switch back to MQTT.fx. It may take a few seconds depending on how busy the broker is, but the message should now appear to the right, along with its payload:

A received message in MQTT.fx

Excellent work!

Now we can use everything we’ve learned, and periodically publish real temperature data. Let’s cook up a little module to do that.

A Module to Publish Temperature

I’ve written up a little module which uses MQTTClient and TemperatureSensor (from our first tutorial) to publish temperature data. Create temperature_client.py:

import time

from umqtt.robust import MQTTClient

from temperature import TemperatureSensor


class TemperatureClient:
    """
    Represents an MQTT client which publishes temperature data on an interval
    """

    def __init__(self, client_id, server, pin, fahrenheit=True, topic=None,
                 **kwargs):
        """
        Instantiates a TemperatureSensor and MQTTClient; connects to the
        MQTT broker.
        Arguments `server` and `client_id` are required.

        :param client_id: Unique MQTT client ID
        :type client_id: str
        :param server: MQTT broker domain name / IP
        :type server: str
        :param pin: 1-Wire bus pin
        :type pin: int
        :param fahrenheit: Whether or not to publish temperature in Fahrenheit
        :type fahrenheit: bool
        :param topic: Topic to publish temperature on
        :type topic: str
        :param kwargs: Arguments for MQTTClient constructor
        """
        self.sensor = TemperatureSensor(pin)
        self.client = MQTTClient(client_id, server, **kwargs)
        if not topic:
            self.topic = 'devices/%s/temperature/degrees' % \
                         self.client.client_id
        else:
            self.topic = topic
        self.fahrenheit = bool(fahrenheit)

        self.client.connect()

    def publishTemperature(self):
        """
        Reads the current temperature and publishes it on the configured topic.
        """
        t = self.sensor.read_temp(self.fahrenheit)
        self.client.publish(self.topic, str(t))

    def start(self, interval=60):
        """
        Begins to publish temperature data on an interval (in seconds).
        This function will not exit! Consider using deep sleep instead.
        :param interval: How often to publish temperature data (60s default)
        :type interval: int
        """
        while True:
            self.publishTemperature()
            time.sleep(interval)

Upload this to your board:

$ ampy --port /dev/tty.SLAB_USBtoUART put temperature_client.py

Your standalone MQTT client app should still be online. Let’s send a message in the REPL, then view the result in the standalone client (please create your own client ID below):

from temperature_client import TemperatureClient
tc = TemperatureClient('boneskull-test-1516667340',
                       'test.mosquitto.org', 12, 
                       topic='boneskull/test/temperature')
tc.start(10) # publish temperature every 10s

A word of warning: once you execute the above, the REPL will “hang,” since the start() method is just busy-waiting.

Even though this is a busy-wait, time.sleep() does not mean that "nothing happens"; the tick rate in the underlying operating system is 10ms; any sleep time (necessarily using time.sleep_ms() or time.sleep_us()) equal to or less than 10ms will preempt other tasks!

Tab back to MQTT.fx:

Real temperature data in MQTT.fx!

This will loop indefinitely, so when ready, push the “reset” button on your dev board to get back to the REPL (you don’t need to quit your serial terminal beforehand).

Important to note: the “time and date” you see in the payload detail does not mean “when the originating client sent the message.” Rather, it means “when the receiving client received the message.” MQTT messages do not contain a “sent on” timestamp unless you add one yourself!

(To do this, you'd need to ask an NTP server or an external RTC module, which is beyond our scope.)

We’re successfully published a number! That is great news, except, that number could refer to anything. It’d be helpful to include the unit—either Fahrenheit or Celsius—in the payload. I’ll show you how.

Working with JSON

As I’ve beaten to death, MQTT payloads contain anything. That means if you want to send some structured data, you are responsible for serialization and deserialization.

JSON is a common data interchange format for which MicroPython contains built-in support (unlike, say, that vile Arduino API). It’s trivial to “stringify” a dict and publish the result.

To work with JSON—just like in Real Python—we will need to import another module in temperature_client.py:

import json

Then, add the data to the payload within the publishTemperature method:

    def publishTemperature(self):
        """
        Reads the current temperature and publishes a JSON payload on the
        configured topic, e.g., `{"unit": "F", "degrees": 72.5}`
        """
        t = self.sensor.read_temp(self.fahrenheit)
        payload = dict(degrees=t)
        if self.fahrenheit:
            payload['unit'] = 'F'
        else:
            payload['unit'] = 'C'
        self.client.publish(self.topic, json.dumps(payload))

Notice that we didn’t need to coerce the temperature (“degrees”) into a str for purposes of publishing, because JSON is a str itself—the recipient of this payload will decode the JSON into a numeric value.

Disconnect from the REPL (that’s Ctrl-] if you happen to be using miniterm), and upload temperate_client.py to the ESP32 again, then reconnect to the REPL. We don’t need to begin an infinite loop to test it, since we can just call publishTemperature() directly:

from temperature_client import TemperatureClient
tc = TemperatureClient('boneskull-test-1516667340',
                       'test.mosquitto.org', 12, 
                       topic='boneskull/test/temperature')
tc.publishTemperature()

The above will send a single message. On the receiving end:

Pretty-printed JSON in MQTT.fx

If you resize your MQTT.fx window to be tall enough, you’ll see the “Payload decoded by” dropdown in the lower-right. You can see the pretty-printed payload appears as we ‘spected.

MQTT.fx also includes Base64 and hex decoders, but the default is “plain text”.

I think you have the basics down. But maybe you aren’t going to run your own private MQTT broker. Let’s take this one step further and interface with an IoT platform.

Use an ESP32 with MicroPython on IBM Cloud

Watson IoT Platform is a service in IBM Cloud (formerly Bluemix). I’ve written a MicroPython module to interface with it, and we’ll use that to save some time.

Watson IoT Platform Quickstart

You can experiment with this platform without needing to sign up for an account.

1. Visit the Quickstart page:
   Watson IoT Platform's Quickstart Page
2. Tick “I Accept” after carefully reading the entire terms of use.
3. Enter a unique device identifier in the input box. I’m calling mine “boneskull-esp32-test”. Click “Go”.

Keep this browser window open; you’re now ready to send data, and see the result in real-time. Let’s get to it.

Upload the micropython-watson-iot module

micropython-watson-iot is the module I referenced earlier. Its README contains installation instructions using upip, but essentially it’s the same as before, via the REPL:

import upip
upip.install('micropython-watson-iot')

To verify installation, run:

from watson_iot import Device

Assuming that didn’t throw an exception, we can use it like so:

d = Device(device_id='boneskull-esp32-test')
d.connect()
d.publishEvent('temperature', {'degrees': 68.5, 'unit': 'F'})

You should see it reflected in your browser. In fact, if you do something like this…

import time
d.publishEvent('temperature', {'degrees': 68.5, 'unit': 'F'})
time.sleep(5)
d.publishEvent('temperature', {'degrees': 69.5, 'unit': 'F'})
time.sleep(5)
d.publishEvent('temperature', {'degrees': 67.5, 'unit': 'F'})
time.sleep(5)
d.publishEvent('temperature', {'degrees': 66.5, 'unit': 'F'})

…you should see a nifty line graph:

Real-time graph of our temperature data

You’re welcome to play with this in more depth; Watson IoT Platform has a free tier. To sign up, you need to:

1. Register with IBM Cloud (no credit card needed)
2. Create a Watson IoT Platform service instance using the “free plan” from the catalog
3. Click “Launch” to explore the platform.
4. Also, check out the docs.

The micropython-watson-iot library offers a few “quality of life” benefits—as IoT platforms typically do—when compared to a vanilla MQTT client and/or broker:

1. Messages contain metadata such as “published on” time, handled by the cloud platform
2. You can group devices via logical “device types”
3. Structured data can be automatically encoded/decoded to/from JSON (it does this by default)
4. Create your own custom encoders and decoders (e.g., numeric, Base64)
5. Create custom “command handlers,” which cause the device to react upon reception of a “command”-style MQTT message. For example, you could send a command to blink an onboard LED or reboot the device.

I’ve committed a few micropython-watson-iot examples; you can use adapt these patterns to your own code.

There’s really a lot more going on here than just MQTT—dashboards and gateways and all sorts of hoodoo that I am not going to go into. But now it’s easy to use with MicroPython on an ESP32, thanks to ME.

Ahem…

Recap, Obligatory Link Dump, & Goodbyes

In this tutorial, we’ve learned:

1. What MQTT is (and what it’s for)
2. How to talk to an MQTT broker using MicroPython and an ESP32
3. How to publish structured data
4. Install MicroPython libraries from PyPi via upip
5. How to subscribe to simple topics via a standalone MQTT client
6. How to publish data to Watson IoT Platform via its Quickstart site, using micropython-watson-iot

Check out the README of micropython-watson-iot for more info on usage and discussion of its limitations.

I’ve posted the complete example files in this Gist for your convenience.

Thanks for reading! Extra thanks for doing, too.

I’m going to show you how to turn on your funk motor get started with MicroPython on an Espressif ESP32 development board. In this first part of this tutorial, I’ll show you how to:

• Get up & running with MicroPython on the ESP32
• Connect to WiFi
• Upload scripts to the board
• Read the ambient temperature (everyone loves that, right?)

In the forthcoming second part of this tutorial, I’ll show you publish the data you’ve collected with MQTT.

This guide expects you to possess:

• …familiarity with the command-line
• …basic experience interfacing with development boards (like Arduino)
• …a basic understanding of programming in Python

If I’ve glossed over something I shouldn’t have, please let me know!

Before we begin, you will need some stuff.

Bill of Stuff

You need Stuff in the following categories.

Hardware

Not necessarily this stuff, but same idea. Photo by Alexandra Cárdenas

• One (1) ESP32 development board such as the SparkFun ESP32 Thing (any kind will do; they are all roughly the same)
• One (1) DS18B20 digital thermometer (datasheet) in its TO-92 package
• One (1) 4.7kꭥ resistor
• Four (4) jumper wires
• One (1) 400-point or larger breadboard
• One (1) USB Micro-B cable

If you need to solder header pins on your dev board: do so.

If you have a DS18B20 “breakout board”: these typically have the resistor built-in, so you won’t need it. You will need to figure out which pin is which, however.

Software

You will need to download and install some software. Some of these things you may have installed already. Other things may need to be upgraded. This guide assumes you ain’t got jack squat.

I apologize that I don't have much information for Windows users! However, I assure you that none of this is impossible.

VCP Driver

If you're running macOS or Windows, you may need to download and install a Virtual COM Port (VCP) driver, if you haven't done so already. Typically, the USB-to-serial chip on these boards is a CP210x or FT232RL; check the datasheet for your specific board or just squint at the IC near the USB port.

Newer Linux kernels have support for these chips baked-in, so driver installation is unnecessary.

Here's an example of a CP2104 on an ESP32 dev board of mine:

A SiLabs CP2104. Thanks, macro lens!

To assert the driver is working, plug your dev board into your computer. If you’re on Linux, check for /dev/ttyUSB0:

$ ls -l /dev/ttyUSB0
crw-rw---- 1 root dialout 188, 0 Dec 19 17:04 /dev/ttyUSB0

Or /dev/tty.SLAB_USBtoUART on macOS:

$ ls -l /dev/tty.SLAB_USBtoUART
crw-rw-rw-  1 root  wheel   21,  20 Dec 19 17:10 /dev/tty.SLAB_USBtoUART

Serial Terminal

A free, cross-platform, GUI terminal is CoolTerm. Linux & macOS users can get away with using screen on the command line. More purpose-built solutions include miniterm, which ships with Python 3, and can be launched via python3 -m serial.tools.miniterm, and minicom.

Python, Etc.

You will also need:

• Python v3.6.x
• For extra libraries, a clone or archive of micropython/micropython-lib (git clone https://github.com/micropython/micropython-lib)

How you install these will vary per your installation of Python:

• To flash the board, esptool (version 2.2 or newer)
• To manage files on the board, adafruit-ampy

You could try pip3 install esptool adafruit-ampy. This worked for me on macOS with Homebrew; YMMV. You might need to preface that with sudo if not using Homebrew.

MicroPython Firmware

Finally, you’ll need to download the latest MicroPython firmware for ESP32.

Now that our tools are at hand, we can begin by flashing the ESP32 board with MicroPython.

Flashing MicroPython & First Steps

Unless MicroPython is already installed on your ESP32, you will want to start by connecting it to your computer via USB, and erasing its flash:

In the below examples, replace /dev/tty.SLAB_USBtoUART with the appropriate device or COM port for your system.

$ esptool.py --chip esp32 -p /dev/tty.SLAB_USBtoUART erase_flash
esptool.py v2.2
Connecting........___
Chip is ESP32D0WDQ6 (revision 1)
Uploading stub...
Running stub...
Stub running...
Erasing flash (this may take a while)...
Chip erase completed successfully in 4.6s
Hard resetting...

Now, we can flash it with the firmware we downloaded earlier:

$ esptool.py --chip esp32 -p /dev/tty.SLAB_USBtoUART write_flash \
  -z 0x1000 ~/Downloads/esp32-20171219-v1.9.2-445-g84035f0f.bin
esptool.py v2.2
Connecting........_
Chip is ESP32D0WDQ6 (revision 1)
Uploading stub...
Running stub...
Stub running...
Configuring flash size...
Auto-detected Flash size: 4MB
Compressed 936288 bytes to 587495...
Wrote 936288 bytes (587495 compressed) at 0x00001000 in 51.7 seconds (effective 144.8 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting...

If you’re feeling dangerous, you can increase the baud rate when flashing by using the --baud option.

If that worked, you should be able to enter a MicroPython REPL by opening up the port:

# 115200 is the baud rate at which the REPL communicates
$ screen /dev/tty.SLAB_USBtoUART 115200

>>> 

Congratulations, >>> is your REPL prompt. This works similarly to a normal Python REPL (e.g. just running python3 with no arguments). Try the help() function:

>>> help()
Welcome to MicroPython on the ESP32!

For generic online docs please visit http://docs.micropython.org/

For access to the hardware use the 'machine' module:

import machine
pin12 = machine.Pin(12, machine.Pin.OUT)
pin12.value(1)
pin13 = machine.Pin(13, machine.Pin.IN, machine.Pin.PULL_UP)
print(pin13.value())
i2c = machine.I2C(scl=machine.Pin(21), sda=machine.Pin(22))
i2c.scan()
i2c.writeto(addr, b'1234')
i2c.readfrom(addr, 4)

Basic WiFi configuration:

import network
sta_if = network.WLAN(network.STA_IF); sta_if.active(True)
sta_if.scan()                             # Scan for available access points
sta_if.connect("<AP_name>", "<password>") # Connect to an AP
sta_if.isconnected()                      # Check for successful connection

Control commands:
  CTRL-A        -- on a blank line, enter raw REPL mode
  CTRL-B        -- on a blank line, enter normal REPL mode
  CTRL-C        -- interrupt a running program
  CTRL-D        -- on a blank line, do a soft reset of the board
  CTRL-E        -- on a blank line, enter paste mode

For further help on a specific object, type help(obj)
For a list of available modules, type help('modules')

If you’ve never seen this before on an MCU: I know, crazy, right?

You can type in the commands from “Basic WiFi configuration” to connect. You will see a good deal of debugging information from the ESP32 (this can be suppressed, as you’ll see):

>>> import network
>>> sta_if = network.WLAN(network.STA_IF)
I (323563) wifi: wifi firmware version: 111e74d
I (323563) wifi: config NVS flash: enabled
I (323563) wifi: config nano formating: disabled
I (323563) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (323573) system_api: Base MAC address is not set, read default base MAC address from BLK0 of EFUSE
I (323593) wifi: Init dynamic tx buffer num: 32
I (323593) wifi: Init data frame dynamic rx buffer num: 64
I (323593) wifi: Init management frame dynamic rx buffer num: 64
I (323603) wifi: wifi driver task: 3ffe1584, prio:23, stack:4096
I (323603) wifi: Init static rx buffer num: 10
I (323613) wifi: Init dynamic rx buffer num: 0
I (323613) wifi: Init rx ampdu len mblock:7
I (323623) wifi: Init lldesc rx ampdu entry mblock:4
I (323623) wifi: wifi power manager task: 0x3ffe84b0 prio: 21 stack: 2560
W (323633) phy_init: failed to load RF calibration data (0x1102), falling back to full calibration
I (323793) phy: phy_version: 362.0, 61e8d92, Sep  8 2017, 18:48:11, 0, 2
I (323803) wifi: mode : null
>>> sta_if.active(True)
I (328553) wifi: mode : sta (30:ae:a4:27:d4:88)
I (328553) wifi: STA_START
True
>>> sta_if.scan()
I (389423) network: event 1
[(b'SON OF ZOLTAR', b"`\xe3'\xcf\xf4\xf5", 1, -57, 4, False), (b'CenturyLink6105', b'`1\x97%\xd9t', 1, -96, 4, False)]
>>> sta_if.connect('SON OF ZOLTAR', '<REDACTED>')
>>> I (689573) wifi: n:1 0, o:1 0, ap:255 255, sta:1 0, prof:1
I (690133) wifi: state: init -> auth (b0)
I (690133) wifi: state: auth -> assoc (0)
I (690143) wifi: state: assoc -> run (10)
I (690163) wifi: connected with SON OF ZOLTAR, channel 1
I (690173) network: event 4
I (691723) event: sta ip: 10.0.0.26, mask: 255.255.255.0, gw: 10.0.0.1
I (691723) network: GOT_IP
I (693143) wifi: pm start, type:0

>>> sta_if.isconnected()
True

Cool, huh?

Now that we know we can connect to WiFi, let’s have the board connect every time it powers up.

Creating a MicroPython Module

To perform tasks upon boot, MicroPython wants you to put code in a file named boot.py, which is a MicroPython module.

Let’s create boot.py with code modified from the MicroPython ESP8266 docs, replacing where indicated:

def connect():
    import network
    sta_if = network.WLAN(network.STA_IF)
    if not sta_if.isconnected():
        print('connecting to network...')
        sta_if.active(True)
        sta_if.connect('<YOUR WIFI SSID>', '<YOUR WIFI PASS>')
        while not sta_if.isconnected():
            pass
    print('network config:', sta_if.ifconfig())

We can also create a function to disable debugging output. Append to boot.py:

def no_debug():
    import esp
    # this can be run from the REPL as well
    esp.osdebug(None)

These functions will be defined at boot, but not called automatically. Let’s test them before making them automatically execute.

To do this, we can upload boot.py. You’ll need to close the connection to the serial port. If you’re using screen, type Ctrl-A Ctrl-\, then y to confirm; otherwise disconnect or just quit your terminal program.

Uploading a MicroPython Module

Though there are other ways to do this, I’ve found the most straightforward for the ESP32 is to use ampy, a general-purpose tool by Adafruit. Here’s what it can do:

$ ampy --help

Usage: ampy [OPTIONS] COMMAND [ARGS]...

  ampy - Adafruit MicroPython Tool

  Ampy is a tool to control MicroPython boards over a serial
  connection.  Using ampy you can manipulate files on the board's
  internal filesystem and even run scripts.

Options:
  -p, --port PORT  Name of serial port for connected board.  Can
                   optionally specify with AMPY_PORT environemnt
                   variable.  [required]
  -b, --baud BAUD  Baud rate for the serial connection (default
                   115200).  Can optionally specify with AMPY_BAUD
                   environment variable.
  --version        Show the version and exit.
  --help           Show this message and exit.

Commands:
  get    Retrieve a file from the board.
  ls     List contents of a directory on the board.
  mkdir  Create a directory on the board.
  put    Put a file or folder and its contents on the...
  reset  Perform soft reset/reboot of the board.
  rm     Remove a file from the board.
  rmdir  Forcefully remove a folder and all its...
  run    Run a script and print its output.

MicroPython stores files (scripts or anything else that fits) in a very basic filesystem. By default, an empty boot.py should exist already. To list the files on your board, execute:

$ ampy -p /dev/tty.SLAB_USBtoUART ls
boot.py

Using the get command will echo a file’s contents to your shell (which could be piped to a file, if you wish):

$ ampy -p /dev/tty.SLAB_USBtoUART get boot.py
# This file is executed on every boot (including wake-boot from deepsleep)

We can overwrite it with our own boot.py:

$ ampy -p /dev/tty.SLAB_USBtoUART put boot.py

And retrieve it to see that it overwrote the default boot.py:

$ ampy -p /dev/tty.SLAB_USBtoUART get boot.py
def connect():
    import network
    sta_if = network.WLAN(network.STA_IF)
    if not sta_if.isconnected():
        print('connecting to network...')
        sta_if.active(True)
        sta_if.connect('<YOUR WIFI SSID>', '<YOUR WIFI PASS>')
        while not sta_if.isconnected():
            pass
    print('network config:', sta_if.ifconfig())

def no_debug():
    import esp
    # this can be run from the REPL as well
    esp.osdebug(None)

Success! This is the gist of uploading files with ampy. You can also upload entire folders, as we’ll see later.

From here, we can open our REPL again, and run our code. No need to restart the board!

Running a MicroPython Module

In following examples, I will eliminate the command prompt (>>>) from code run in a REPL, for ease of copying & pasting.

Re-connect to the REPL.

$ screen /dev/tty.SLAB_USBtoUART 115200

First, we’ll disconnect from WiFi:

import network
sta_if = network.WLAN(network.STA_IF)
sta_if.disconnect()

Debug output follows:

I (3299583) wifi: state: run -> init (0)
I (3299583) wifi: n:1 0, o:1 0, ap:255 255, sta:1 0, prof:1
I (3299583) wifi: pm stop, total sleep time: 0/-1688526567
I (3299583) wifi: STA_DISCONNECTED, reason:8

Then, we can import the boot module. This will make our connect and no_debug functions available.

import boot
connect()

Output:

connecting to network...
I (87841) wifi: n:1 0, o:1 0, ap:255 255, sta:1 0, prof:1
I (88401) wifi: state: init -> auth (b0)
I (88401) wifi: state: auth -> assoc (0)
I (88411) wifi: state: assoc -> run (10)
I (88441) wifi: connected with SON OF ZOLTAR, channel 1
I (88441) network: event 4
I (90081) event: sta ip: 10.0.0.26, mask: 255.255.255.0, gw: 10.0.0.1
I (90081) network: GOT_IP
network config: ('10.0.0.26', '255.255.255.0', '10.0.0.1', '10.0.0.1')
I (91411) wifi: pm start, type:0

Super. Let’s silence the noise, and try again:

no_debug()
sta_if.disconnect()
connect()

Output:

connecting to network...
network config: ('10.0.0.26', '255.255.255.0', '10.0.0.1', '10.0.0.1')

LGTM.

The IP addresses above depend upon your local network configuration, and will likely be different.

Disconnect from the port (if using screen: Ctrl-A Ctrl-\, y) and append these lines to boot.py:

no_debug()
connect()

Upload it again via ampy put boot.py, which will overwrite the existing boot.py. Hard reset (“push the button”) or otherwise power -cycle the board. Reconnect to the REPL and execute connect() to assert connectivity:

connect()

Output:

network config: ('10.0.0.26', '255.255.255.0', '10.0.0.1', '10.0.0.1')

You’ll notice “connecting to network...” was not printed to the console; if already connected, the connect() function prints the configuration and returns. If you’ve gotten this far, then your board is successfully connecting to Wifi at boot. Good job!

We now have two more items to check off our list, unless you forgot what we were trying to do:

1. We need to read the ambient temperature on an interval.
2. We need to publish this information to an MQTT broker.

Next, we’ll knock out that temperature reading.

Temperature Readings in MicroPython

As we write our code, we can use the REPL to experiment.

I’m using the example found here. You’ll need to import three (3) modules, machine, onewire and ds18x20 (note the x):

import machine, onewire, ds18x20

I’ve connected my sensor to pin 12 on my ESP32. Your breadboard should look something like this:

Example breadboard wiring for ESP32 dev board and DS18B20

To read temperature, we will create a Matryoshka-doll-like object by passing a Pin instance into a OneWire constructor (read about 1-Wire) and finally into a DS18X20 constructor:

pin = machine.Pin(12)
wire = onewire.OneWire(pin)
ds = ds18x20.DS18X20(wire)

Note that if the output of the following command is an empty list ([]), the sensor couldn't be found. Check your wiring!

Now, we can ask ds to scan for connected devices, and return their addresses:

ds.scan()

Output:

[bytearray(b'(\xee3\x0c"\x15\x004')]

ds.scan() returns a list of device addresses in bytearray format. Yours may look slightly different. Since we only have one, we can save its address to a variable. To read temperature data, we tell the 1-Wire bus to reset via ds.convert_temp(), take a short pause of 750ms (in case you're pasting this):

import time
addr = ds.scan().pop()
ds.convert_temp()
time.sleep_ms(750)
temp = ds.read_temp(addr)
temp

Output:

19.875

This reading is in Celsius. If you’re like me, you don’t speak Celsius, so maybe you want to convert it to Fahrenheit:

(temp * 1.8) + 32

Output:

67.775

…which is right around what I expected!

Let’s take what we’ve done and create a new file, temperature.py:

import time
from machine import Pin
from onewire import OneWire
from ds18x20 import DS18X20


class TemperatureSensor:
    """
    Represents a Temperature sensor
    """
    def __init__(self, pin):
        """
        Finds address of single DS18B20 on bus specified by `pin`
        :param pin: 1-Wire bus pin
        :type pin: int
        """
        self.ds = DS18X20(OneWire(Pin(pin)))
        addrs = self.ds.scan()
        if not addrs:
            raise Exception('no DS18B20 found at bus on pin %d' % pin)
        # save what should be the only address found
        self.addr = addrs.pop()

    def read_temp(self, fahrenheit=True):
        """
        Reads temperature from a single DS18X20
        :param fahrenheit: Whether or not to return value in Fahrenheit
        :type fahrenheit: bool
        :return: Temperature
        :rtype: float
        """
        self.ds.convert_temp()
        time.sleep_ms(750)
        temp = self.ds.read_temp(self.addr)
        if fahrenheit:
            return self.c_to_f(temp)
        return temp

    @staticmethod
    def c_to_f(c):
        """
        Converts Celsius to Fahrenheit
        :param c: Temperature in Celsius
        :type c: float
        :return: Temperature in Fahrenheit
        :rtype: float
        """
        return (c * 1.8) + 32

Disconnect from the REPL. Upload temperature.py via ampy:

$ ampy -p /dev/tty.SLAB_USBtoUART put temperature.py

Then we can open our REPL once again, and try it:

from temperature import TemperatureSensor
t = TemperatureSensor(12)
t.read_temp() # use t.read_temp(False) to return Celsius

Seems to have warmed up a bit. Output:

68.7875

Good work!

Conclusion of Part One (1)

In the first part of this tutorial, we’ve learned how to:

1. Flash an ESP32 dev board with MicroPython
2. Use MicroPython’s REPL to experiment
3. Connect the ESP32 to WiFi
4. Upload and execute MicroPython scripts
5. Read the temperature with a 1-Wire DS18B20 sensor

In the forthcoming second part of this tutorial, we’ll learn about MQTT, how to publish our temperature data to an MQTT broker, and likewise interface with an MQTT-based cloud “IoT platform”.

In late 2013, I received a SparkFun Inventor’s Kit as a non-denominational holiday present from a coworker. Without hyperbole, this was the best present I’ve ever received (thanks Anthony).

Before this, I knew nothing of electronics, hardware, or firmware. The SparkFun RedBoard (an Arduino Uno-compatible device) and its guidebook exposed to me the vast world of physical computing. Fascinated with the possibilities, I soon found myself a dedicated hobbyist.

My introduction to hardware: The SparkFun RedBoard

As a “full stack” web developer by trade, my background is in higher-level languages—mainly JavaScript and Python. I was ripping out my luxuriant locks writing anything more than the most basic “sketch” in C/C++. Granted, the easy was easy. But it’s like putting lipstick on a pig. Just beneath the façade of digitalWrite and void loop is the seamy underbelly of C/C++, wallowing in its own filth. 🐷

Stepping out of my comfort zone into physical computing was more than enough of a challenge. I could certainly get by—since 2013, I’ve written more C/C++ than I had in the previous twelve years. And I realize a “close to the metal” language is often necessary for embedded systems. But it wasn’t fun.

What follows is the story of how this web developer started to have fun writing firmware. YMMV.

Programming an Arduino Using Johnny-Five & Node.js

In the name of “fun”, I sought out a higher-level language. I found NodeBots and Rick Waldron’s Johnny-Five project.

Opening a REPL w/ Johnny-Five and interacting with the hardware "live" was more than the Arduino IDE could offer (not saying much! zing!).

There's a catch, of course. J5 requires a serial connection or a board which can run Node.js natively. A tall order to be sure: this requires a supported OS and considerable memory resources. In other words, not an 8-bit MCU like the Uno’s ATmega328p.

A Linux-capable device satisfies requirements for many projects. For example, J5 is an excellent choice when working with GPIO on a Raspberry Pi or Tessel. It’s “just Node.js”—with excellent documentation and examples, right down to the Fritzing diagrams—and for anyone familiar with JavaScript, it’s trivial to learn (and not much more difficult to hack on).

Meanwhile, in the summer of ’14, while I was busy hacking on Johnny-Five, an unknown manufacturer out of Shanghai jumped into the pool with a tiny SoC—and it hit the water like a cannonball.

The ESP8266 Has Landed

This Hackaday post was the first introduction for many to the ESP-01 module, built with the ESP8266 MCU from Espressif. Its price point was—and still is—profoundly lower than any other WiFi-capable SoC.

Texas Instruments’ CC3000, introduced in 2013, was likely the lowest-cost module available at the time. It gained a reputation as an unreliable pest—and even today, you’re looking at nearly $40 USD for an obsolete eval board.

The ESP8266 threw the doors open for hobbyists to dabble in IoT and home automation. This thing is now everywhere. Better yet, the ESP8266 has since proven to be a highly reliable module.

Johnny-Five can drive an ESP8266 via WiFi (with some effort), but the device will still be tethered, and will fail if the connection is interrupted.

At first, it was a bit of a curiosity. In 2014, docs or educational resources in English were nearly nonexistent. We'd wire the ESP-01 module to an Arduino, and use it as a serial bridge over WiFi, using Hayes-like AT commands to control its behavior.

// example lifted from https://github.com/alokdhari/ESP8266

#include <SoftwareSerial.h>

SoftwareSerial mySerial(10, 11); // RX, TX

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    // busy wait!
  }


  Serial.println("Hello World");

  // set the data rate for the SoftwareSerial port
  mySerial.begin(9600);
  callWifiBoardWithCommand("AT", 1000);
  callWifiBoardWithCommand("AT+CWJAP=\"MY-SSID\",\"MY-PASSWORD\"", 5000);
  callWifiBoardWithCommand("AT+CIPSTATUS", 1000);
  callWifiBoardWithCommand("AT+CIPSTART=\"TCP\",\"www.bonesukll.com\",80", 4000);
  String cmd = "GET /index.html HTTP/1.1 Host: www.boneskull.com";
  
  callWifiBoardWithCommand("AT+CIPSEND=" + String(cmd.length()), 0);
 
  callWifiBoardWithCommand(cmd, 0);
  
  while(mySerial.available()){
    if(mySerial.find('O')){
      mySerial.println("+IPD");
    }
  }
}

void loop() { // run over and over
  if (mySerial.available()) {
    Serial.write(mySerial.read());
  }
  if (Serial.available()) {
    mySerial.write(Serial.read());
  }
}

void callWifiBoardWithCommand(String command, int waitFor)
{
  delay(1000);
  mySerial.println(command);
  delay(waitFor);
  while(mySerial.available()){
    Serial.write(mySerial.read());
  }  
}

Good grief.

We knew the ESP8266’s hardware was certainly capable of use as a project’s main controller, but the software wasn’t yet ready.

Mercifully, we wouldn't have to wait long.

NodeMCU Makes Progress

Fast-forward some months (IIRC), and a native SDK appeared from Espressif. We started seeing the potential of the ESP8266, instead of just providing serial-over-WiFi to a micro.

This native SDK powered an ambitious project, NodeMCU. It drove the ESP8266 by way of Lua, a scripting language often found in moddable video games like WoW.

It had an associated development board (also confusingly called “NodeMCU”), which provided easy flashing via a USB port; before the NodeMCU board, ESP8266-based devices were all simple, breadboard-hostile modules:

An ESP-01 needs a breadboard adapter, as the pins are all adjacent

They remained inhospitable even if you did manage to get them onto your breadboard:

This circuit was really fiddly and tedious to flash

My early attempt at using an ESP-07 (and its adapter) to transmit temperature from an DS18B20 (datasheet) required considerable effort:

Four pins exposed for flashing & serial comms; the jumper switches flashing mode

The NodeMCU board, on the other hand, was a complete package:

Note USB Micro-B receptacle, reset button, onboard passive components

While the NodeMCU board lives on in countless clones, contributions to NodeMCU-the-project waned. The community (and Espressif, which were wise to embrace open source) directed resources into the Arduino on ESP8266 project, which provided a familiar framework, and allowed firmware authors to leverage the existing Arduino ecosystem. Once this Arduino core became stable, many libraries “just worked” out-of-the-box, and those which didn't were easily ported via C preprocessor macros.

Lua isn’t a “bad” language by any means—just relatively obscure. I found the main drawback of the NodeMCU project was its small community; if the official project didn’t support a certain sensor or other component, you’d likely have to write that library yourself.

While the ESP8266 community buzzed, another project out of the UK slowly gathered momentum: MicroPython.

MicroPython Gets Crowdfunded

MicroPython is a Python 3 (-based) runtime designed for deployment on microcontrollers. Its genesis was a 2013 Kickstarter campaign featuring the PyBoard.

After the success of the original campaign (which eventually allowed its creator, Damien George, to work on MicroPython full-time), Mr. George launched another in early 2016, targeting our revered ESP8266. This well-timed campaign succeeded, despite the “reward” being little more than open-source software!

By the time it landed, many hackers already had ESP8266 boards in their clutches. Now that the first port of the framework was public, it was just a matter of time before the community extended MicroPython support to other platforms:

• Adafruit ported it to Atmel's ARM Cortex-M0+-based SAM D processors by way of a fork
• The Micro:bit Educational Foundation (presumably) ported to the BBC micro:bit, which runs Nordic's popular nRF51822 SoC for BLE
• Pretty much my favorite: PyCom helped develop a port for Espressif's newer SoC, the ESP32
• A few others I'm unfamiliar with (please comment if you've used them!)

MicroPython works well on ESP8266-based boards, as they are reasonably speedy at 80MHz. However, the ESP8266 has only 160KiB of SRAM, less 64KiB for the bootloader, less MicroPython's own overhead. You're really going to be squeezed here. Speaking from experience, it’s easy to run out of memory via object allocation, resources left open, or even too many lines of code—via import statements or otherwise!

While ESP8266-based dev boards are cheap and plentiful (a good one can be purchased direct from China for under $3 USD), for a few dollars more—under $7 USD—you can get your meathooks on an ESP32-based board.

This is important because the ESP32 runs MicroPython like a boss.

MicroPython Ported to the ESP32

The ESP32 isn’t a “successor” to the ESP8266; rather, it’s a step up: a SoC for more demanding applications.

From the chart below, you can see it's a far more capable device. But the big differentiator for us is the available RAM, which I've circled:

It's all about the RAM.

This gives users of MicroPython much more flexibility--in fact, this is 320KiB more RAM than the PyBoard, which MicroPython was originally designed for!

In practice, this means you can stop worrying (as much) about how many modules you're import-ing, how many objects you're allocating, etc.

Is this overkill? Is this wasteful? Is it rad? Yes, yes, and yes.

As a library author, writing MicroPython for the ESP32 means you have the luxury of implementing better abstractions. Without the extra RAM to play with, low-level APIs and cut corners are the name of the game.

With the ESP32, hardware won’t be holding you back from using MicroPython. That said, MicroPython and its ecosystem has “quirks” that Pythonistas need to understand before digging in. I’ll cover a few below.

MicroPython Has Peculiarities

If you are hoping to pull random module off of PyPi and just import it, steel yourself for disappointment. I have yet to find non-trivial code written for Python 3 that “just works” in MicroPython. I ran into four (4) differences which were particularly problematic for Pythonic portability:

1. MicroPython does not implement the entire Python standard library. For example, much of sys doesn’t make sense in the context. Metaprogramming tools like typings or abc don’t exist. This doesn’t necessarily mean you can’t implement them yourself!
2. Of the standard library which it does implement, it wants to prefix those libraries with a u (e.g. ujson).
3. You cannot chain exceptions in MicroPython. This means you cannot catch an exception and re-raise it as a different one. All you can do with try/except is eat exceptions.
4. Subclassing builtins doesn’t always work the way you’d expect.

In addition, the ecosystem is immature. MicroPython lacks official guidelines and best practices for module authors. Installation of modules on devices involves manually copying files around. Fragmentation may become more of an issue due to various forks.

Better tooling and best practices will come in time, but we can still get things done and have fun doing it with MicroPython. I can’t wait to rewrite a bunch of my custom firmware with it. Not joking.

In my next post on the subject, I’ll walk through connecting a MicroPython-laden ESP32 to an MQTT broker. Just keep hitting refresh.

Mocha v4.0.0 is nearing release. With this new version also comes the obligatory breaking changes, and I'll enumerate them below.

UPDATE (April 18, 2018): Mocha v4 was released on October 2, 2017.

Mocha Will No Longer Support Node.js Pre-v4.0.0

There are several reasons for this:

1. These versions of Node.js are no longer maintained and (some) have known security vulnerabilities which will not receive official patches. Users of these versions should upgrade Node.js. For example, v0.10.0 and v0.12.0 have been end-of-lifed since October and December 2016, respectively.
2. Mocha's own dependencies have already dropped support for these platforms, and so Mocha cannot address to security vulnerabilities or critical bug fixes in reasonable manner nor timeframe.
3. Mocha's development environment can no longer run out-of-the-box in a pre-v4.0.0 environment.

The following unmaintained versions of Node.js will no longer be supported by Mocha:

• v0.10.0
• v0.11.0
• v0.12.0
• iojs
• v5.x

Node.js v5.x is likely to work as long as Node.js v4.x does, but will not be in Mocha's build matrix.

It's important to note that Mocha v3.x will still work on these platforms, but users can no longer expect upgrades.

For further reading, see this article by Eran Hammer and also the Node.js Release Schedule.

The above change also means:

Mocha Will No Longer Support npm Older Than v2.15.11

Reasoning:

• npm v2.15.11 is the version which shipped with Node.js v4.0.0.
• Even older versions didn't support scoped packages (@foo/bar), nor the caret (^) semver range specifier. This made it virtually impossible to include production dependencies which used either of these features.

Users are encouraged to upgrade to the latest version of npm.

Users of Bower are also encouraged to upgrade, because:

Bower rides into the sunset. Photo by Alex Wigan / Unsplash

Mocha Will No Longer Support Bower

This may or may not make it into v4.

UPDATE (April 18, 2018): This made it.

Bower is an excellent tool to install front-end dependencies. However, over the years, more robust solutions have evolved. Its time has passed.

Mocha v4 will remove its "browser bundle" (mocha.js) from version control. This means bower will not install Mocha v4 out-of-the-box. There may be a workaround by means of a "resolver", but YMMV.

A Warning To Those Bundling Mocha

This change may also impact those users bundling Mocha themselves via browserify, webpack, etc. The browser field will now point to mocha.js, which is already bundled. You may no longer need to bundle yourself, or you may need to find a different way to do it. If you cannot find a reasonable workaround, please open an issue on our tracker.

Finally, my favorite:

Mocha Will No Longer Support Non-ES5-Compliant Environments

Mocha is likely the last major actively maintained testing framework to officially support these browsers.

While Mocha's maintainers have proudly sustained the poor, piteous developers who have such business (or government!) requirements, the overhead of retaining compatibility has become an albatross.

Much like supporting older versions of Node.js, this has severely limited the project's agility. Mocha should shed some weight after this, as it contains a plethora of handrolled shims.

The following browser environments will no longer be supported in Mocha v4:

• Internet Explorer 7
• Internet Explorer 8
• PhantomJS 1.x

API and output changes follow.

Other Breaking Changes

Read this to save yourself some time.

Mocha Won't Force Exit

This may or may not make it into v4.

UPDATE (April 18, 2018): This made it.

To avoid false positives and encourage better testing practices, Mocha will no longer automatically kill itself via process.exit() when it thinks it should be done running.

If the mocha process is still alive after your tests seem "done", then your tests have scheduled something to happen (asynchronously) and haven't cleaned up after themselves properly. Did you leave a socket open?

Supply the --exit flag to use pre-v4 behavior.

If you're having trouble figuring out where the hangup is, this package might help.

UPDATE (April 17, 2018): I changed the link above; I've since had better success using wtfnode to debug this class of problems. Be sure to run it against the _mocha executable (not mocha)!

UPDATE (April 23, 2018): Fixed link to wtfnode.

Output

There will be a few changes to reporters, which may negatively affect projects consuming Mocha's output directly:

• The "unified diff" will now contain separators, as it has been difficult to read.
• Upon error, the test contexts will be indented instead of smooshed into a single line.

Forward!

By shedding support for older environments, Mocha becomes more nimble, and positions itself to leverage the present and future innovations of Node.js and JavaScript.

A glorious thing nowadays is that you needn't be an AI researcher nor have expensive hardware to leverage machine learning in your projects.

Granted, a domain-specific design will net greater benefits in the long run. Yet, until recently, a general-purpose, off-the-shelf solution wasn't easily consumable by your average developer (that's me). Nor was such a monster available—by virtue of APIs—to resource-constrained devices.

Below, I'll introduce the reader (that's you) to API-based object recognition, and how to implement with cheap hardware and JavaScript.

The Raspberry Pi Zero W

Firstly, you will need an internet-enabled Raspberry Pi.

For this project, the most value you'll get for your money is probably a Raspberry Pi Zero W.

Got a different Raspberry Pi?

Most RPi boards have a camera interface. A RPi Zero v1.3 (the non-WiFi one with the camera interface) will also need a USB WiFi dongle, Ethernet adapter, or "hat" providing connectivity.

The "original" RPi Zero, v1.2, does not have a camera interface, and will not work.

While the Zero isn't fast, it can run Linux, which makes it more capable than your garden-variety microcontroller. As you can see, it huffs & puffs to execute a Node.js "useless script":

$ time node -e 'process.exit()'
node -e 'process.exit()'  5.94s user 0.16s system 99% cpu 6.157 total

From the above, I'm going to gingerly assume training a convolutional neural network on this ARMv6-based single-board computer would be a fool's errand. But that's not why you'd buy a Pi Zero W, or build anything with it. This is why:

• It's ten bucks.
• It's smaller than a credit card in two out of the three dimensions which count.
• It's ten (10) dollars, USD.
• With some effort and more cheap hardware, it can be powered via ethernet.
• It exposes GPIO pins. Go nuts.
• Did I mention it's $10?

Once we've got an RPi to work with, we'll need a camera.

What about Brand X single-board computer?

The Node.js code leverages the raspicam package, which is a wrapper around raspistill. So, if it can't run raspistill, we can't use it for this tutorial.

The Camera

A supported module based on OV5647 ("v1"; datasheet) or IMX219 ("v2"; datasheet) will work. There are "official" modules which can run up to $30, but I've seen a knockoff "v1" from China around $6 on the low end. You don't need an 8MP camera to do this; we'll be taking rather low-resolution photographs.

These cameras are equipped with fixed-focus lenses. I've found that you want to position the camera no less than about 12" (30.48 cm) from the target (another option may be attaching a zoom lens). I'll leave this as exercise to the reader, but here's my solution:

The camera module connects to the RPi via flexible flat cable to a ZIF socket. A RPi Zero supports a cable of width 11.5mm, but the other interfaces expect a width of ~16mm. Adapters and conversion cables exist; one such cable comes with the official case.

Building with LEGO?

For those attempting to build a custom tripod with LEGO, I note that the dimensions of my "v1" camera module are (in one dimension, anyway) roughly 24mm, which corresponds to a length of 3L, or the length of a 3623 plate. 1 x 5 Technic plates 32124 and 2711 are helpful here, as well as 32028 to secure the module in place.

Now that we have the basic hardware together, let's get Node.js installed.

The Node.js

I'm going to assume you've got Raspbian Jessie installed. Theoretically, any distro based upon Debian Jessie should work. Maybe others too, but I haven't tried them!

For this project, we're using Node.js 8 (version 7.x may work with certain command-line flags, but I haven't tried it). Normally, I'll grab binaries from NodeSource. However, they don't support ARMv6.

If you are using a RPi 3, go right ahead and use NodeSource's distributions, then skip to the next section.

But for the Zero, you have several options, two of which I can recommend:

1. Manually install a tarball from nodejs.org; as a superuser, untar the archive and extract it over /usr or /usr/local, or

2. My preferred method: install via Node Version Manager. As a normal user (e.g. pi), follow the instructions on the site and in the terminal to install NVM. Then, run:

   $ nvm install 8
   

   This will install the latest version of Node.js 8 under your home directory, then enable it. Run node -v to test your install.

The next piece of the puzzle is an API key.

The Cloud

This project uses IBM's Watson Visual Recognition (hereafter "WVR"). It's available from within IBM's PaaS, Bluemix (wiki).

Use may use an existing Bluemix login, or sign up here. Once you're logged in, from the same page, create a service instance; name it whatever you like.

After it's ready, you'll land on the dashboard for the instance. Here, you can find your API key:

1. Click "Service credentials".
2. Click "View credentials" under "Actions".
3. Copy the API key and paste it somewhere safe (like a password manager app) to keep it handy.

Armed with our API key, let's take a short detour into concepts. I promise this won't hurt.

The Concepts

You'll need to know this stuff or you will be arrested by the police.

The Class

The most important concept you need to understand is the "class". In fact, the picture on the WVR site illustrates this well:

In the picture above, we have five (5) classes:

1. Green: the subject of the image is green
2. Leaf: the subject of the image contains a leaf
3. Plant stem: The subject contains a plant stem
4. Herb: the subject of the image is in the "herb" category of plants
5. Basil: the subject is specifically a basil herb

It's important to note that a class may be as narrow or broad as you wish. For example, there are many shades of the color "green"--but only one plant named "basil"!

While WVR has some pre-existing classes which work out-of-the-box, our aim is to create our own custom classes.

To do this, we will need to create a classifier.

The Classifier

A "classifier" can be thought of as a logical collection of classes. For example, say you had four friends and family you wanted to be able to recognize the faces of. Each individual could correspond to a "class":

1. Uncle Snimm
2. Aunt Butters
3. Sister Clammy
4. Bill

The classifier would be "faces of friends & family", or something of that nature. Perhaps you would add another class to this classifier which was only "family"--you could re-use the same images.

In addition to this, WVR allows have a single special class within your classifier representing images which are not in the classifier. For example, you could put images of random strangers (or your enemies) in this "negative" class. This helps the underlying network avoid false positives.

If you don't have any enemies to use for this project, I can provide a few pointers on how to acquire them. I'll save that for a future post.

More use-cases of classifiers include:

• By limiting the scope of the classes to which WVR compares an image, we increase the likelihood of a good match
• Similarly If we know our picture won't be in classifier X, then we don't need to classify using classifier X
• Limiting scope will increase performance (though I don't know by how much--seems logical, however!)

So, how do we create classes and classifiers?

The Training Regimen

When we create a class, we give WVR an archive (a .zip file) of images. These images are positive examples of class members. Once this archive is uploaded, the training process begins. Training is a process of "learning" in "machine learning". Depending on the number of images in your archive(s), this can take a little while (on the order of minutes for just a paucity of images).

Remember, you can also supply your new classifier a single .zip archive of negative examples.

In other words, in WVR, the action of creating a classifier implies training it as well.

Now, for the payoff. Once we have trained a classifier, we get to classify images!

The Classification

Classification is the action of providing WVR one or more images to a classifier, and receiving information about how well each image might "belong" to its classes.

For each image, WVR will give you zero or more classes with a corresponding fraction between 0 and 1. This fractional number represents confidence, not accuracy. Then, for some classifiers, a confidence for class X of 0.6 could imply "member of class X", but for others it could disqualify an image completely.

If WVR's confidence drops below a certain threshold, it won't return a number at all. This threshold is configurable; the default is 0.5. If you're only using 10-50 images, you may want to drop it to 0.3-0.4.

Let's recap the four terms we need to know:

• Class: A set of images having a common attribute which we intend to recognize
• Classifier: A logical collection of classes
• Classification: Using WVR to decide which class(es) an arbitrary image could "belong" to, by reporting a confidence level
• Training: In WVR, we train a classifier; we provide images to the service which we will then use for classification

What classifiers will you create? Wait--before you answer--let me rain on your parade. I'll tell you what I wanted to do until reality sunk in. Gather 'round and weepe, while I bid mine own tale of woe!

The Tale of Woe

I like LEGOs. Inspired by Jacques Mattheij's LEGO sorting project, I wanted to see if I could easily spin up an accurate classifier for different categories of LEGO pieces. For example, could I recognize "plates":

versus "bricks"?

Could I do this? No. Of course not. The long answer:

Once I had a working PoC of my tool (see below), I took many, many pictures of LEGO bricks, plates, etc. They looked something like this:

But the classification worked poorly. I tried a lot of different things, such as removing color information, changing backgrounds:

Or fiddling with the color temperature:

Soul-crushing, abject failure. Every. Time.

One thing I did keep was a lower resolution--high resolution images will not necessarily net better results! In fact, often the opposite: a higher-resolution image will potentially contain an unnecessary level of detail, resulting in extra useless information.

Like usual, I pondered on "useless information".

Look at the previous image. Its resolution is 428x290; multiply and we get 124120 pixels. If we rotate it slightly, then crop down to the relevant information, we get:

That's 20x202 or 4040 pixels. So:

4040 / 124120 = ~0.0325
0.0325 * 100 = ~3.25

That means a bit over 3% of the photos I was taking contained relevant information. It follows that 97% of each photo was useless, wasteful trashpixels.

Remember, the RPi cameras are fixed-focus. If I had a better camera or and/or macro lens, I probably could have made this work. Alas!

LEGOs were too small. I needed something larger; something with fewer important details.

My eyes darted around the room. What would be a good size for a picture taken about 12" away? Maybe kitchen utensils? Cups? That seems boring. Regrets? What do I have a lot of... (I realize you can't answer this)?

Maybe you have a few of these around:

Wall Warts!

If you're into hobby electronics, you might actually collect wall warts. I have ...a few extras.

You may not have, say, 20 or 30 of these handy (without having to, you know, unplug stuff). But I do. If you can put aside your envy, you'll notice the signal-to-noise ratio improves dramatically:

The images are still a bit blurry, but it doesn't matter--we're not trying to read the fine print.

Also, scavening similar-sized objects for a "negative example" class was almost enjoyable:

I settled on a resolution of 640x480, and chose to discard color information. See the end of this post for links to my class archives, if you'd like to try them yourself!

Given wall warts are usually black, maybe I would have better results if I kept the color data???

I can offer some general advice for taking your own snapshots:

1. Keep the signal-to-noise ratio high; don't include unnecessary pixels!
2. Color temperature, shadows, lighting--the less consistent, the more images you'll need.
3. Don't worry too much about blurriness (OCR this ain't)
4. Consider different placements and angles of your objects
5. 50 images per class or more. WVR's lower limit is 10 per, but 50 is recommended as the absolute minimum!
6. Even a "low" confidence level can work in practice. Adjust your threshold; as long as the network is more confident when you expect it to be, then you're doing fine!

To help me:

1. Take all these pictures,
2. Put them in the correct buckets,
3. Archive them, and
4. Upload them to Watson,

I ended up writing a tool. That tool is called puddlenuts. No, really.

Introducing puddlenuts

puddlenuts is what I wrote to ease the insufferable process of taking hundreds of pictures.

Don't freak. You don't need to take them all at once! You can always add more images to a class later. This is called retraining. puddlenuts can help with this.

At this point, you should have your RPi configured, with Node.js installed and camera connected. If you don't, what is wrong with you?

On your RPi, install puddlenuts, then go mow the lawn while you wait:

# this may require `sudo` if you aren't using NVM
$ npm install --global puddlenuts
- [ ] # ... time passes ...
+ puddlenuts@0.2.4
added 245 packages in 488.451s

puddlenuts isn't a library; it's a command-line tool. What can it do?

$ puddlenuts --help

Commands:
  classify [..classifier]         Classify an image against one
                                  or more classifiers by a
                                  snapshot or existingimage.
                                  Default is to run against all
                                  classifiers.
  shoot <classifier> <classes..>  Take snapshots to train
                                  classifier with two (2) or
                                  more positive example classes,
                                  OR one (1) or more positive
                                  example classes, and one (1)
                                  negative example class (see
                                  "-n")
  train <classifier>              Train Watson with existing
                                  .zip archives

IO
  --color     Enable color output, if available
                                       [boolean] [default: true]
  --loglevel  Logging level
  [choices: "error", "warn", "info", "debug", "silly"] [default:
                                                         "info"]
  --debug     Shortcut for '--loglevel debug'
                                      [boolean] [default: false]

Watson
  --api-key  Set PUDDLENUTS_API_KEY env var instead!
                                             [string] [required]

Options:
  --help  Show help                                    [boolean]

We want to take photos, so shoot is the command we want.

Shoot

Here's the dirt on shoot:

$ puddlenuts shoot --help
puddlenuts shoot <classifier> <classes..>

Camera control
  --raspistill, -r   Options for raspistill in dot notation
                     (e.g. "-r.width 640 -r.height 480")
                                                       [default:
           {"width":640,"height":480,"quality":100,"timeout":1}]
  --limit, -l        Limit to this many snapshots per class
                                          [number] [default: 50]
  --delay, -d        Delay between snapshots in ms
                                   [number] [default: 3000 (3s)]
  --class-delay, -D  Delay between classes in ms
                                 [number] [default: 10000 (10s)]
  --trigger, -t      Set trigger interrupt on this GPIO pin (RPi
                     only)        [number] [default: No trigger]

Watson
  --api-key  Set PUDDLENUTS_API_KEY env var instead!
                                             [string] [required]
  --retrain  Retrain classifier (if exists)
                                      [boolean] [default: false]
  --dry-run  Don't actually upload anything
                                      [boolean] [default: false]

Class
  --negative, -n  Include negative example class in training
                  (will be final class)
                                      [boolean] [default: false]

IO
  --color     Enable color output, if available
                                       [boolean] [default: true]
  --loglevel  Logging level
  [choices: "error", "warn", "info", "debug", "silly"] [default:
                                                         "info"]
  --debug     Shortcut for '--loglevel debug'
                                      [boolean] [default: false]

Options:
  --help  Show help                                    [boolean]

Examples:
  blueface/bin/puddlenuts.js shoot  Take snapshots to train or
  dogs poodles -n --retrain         retrain the "dogs"
                                    classifier, with a positive
                                    example set of "poodles" and
                                    a negative example set (i.e.
                                    non-dogs); upload to Watson
  blueface/bin/puddlenuts.js shoot  Take snapshots to train (do
  fish catfish swordfish --dry-run  not retrain if "fish"
                                    exists") the "fish"
                                    classifier with positive
                                    examples of "catfish" and
                                    "swordfish"; don't upload

The "camera control" options will allow you granular control over raspistill, which is the official command-line interface for the RPi cam. This is how you can change the resolution, fiddle w/ color correction, silly effects, etc.

These options also allow you to define how many pictures to take and how quickly to take them. After each picture is taken, there's a short pause. I found a delay (--delay) of less than three (3) seconds between pictures isn't quite enough time to comfortably switch an object out for another, or readjust, so this is the default.

Since you tell puddlenuts to take snaps for multiple classes, you can also tell it how long to pause between switching from the last picture of one class to the first picture of the next. I was taking a bit longer to get setup when the class changed (e.g., swapping my pile of wall warts for a pile of random, non-wall-wart objects)--this defaults to ten (10) seconds.

Finally, --limit will limit each class to exactly the number of images you provide it (minimum 10).

The --trigger option allows you to wire a switch to one of the RPi's GPIOs. If the GPIO is "high", snaps will be taken (with specified delays). But if it's "low", puddlenuts will pause until you flip the switch back "high" again. Neat!

I realize this first example might get me some unintended search engine traffic, but here we go:

$ puddlenuts shoot dogs poodles --negative --retrain

But what the above command will do, in gory detail, is:

1. Take 50 pictures of "poodles", with a 3s delay between each
2. Pause 10s
3. Take 50 pictures of "not dogs", with a 3s delay between each
4. Create .zip archives for each set of 50
5. If the "dogs" classifier doesn't exist, it gets created
6. If the "poodles" class doesn't exist, it gets created/trained
7. If the "poodles" class does exist, the 50 images are used for more training
8. If the "negative examples" ("not dogs") class doesn't exist, it gets created/trained
9. If the "negative examples" class does exist, the 50 images are used for more training

You'll also see plenty of beautiful console output while this is happening.

There's certainly room for improvement here; try it out and let me know what could be easier.

Train

Execute puddlenuts train --help for more information, as I realize it's silly to copy and paste the output here.

The train command allows you create (or retrain) classes using existing .zip archives. It doesn't take pictures.

For example, if you have to cobble together several "shoot" runs (use puddlenuts shoot --dry-run to create .zip files w/o uploading; see log output for their location), or need to collect some images via other means, you should use puddlenuts train.

Classify

This is the "fun" command—it will take a picture and attempt to classify it against the classifier(s) you provide.

If you don't provide a classifier, the image will be compared against all classifiers. Watson provides a "default" classifier, which may be of use—give it a shot and see.

Two more options of note:

• You can also tell puddlenuts classify to just upload a file (via the --input <path/to/file> option) instead of take a picture.
• You can specify the confidence threshold with --threshold <number between 0 and 1 inclusive>. You probably don't want to set this to 0 or 1, as the former will give you way too much information, and the latter will give you diddly squat.

What this command provides is a pretty-printed data structure with the classification information. This is an unwieldy tree, and I wasn't sure how to better distill and/or represent it. So you just get a dump. You must admit, it's really all you deserve. Regardless, please let me know if you have a better idea.

For the conclusion, let's stop.

Conclusion

A novice consumer of ML API's may trip up or become frustrated when a system doesn't do what you expect. You must remember that bringing this kind of power down to "our" level will come with caveats. There are limitations in what these shrinkwrapped solutions can offer, but with some persistence, I believe these technologies are widely applicable.

It's my hope you learn from my mistakes (and I hope I learn from them as well). All things considered, it's way easier than I would have expected to get started with this stuff. And cheaper. It's trivial (JavaScript) to do more (computer vision) with less ($10 computers).

My prediction is this trend will continue. In a future post, I'll explain how to do nearly everything using almost nothing.

Addendum

Below are links to the images I used for my "wall warts" classifier. There are only two classes:

• Positive examples (direct download) (wall warts)
• Negative examples (direct download) (not wall warts)

And here's my slide deck associated with a talk I gave on this subject at the JavaScript & the Internet of Things meetup in Portland, Oregon, on August 22 2017.

So there I was, writing a CLI tool in plain old JavaScript. I had pulled in yargs, as I often do.

Here's the source of my executable:

I'm looking at the above and wondering why demandCommand is highlighted as an instance member function, but usage, command, help etc. are not.

I note that when typing the call to usage(), I am not offered any information about the parameters or the types thereof.

WebStorm & TypeScript

Some time ago, I had noticed WebStorm will pull in type definitions from 3rd-party libraries which ship with TypeScript definitions. WebStorm will use those definitions to improve highlighting, provide type hints, inline docs, better refactoring, etc--even if I'm not actually using TypeScript in my project.

One day, I decided to see what would happen if I simply added @types/yargs as a dev dependency:

$ npm i @types/yargs -D

I ran the above, then reloaded my source. Imagine my surprise when I saw:

Huh. What if...

Wow.

That's Cool, But...

... but I don't really want to install a bunch of @types/* as development dependencies. The modules aren't particularly heavy. Yet, stuffing this in the package.json of a pure JavaScript project:

{
  "devDependencies": {
    "@types/yargs": "^8.0.2"
  }
}

...provides no functional value whatsoever.

Furthermore, even if this works in WebStorm, I dunno if it works in VS Code, Atom, Sublime, etc. I don't expect contributors to use the same IDE or editor I do. A dotfile is one thing, but we're talkin' 'bout a dependency.

External library—get it? Laëtitia Buscaylet / Unsplash

The Hack: External Libraries

WebStorm allows you to add arbitrary "external libraries" to your project.
This strategy is how WebStorm manages to provide code assistance for Node.js core modules, ES5, ES2015+, the DOM API, WebGL, etc. These are enabled by default, as well as the local node_modules/ dir.

If you reference a lib in a <script> tag which points to a CDN, WebStorm will complain that it doesn't know anything about that script; it prompts you to download the script and add it as an external library.

But this can point to a directory or file on disk. So I removed @types/yargs from my project, and installed it globally:

$ npm i -g @types/yargs

I then went into the "external libraries" settings of my project in WebStorm, and added the destination directory (for me, this was /usr/local/lib/node_modules/@types/yargs; yours may be different). I marked this library as "global", so that I could use it in any project I open in WebStorm.

I then configured my project to actually use the library (via something called "Manage Scopes", which isn't the same thing as the other "Scopes" in WebStorm).

Sure enough, this enabled the extra code assistance—even if @types/yargs was no longer a dev dependency.

Making It Suck Less

This was dope and sick, but it also meant I would need to install the definitions I wanted manually, and then keep them up-to-date, which was not dope nor sick.

I knew that all of the type definitions live in a single repo.

I tried to install that globally, but there is no definitely-typed package (see for yourself). I tried to install the Git repo globally, which worked:

$ npm install --global \
    https://github.com/DefinitelyTyped/DefinitelyTyped.git

But upon closer inspection of /usr/local/lib/node_modules/definitely-typed, it was not actually a Git working copy. This means git pull wouldn't just work.

Still, if I had a working copy, I could just symlink its types dir to /usr/local/lib/node_modules/@types...

The all-types Package

...which is exactly what I automated with all-types.

The README has detailed instructions about how to install, set things up in WebStorm, and keep the install up-to-date.

If you pay attention to nothing else in the README, I suggest that you don't add all of DefinitelyTyped at once. Not because of performance--rather, it just adds a lot of noise to the code assistance environment.

I'm interested in making this easily consumable in other editors & IDEs, if possible. If you find your editor of choice can do this, please contribute with instructions!

Something for Nothing?

Let's agree that a great thing about TypeScript is the code assistance it provides to editors and IDEs. Even if you abhor TypeScript, that's tough to argue with. But what if you got that for free?

This hack provides at least some of the value from the TypeScript ecosystem without having to commit to TypeScript.

Yet, the most obvious deficiency is that unless your own code is written in TypeScript, this is limited. But it's much, much better than nothing.

It's worth mentioning that editors & IDEs are free (as in beer). Do they work with all-types? I have no idea. Clue me in!

Title image by Windell Oskay; part of a series of LEGO Abominations

While trying to get going on a new blog post, I found myself frustrated with GitHub Pages' Jekyll sandbox.

My previous site theme was hand-rolled (I bet you'd never guess!) from a Material Design Lite template. Admittedly, this was hubris. I am ignorant about "responsive" design (does that actually mean anything?), PWAs, accessibility, et cetera.

Leveraging Jekyll plugins would ultimately mean more overhead to publish my site, because I could not rely on GitHub to generate it for me. So I threw in the proverbial towel, and decided to look at other blog software.

Blog Software Requirements

1. Has some sort of GUI.
2. Easy on the wallet.
3. Not WordPress.

I hate the word "blog".

I Picked Ghost

I decided to self-host using Ghost for now. What I like about Ghost:

1. It's easy to get up & running with a self-hosted instance.
2. Admin interface(s). While I generally enjoy puttering around in my terminal and editor, this is not one of those times. I just want to write.
3. Ghost is implemented in Node.js, so I can hack on it if I need to.
4. I could import my post (sic) from Jekyll pretty easily using nodejs-jekyll-to-ghost.
5. Not WordPress.

Finally, Since I have no design sense to speak of, I threw down for some themes by Malvouz, and tweaked one of them to be more boneskullish.

That's my story.