Speed up NodeJS server-side development with Webpack 4 + HMR.

When webpack 4.0 came out, I decided to migrate my codebase implementing webpack for the frontend to webpack 4.0.

One thing I hadn’t invested time to do however, is to enable HMR in my backend code. I have a project that takes approx 30 seconds to start up, mainly because of babel transpiling my whole codebase on memory before running my server, which takes a lot of time and resources. In production, I was using babel-node to transpile my entire code before actually running the backend server.

I decided to use webpack for my backend server and I achieved to speed up compilation by 40x.

Structure and Logic:

As you can see, I have 2 webpack configurations, one for the server, and one for the client. The client-side, is just your run-of-the-mill, standard webpack configuration, so we won’t go into this now. The server-side code lies inside the “server” folder.

As it stands now, for development, the entry point of my backend code is index.js.

require('babel-core/register');
require('babel-polyfill'); 
global.Promise = require("bluebird");
require('./server.js');

Inside server.js, I am setting up an express application, and in the end, I am listening to the server.

import express from "express"
const PORT = process.env.PORT || 3000;
const app = express();

/*set up all your server config here*/

server.listen(PORT, function () {
	console.log('Server listening on', PORT);
});

When starting the server using index.js as my entry point, babel transpiles all my backend code starting with server.js and loads everything in memory. If I make a change to my code, I will have to stop the server, and rerun it. When you have many .js files for your backend, this can be a tedious procedure, as it may take a while for your server to start up.

I decided to make my life a bit easier, and use webpack for my backend code. I also decided to get rid of babel cli in the process. I would use webpack to bundle my backend code for production use.

The idea

The process of speeding up development of the backend, is summed up in the following process:

• Use webpack to produce the first output using babel loaders, in a single file.
• Enable the HotModuleReplacementPlugin() so that subsequent compilations only change what’s necessary in the output file.
• Use webpack’s watch feature to enable recompilation and updating the server file automatically.
• Start nodemon on the output file, so that the server is automatically restarted when anything changes inside the file.

Implementation

The following code represents my server.config file.

const webpack = require('webpack');
const path = require('path');
const nodeExternals = require('webpack-node-externals');
const nodeEnv = process.env.NODE_ENV;
const isProduction = nodeEnv !== 'development';

// Common pluginslet plugins = [new webpack.DefinePlugin({'process.env': {NODE_ENV: JSON.stringify(nodeEnv),},}),new webpack.NamedModulesPlugin()];

if (!isProduction) {
    plugins.push(new webpack.HotModuleReplacementPlugin())
}

const entry = isProduction ? ['babel-polyfill', path.resolve(path.join(__dirname, './server.js'))] : ['webpack/hot/poll?1000', 'babel-polyfill', path.resolve(path.join(__dirname, './server.js'))];

module.exports = {
    mode: 'development',
    devtool: false,
    externals: [nodeExternals()],
    name: 'server',
    plugins: plugins,
    target: 'node',
    entry: entry,
    output: {
        publicPath: './',
        path: path.resolve(__dirname, './'),
        filename: 'server.prod.js',
        libraryTarget: "commonjs2"
    },
    resolve: {
        extensions: ['.webpack-loader.js', '.web-loader.js', '.loader.js', '.js', '.jsx'],
        modules: [path.resolve(__dirname, 'node_modules')]
    },
    module: {
        rules: [{
            test: /.(js|jsx)$/,
            loader: "babel-loader",
            options: {
                babelrc: true
            }
        }],
    },
    node: {
        console: false,
        global: false,
        process: false,
        Buffer: false,
        __filename: false,
        __dirname: false,
    }
};

A little explanation about certain parameters:

node: {
    console: false,
    global: false,
    process: false,
    Buffer: false,
    __filename: false,
    __dirname: false,
}

In this configuration we tell webpack to not replace / process certain commands that we really care about, such as __filename and __dirname. If __dirname was processed by webpack, then it would return an empty string, and not the proper location of the file currently calling the function. This would be devastating for commands like fs.readFileSync and other node-specific commands.

We explicitly set the mode to development, to allow webpack to figure out that we are building for the backend and not the frontend. We also exclude any node_modules from the compilation process, which means that anything that webpack processes should be included in your main code. The final output file will not contain anything from node_modules, nor will any transpilation take place inside them. This will not only speed up compilation time, but will also save resources, as node_modules for the backend already support node 6.0 and above without transpilation. For this, I used webpack-node-externals webpack plugin, which proved to be very handy in constructing rules to be passed to webpack’s externals configuration in order to exclude node_modules from being processed.

The configuration also changes a bit regarding webpack’s entry point:

const entry = isProduction ? ['babel-polyfill', path.resolve(path.join(__dirname, './server.js'))] : ['webpack/hot/poll?1000', 'babel-polyfill', path.resolve(path.join(__dirname, './server.js'))];

Whether in development or in production, the entry point is now my server.js file, and not my index file. Since our webpack configuration uses babel for processing the javascript files, and since we have already included ‘babel-polyfill’ as our entry point before including our main file, we shouldn’t use index.js as our entry file anymore when using webpack.

Running the scripts

For this methodology to properly work, we should configure our npm scripts to do the following with this order:

1. Build the server bundle once, and use it as an output so that the file is initially created.
2. Run webpack again with the --watch mode, so that code changes are being recompiled in the output.
3. In parallel with (2), run nodemon to watch for changes in the output file.

Below is a sample package.json configuration:

"scripts": {     "build:server:once": "cross-env NODE_ENV=development webpack --config webpack.config.server.js",     "dev:server": "npm run build:server:once && npm-run-all --parallel nodemon:prod watch:server",     "watch:server": "NODE_ENV=development NODE_ENV=development webpack --config webpack.config.server.js --watch",     "nodemon:prod": "cross-env NODE_ENV=development node-dev server.prod.js"   },

When you type dev:server in your terminal, webpack will do its magic, and you will be able to see your server being reloaded in the blink of an eye every time you make changes to your server code – and server restarting will be much faster that if you were restarting it using any index.js file which transpiled babel on the fly.

Why is it so much faster?

It all comes down to babel’s transpilation method, and the way we are restarting the server wthout webpack.

Upon each restart, babel requires all of your code, and transpiles it to something that your node version is able to understand. Babel does that each time you are restarting your server.

When using webpack with HMR, however, your entire codebase is transpiled once, and further changes you make to your code will only trigger a transpilation on this specific part of your code, not your entire codebase. New changes are then reflected into your existing server.prod.js file, which in turn triggers nodemon to restart it. Therefore, you are actually saving the time that Babel would consume if it was transpiling your entire codebase again.

In big projects, this can be a huge timesaver.

Results

When using webpack in production instead of babel-cli, in a project with 1000 source files (excluding node_modules), server booting took 3 seconds. When using the babel-cli output, server booting was taking 15 seconds! It seems that apart from transpiling using babel, Webpack makes additional optimizations in requiring the necessary code. The performance increase in booting was astounding.

During development, booting using index.js, where babel was transpiling the code on the fly, each server restart took 30-40 seconds on my machine. Now, I am looking at a server boot time of 3 seconds tops, with nodemon restarting the server each time a small portion of the output file changes. This has been a huge increase in my productivity.

Simplicity also played a huge factor. Most of the other examples I found on the internet used webpack-dev-middleware and webpack-hot-middleware directly from code, and used an express server as their main server (which allowed them to be compatible with webpack-hot-middleware‘s embedded express server. However I wanted to be server agnostic, and not make any changes to my main codebase (for example, I can now use a Hapi server – although I much prefer Koa and Express)

Caveats

There are some cases where this methodology will not work. It cannot work in cases where you are building an isomorphic server, for example. Theoritically, it should work, but I have yet to discover an efficient way of bundling .jsx files in the backend environment with HMR and nodemon. Kriasoft has managed to do it in an isomorphic application (react-starter-kit) which may prove useful to you if you wish to do such a thing.