How do you solve a problem like grunt packaging?

July 4th, 2014
grunt, js, javascript, process, front-end

tl:dr

(Too long, didn't read)

  • Big projects using grunt can end up with long and hard to read gruntfiles
  • When working on multiple similar projects, grunt task config and alias tasks are often repeated
  • How do we package these files to make them self-contained with dependencies and versioning?
  • There's a bunch of contrib options with pros/cons, but in the end I went with a basic self-written grunt plugin using grunt.config.merge()

Grunt is so useful, no?

I love using Grunt. It has become the de-facto tool of choice for front-end developers for development, testing and build workflows. I make extensive use of Grunt, both for work projects and personal projects, even going for so far as to use it in place of a makefile/bash script to automate deployment and server tasks in some situations.

However, a couple of things bother me about Grunt that I'd like to try to improve:

  • The common pattern of mixing configuration and code (task definitions) in one file (initially simple, much more confusing as the project grows)
  • Quite often common contrib task config is repeated in the gruntfiles of multiple similar projects, so I'd like a way to package and share common configs/tasks

Mixing config and code, and file readability

One of the strengths of Grunt is the easy to use configuration and its great ecosystem of contributed plugins. This works brilliantly for small projects, or if you only use contrib modules which just require configuration. However, this initial strength is also a problem when your projects grow and become more complex.

Once you start working on much larger projects with many contrib modules to configure, plus many custom coded tasks and also many alias tasks for your build workflow, the Gruntfile becomes a victim of its own success. It becomes large and hard to read, and task configuration can become mixed up with task code in a spaghetti mess.

To illustrate, I measured the number of alias tasks in one of our default Incuna gruntfiles and counted:

  • 46 configured contrib tasks
  • 19 alias tasks (some of these are tiny function tasks)

Having these all in one file makes for a hard to read mess, and doesn't show relationships between tasks and config. Imagine the basic gruntfile skeleton below with the properties filled in, and multiplied tenfold with more tasks:


module.exports = function (grunt) {

grunt.initConfig({
        project: {
            pkg: grunt.file.readJSON('package.json'),
            baseDir: 'project/',
            ...
        },
        clean: {
            target1: ...,
            target2: ...
            ...
        },
        copy: {
            templates: ...
            scripts: ...
            css: ...
        },
        usemin: {
            css: ...
            js: ...
        }
        connect: {
            dev: ...
        },
        compass: {
            dev: ...
            dist: ...
        }
});

    grunt.registerTask(
        'dev',
        'Compile CSS and run dev server',
        ['clean:css', 'compass:dev', 'connect: dev']
    );

    grunt.registerTask(
        'build',
        'Compile all required files',
        ['clean:css', 'clean:templates', 'clean:scripts', 'compass:dist', 'usemin', 'copy:css', 'copy:templates', 'copy:scripts']
    );

Code and config repetition

I work on a lot of projects which share the same codebase. This is very common in a commercial situation, where you are re-selling variations of a product, but is also generally applicable to any project. Projects using a common base often end up using the same basic grunt management tasks. If I'm building a drupal site, or an app in a specific node framework, I usually have a common set of grunt contrib configuration tasks for that type of project. I'll also put these config tasks together in blocks as alias tasks. Multiple alias tasks then often go together to make up another task. e.g. Compiling Sass, compiling templates and a full build or deploy script.

If you make common use of these batches of config and alias tasks in many projects, the code is repeated between projects and becomes hard to track and update when someone adds a feature or a fix. If task is customised slightly for a particular project, how do you easily see that it has different from your standard set? This is particularly relevant in an agency environment, where you might commonly build multiple customised versions of a product for different customers and make slight alterations to your base grunt tasks.

In the rest of our coding projects we try to keep code DRY, make it modular, versioned and re-usable, so why not apply these principles to our grunt code as well?

So what can we do to make things better?

Specifically, I'd like to be able to:

  • Package grunt plugin configuration tasks with the alias tasks which use those config tasks
  • Specify dependencies (and versions) for required grunt plugins needed in the package (and auto-install)
  • Version the above
  • Override the imported tasks easily in the project

I don't think these can be unusual problem/concerns, so naturally the first thing I did was to go searching on github and npm for any existing contrib modules which might help. I'll try to summarise the methodologies I discovered and why they weren't eventually suitable (YMMV) and what I settled with in the end.

Existing contrib options

grunt-load-options - Thomas Boyt

  • Blog post: http://www.thomasboyt.com/2013/09/01/maintainable-grunt.html
  • npm package: https://www.npmjs.org/package/grunt-load-options
  • Pros: split tasks, easy to see what task in what file
    • Each task gets it's own file, so easy to see how many tasks you have, and what they are named
  • Cons:
    • No implied relationships or explicit dependencies between the tasks/config. They are pure extracts from a regular gruntfile. i felt it would be more helpful to group sets of related tasks together
    • A file for every task makes for A LOT of files, especially when you have many alias tasks (over 60 in our case)

grunt-load-configs - Crenders

Other options

I only discovered these three after I'd made my own package, but they all offer the interesting concept of putting a second gruntfile in dependency packages which can have their own grunt configs. This potentially means the sub-package can be properly independent of the grunt packages in its parent project, and even specifiy different versions of contrib modules.

  • grunt-recurse: https://www.npmjs.org/package/grunt-submodule
    • Merges all config, so introduces possible task name conflicts
    • Requires extra boilerplate in main gruntfile (but not much)
  • grunt-subgrunt: https://github.com/tusbar/grunt-subgrunt
    • Seems to require each sub-task to be explicitly declared which was imply adding too much boilerplate again
  • grunt-submodule: https://www.npmjs.org/package/grunt-submodule
    • The most promising option. Tasks in submodules can just be called with grunt submodule:submoduleName/*:taskName

      • Solution

      All of the existing methods have their advantages and disadvantages. Some may be more or less suitable for your needs, but none of them fit ours exactly. In the end, I found the simplest answer lay with the built-in grunt plugin system. Initially this did not seem a promising avenue, as all the examples concentrate on showing you how to build contrib style plugin tasks for npm that are entirely self-contained. This is great programming practice, but no good where you need to package a bunch of tasks that rely on each other and config.

      The saviour in this case was my discovery of grunt.config.merge which is a method added recently in grunt 0.4.5. This meant I could create an npm module with related tasks and config, define alias tasks and appropriate contrib config and merge this into the main grunt config in a recursive manner.

      e.g. 

// Contents of package/config-tasks/data.js

module.exports = function (grunt) {

    var taskConfig = {
        clean: {
            data: ...,
        },
        yaml: {
            data: ...
        }
        copy: {
            data: ...
        },
        watch: {
            data: ...
        }
    };

    grunt.registerTask(
        'compile-data',
        'Compile all datafiles',
        ['clean:data', 'yaml:data', 'copy:data']
    );

Combining many of these tasks together into one package, I ended up with a file structure like this:

  • package.json
  • tasks/
    • standaloneTask1.js
    • standaloneTask2.js
    • config-tasks.js
    • config-tasks/
      • data.js
      • build.js
      • compass.js
      • ...

In this example, config-tasks.js is a simple module which calls grunt.loadTasks() on all the files in the config-tasks subdir.

This is then packaged into a module which is included into your project gruntfile in the usual way grunt.loadNpmTasks('package-name');, and you can use your grunt tasks as if they were defined in the project gruntfile as before.

Since the grunt config is merged with that in the main gruntfile, the default task definitions can easily be overridden in a particular project's gruntfile using grunt.config.merge() after importing.

To ensure the presence of required contrib modules used by the package, we utilise the npm peerDependencies functionality in our package.json

The future and issues

I'm reasonably happy with this layout for the moment, but there's a few things I'd like to fix in the fullness of time:

  • Merging the config with the main gruntfile could result in task name conflicts when pulling in multiple packages. perhaps this should be isolated?
  • Much discussion has been made recently surrounding peerDependencies and whether they should be removed from node, as it goes against the established node philosophy. See: https://github.com/npm/npm/issues/5080#issuecomment-40545599
  • I'd like to investigate grunt-submodule further, as this looks like it can provide proper dependency management for contributed grunt packages, and provide task isolation, which could potentially solve these two problems.

Hopefully this somewhat long post has provided some help to a few pople, and offered some ideas for making more maintainable gruntfiles yourselves I'd love to hear any feedback on this approach and other suggestions in the comments below :)