ExpressJS How to structure an application?

I'm using the ExpressJS web framework for NodeJS.

People using ExpressJS put their environments (development, production, test...), their routes etc on the app.js. I think that it's not a beautiful way because when you have a big application, app.js is too big!

I would like to have this directory structure:

| my-application
| -- app.js
| -- config/
     | -- environment.js
     | -- routes.js

Here's my code:

app.js

var express = require('express');
var app = module.exports = express.createServer();

require('./config/environment.js')(app, express);
require('./config/routes.js')(app);

app.listen(3000);

config/environment.js

module.exports = function(app, express){
    app.configure(function() {
    app.use(express.logger());
    });

    app.configure('development', function() {
    app.use(express.errorHandler({
        dumpExceptions: true,
        showStack: true
    }));
    });

    app.configure('production', function() {
    app.use(express.errorHandler());
    });
};

config/routes.js

module.exports = function(app) {
    app.get('/', function(req, res) {
    res.send('Hello world !');
    });
};

My code works well and I think that the structure of the directories is beautiful. However, the code had to be adapted and I'm not sure that it's good/beautiful.

Is it better to use my structure of directories and adapt the code or simply use one file (app.js)?

Thanks for your advices!

Solution 1:

OK, it's been a while and this is a popular question, so I've gone ahead and created a scaffolding github repository with JavaScript code and a long README about how I like to structure a medium-sized express.js application.

focusaurus/express_code_structure is the repo with the latest code for this. Pull requests welcome.

Here's a snapshot of the README since stackoverflow doesn't like just-a-link answers. I'll make some updates as this is a new project that I'll continue updating, but ultimately the github repo will be the up-to-date place for this information.

Express Code Structure

This project is an example of how to organize a medium-sized express.js web application.

Current to at least express v4.14 December 2016

How big is your application?

Web applications are not all the same, and there's not, in my opinion, a single code structure that should be applied to all express.js applications.

If your application is small, you don't need such a deep directory structure as exemplified here. Just keep it simple and stick a handful of .js files in the root of your repository and you're done. Voilà.

If your application is huge, at some point you need to break it up into distinct npm packages. In general the node.js approach seems to favor many small packages, at least for libraries, and you should build your application up by using several npm packages as that starts to make sense and justify the overhead. So as your application grows and some portion of the code becomes clearly reusable outside of your application or is a clear subsystem, move it to it's own git repository and make it into a standalone npm package.

So the focus of this project is to illustrate a workable structure for a medium-sized application.

What is your overall architecture

There are many approaches to building a web application, such as

• Server Side MVC a la Ruby on Rails
• Single Page Application style a la MongoDB/Express/Angular/Node (MEAN)
• Basic web site with some forms
• Models/Operations/Views/Events style a la MVC is dead, it's time to MOVE on
• and many others both current and historical

Each of these fits nicely into a different directory structure. For the purposes of this example, it's just scaffolding and not a fully working app, but I'm assuming the following key architecture points:

• The site has some traditional static pages/templates
• The "application" portion of the site is developed as a Single Page Application style
• The application exposes a REST/JSON style API to the browser
• The app models a simple business domain, in this case, it's a car dealership application

And what about Ruby on Rails?

It will be a theme throughout this project that many of the ideas embodied in Ruby on Rails and the "Convention over Configuration" decisions they have adopted, though widely accepted and used, are not actually very helpful and sometimes are the opposite of what this repository recommends.

My main point here is that there are underlying principles to organizing code, and based on those principles, the Ruby on Rails conventions make sense (mostly) for the Ruby on Rails community. However, just thoughtlessly aping those conventions misses the point. Once you grok the basic principles, ALL of your projects will be well-organized and clear: shell scripts, games, mobile apps, enterprise projects, even your home directory.

For the Rails community, they want to be able to have a single Rails developer switch from app to app to app and be familiar and comfortable with it each time. This makes great sense if you are 37 signals or Pivotal Labs, and has benefits. In the server-side JavaScript world, the overall ethos is just way more wild west anything goes and we don't really have a problem with that. That's how we roll. We're used to it. Even within express.js, it's a close kin of Sinatra, not Rails, and taking conventions from Rails is usually not helping anything. I'd even say Principles over Convention over Configuration.

Underlying Principles and Motivations

• Be mentally manageable
  • The brain can only deal with and think about a small number of related things at once. That's why we use directories. It helps us deal with complexity by focusing on small portions.
• Be size-appropriate
  • Don't create "Mansion Directories" where there's just 1 file all alone 3 directories down. You can see this happening in the Ansible Best Practices that shames small projects into creating 10+ directories to hold 10+ files when 1 directory with 3 files would be much more appropriate. You don't drive a bus to work (unless you're a bus driver, but even then your driving a bus AT work not TO work), so don't create filesystem structures that aren't justified by the actual files inside them.
• Be modular but pragmatic
  • The node community overall favors small modules. Anything that can cleanly be separated out from your app entirely should be extracted into a module either for internal use or publicly published on npm. However, for the medium-sized applications that are the scope here, the overhead of this can add tedium to your workflow without commensurate value. So for the time when you have some code that is factored out but not enough to justify a completely separate npm module, just consider it a "proto-module" with the expectation that when it crosses some size threshold, it would be extracted out.
  • Some folks such as @hij1nx even include an app/node_modules directory and have package.json files in the proto-module directories to facilitate that transition and act as a reminder.
• Be easy to locate code
  • Given a feature to build or a bug to fix, our goal is that a developer has no struggle locating the source files involved.
  • Names are meaningful and accurate
  • crufty code is fully removed, not left around in an orphan file or just commented out
• Be search-friendly
  • all first-party source code is in the app directory so you can cd there are run find/grep/xargs/ag/ack/etc and not be distracted by third party matches
• Use simple and obvious naming
  • npm now seems to require all-lowercase package names. I find this mostly terrible but I must follow the herd, thus filenames should use kebab-case even though the variable name for that in JavaScript must be camelCase because - is a minus sign in JavaScript.
  • variable name matches the basename of the module path, but with kebab-case transformed to camelCase
• Group by Coupling, Not by Function
  • This is a major departure from the Ruby on Rails convention of app/views, app/controllers, app/models, etc
  • Features get added to a full stack, so I want to focus on a full stack of files that are relevant to my feature. When I'm adding a telephone number field to the user model, I don't care about any controller other than the user controller, and I don't care about any model other than the user model.
  • So instead of editing 6 files that are each in their own directory and ignoring tons of other files in those directories, this repository is organized such that all the files I need to build a feature are colocated
  • By the nature of MVC, the user view is coupled to the user controller which is coupled to the user model. So when I change the user model, those 3 files will often change together, but the deals controller or customer controller are decoupled and thus not involved. Same applies to non-MVC designs usually as well.
  • MVC or MOVE style decoupling in terms of which code goes in which module is still encouraged, but spreading the MVC files out into sibling directories is just annoying.
  • Thus each of my routes files has the portion of the routes it owns. A rails-style routes.rb file is handy if you want an overview of all routes in the app, but when actually building features and fixing bugs, you only care about the routes relevant to the piece you are changing.
• Store tests next to the code
  • This is just an instance of "group by coupling", but I wanted to call it out specifically. I've written many projects where the tests live under a parallel filesystem called "tests" and now that I've started putting my tests in the same directory as their corresponding code, I'm never going back. This is more modular and much easier to work with in text editors and alleviates a lot of the "../../.." path nonsense. If you are in doubt, try it on a few projects and decide for yourself. I'm not going to do anything beyond this to convince you that it's better.
• Reduce cross-cutting coupling with Events
  • It's easy to think "OK, whenever a new Deal is created, I want to send an email to all the Salespeople", and then just put the code to send those emails in the route that creates deals.
  • However, this coupling will eventually turn your app into a giant ball of mud.
  • Instead, the DealModel should just fire a "create" event and be entirely unaware of what else the system might do in response to that.
  • When you code this way, it becomes much more possible to put all the user related code into app/users because there's not a rat's nest of coupled business logic all over the place polluting the purity of the user code base.
• Code flow is followable
  • Don't do magic things. Don't autoload files from magic directories in the filesystem. Don't be Rails. The app starts at app/server.js:1 and you can see everything it loads and executes by following the code.
  • Don't make DSLs for your routes. Don't do silly metaprogramming when it is not called for.
  • If your app is so big that doing magicRESTRouter.route(somecontroller, {except: 'POST'}) is a big win for you over 3 basic app.get, app.put, app.del, calls, you're probably building a monolithic app that is too big to effectively work on. Get fancy for BIG wins, not for converting 3 simple lines to 1 complex line.

• Use lower-kebab-case filenames

  • This format avoids filesystem case sensitivity issues across platforms
  • npm forbids uppercase in new package names, and this works well with that

    express.js specifics

• Don't use app.configure. It's almost entirely useless and you just don't need it. It is in lots of boilerplate due to mindless copypasta.

• THE ORDER OF MIDDLEWARE AND ROUTES IN EXPRESS MATTERS!!!
  • Almost every routing problem I see on stackoverflow is out-of-order express middleware
  • In general, you want your routes decoupled and not relying on order that much
  • Don't use app.use for your entire application if you really only need that middleware for 2 routes (I'm looking at you, body-parser)
  • Make sure when all is said and done you have EXACTLY this order:
    1. Any super-important application-wide middleware
    2. All your routes and assorted route middlewares
    3. THEN error handlers
• Sadly, being sinatra-inspired, express.js mostly assumes all your routes will be in server.js and it will be clear how they are ordered. For a medium-sized application, breaking things out into separate routes modules is nice, but it does introduce peril of out-of-order middleware

The app symlink trick

There are many approaches outlined and discussed at length by the community in the great gist Better local require() paths for Node.js. I may soon decide to prefer either "just deal with lots of ../../../.." or use the requireFrom modlue. However, at the moment, I've been using the symlink trick detailed below.

So one way to avoid intra-project requires with annoying relative paths like require("../../../config") is to use the following trick:

• create a symlink under node_modules for your app
  • cd node_modules && ln -nsf ../app
• add just the node_modules/app symlink itself, not the entire node_modules folder, to git
  • git add -f node_modules/app
  • Yes, you should still have "node_modules" in your .gitignore file
  • No, you should not put "node_modules" into your git repository. Some people will recommend you do this. They are incorrect.
• Now you can require intra-project modules using this prefix
  • var config = require("app/config");
  • var DealModel = require("app/deals/deal-model");
• Basically, this makes intra-project requires work very similarly to requires for external npm modules.
• Sorry, Windows users, you need to stick with parent directory relative paths.

Configuration

Generally code modules and classes to expect only a basic JavaScript options object passed in. Only app/server.js should load the app/config.js module. From there it can synthesize small options objects to configure subsystems as needed, but coupling every subsystem to a big global config module full of extra information is bad coupling.

Try to centralize creation of DB connections and pass those into subsystems as opposed to passing connection parameters and having subsystems make outgoing connections themselves.

NODE_ENV

This is another enticing but terrible idea carried over from Rails. There should be exactly 1 place in your app, app/config.js that looks at the NODE_ENV environment variable. Everything else should take an explicit option as a class constructor argument or module configuration parameter.

If the email module has an option as to how to deliver emails (SMTP, log to stdout, put in queue etc), it should take an option like {deliver: 'stdout'} but it should absolutely not check NODE_ENV.

Tests

I now keep my test files in the same directory as their corresponding code and use filename extension naming conventions to distinguish tests from production code.

• foo.js has the module "foo"'s code
• foo.tape.js has the node-based tests for foo and lives in the same dir
• foo.btape.js can be used for tests that need to execute in a browser environment

I use filesystem globs and the find . -name '*.tape.js' command to get access to all my tests as necessary.

How to organize code within each .js module file

This project's scope is mostly about where files and directories go, and I don't want to add much other scope, but I'll just mention that I organize my code into 3 distinct sections.

1. Opening block of CommonJS require calls to state dependencies
2. Main code block of pure-JavaScript. No CommonJS pollution in here. Don't reference exports, module, or require.
3. Closing block of CommonJS to set up exports

Solution 2:

UPDATE (2013-10-29): Please see my other answer as well which has JavaScript instead of CoffeeScript by popular demand as well as a boilerplate github repo and an extensive README detailing my latest recommendations on this topic.

Config

What you are doing is fine. I like to have my own config namespace set up in a top-level config.coffee file with a nested namespace like this.

#Set the current environment to true in the env object
currentEnv = process.env.NODE_ENV or 'development'
exports.appName = "MyApp"
exports.env =
  production: false
  staging: false
  test: false
  development: false
exports.env[currentEnv] = true
exports.log =
  path: __dirname + "/var/log/app_#{currentEnv}.log"
exports.server =
  port: 9600
  #In staging and production, listen loopback. nginx listens on the network.
  ip: '127.0.0.1'
if currentEnv not in ['production', 'staging']
  exports.enableTests = true
  #Listen on all IPs in dev/test (for testing from other machines)
  exports.server.ip = '0.0.0.0'
exports.db =
  URL: "mongodb://localhost:27017/#{exports.appName.toLowerCase()}_#{currentEnv}"

This is friendly for sysadmin editing. Then when I need something, like the DB connection info, it`s

require('./config').db.URL

Routes/Controllers

I like to leave my routes with my controllers and organize them in an app/controllers subdirectory. Then I can load them up and let them add whatever routes they need.

In my app/server.coffee coffeescript file I do:

[
  'api'
  'authorization'
  'authentication'
  'domains'
  'users'
  'stylesheets'
  'javascripts'
  'tests'
  'sales'
].map (controllerName) ->
  controller = require './controllers/' + controllerName
  controller.setup app

So I have files like:

app/controllers/api.coffee
app/controllers/authorization.coffee
app/controllers/authentication.coffee
app/controllers/domains.coffee

And for example in my domains controller, I have a setup function like this.

exports.setup = (app) ->
  controller = new exports.DomainController
  route = '/domains'
  app.post route, controller.create
  app.put route, api.needId
  app.delete route, api.needId
  route = '/domains/:id'
  app.put route, controller.loadDomain, controller.update
  app.del route, controller.loadDomain, exports.delete
  app.get route, controller.loadDomain, (req, res) ->
    res.sendJSON req.domain, status.OK

Views

Putting views in app/views is becoming the customary place. I lay it out like this.

app/views/layout.jade
app/views/about.jade
app/views/user/EditUser.jade
app/views/domain/EditDomain.jade

Static Files

Go in a public subdirectory.

Github/Semver/NPM

Put a README.md markdown file at your git repo root for github.

Put a package.json file with a semantic version number in your git repo root for NPM.

Solution 3:

The following is Peter Lyons' answer verbatim, ported over to vanilla JS from Coffeescript, as requested by several others. Peter's answer is very able, and anyone voting on my answer should vote on his as well.

Config

What you are doing is fine. I like to have my own config namespace set up in a top-level config.js file with a nested namespace like this.

// Set the current environment to true in the env object
var currentEnv = process.env.NODE_ENV || 'development';
exports.appName = "MyApp";
exports.env = {
  production: false,
  staging: false,
  test: false,
  development: false
};  
exports.env[currentEnv] = true;
exports.log = {
  path: __dirname + "/var/log/app_#{currentEnv}.log"
};  
exports.server = {
  port: 9600,
  // In staging and production, listen loopback. nginx listens on the network.
  ip: '127.0.0.1'
};  
if (currentEnv != 'production' && currentEnv != 'staging') {
  exports.enableTests = true;
  // Listen on all IPs in dev/test (for testing from other machines)
  exports.server.ip = '0.0.0.0';
};
exports.db {
  URL: "mongodb://localhost:27017/#{exports.appName.toLowerCase()}_#{currentEnv}"
};

This is friendly for sysadmin editing. Then when I need something, like the DB connection info, it`s

require('./config').db.URL

Routes/Controllers

I like to leave my routes with my controllers and organize them in an app/controllers subdirectory. Then I can load them up and let them add whatever routes they need.

In my app/server.js javascript file I do:

[
  'api',
  'authorization',
  'authentication',
  'domains',
  'users',
  'stylesheets',
  'javascripts',
  'tests',
  'sales'
].map(function(controllerName){
  var controller = require('./controllers/' + controllerName);
  controller.setup(app);
});

So I have files like:

app/controllers/api.js
app/controllers/authorization.js
app/controllers/authentication.js
app/controllers/domains.js

And for example in my domains controller, I have a setup function like this.

exports.setup = function(app) {
  var controller = new exports.DomainController();
  var route = '/domains';
  app.post(route, controller.create);
  app.put(route, api.needId);
  app.delete(route, api.needId);
  route = '/domains/:id';
  app.put(route, controller.loadDomain, controller.update);
  app.del(route, controller.loadDomain, function(req, res){
    res.sendJSON(req.domain, status.OK);
  });
}

Views

Putting views in app/views is becoming the customary place. I lay it out like this.

app/views/layout.jade
app/views/about.jade
app/views/user/EditUser.jade
app/views/domain/EditDomain.jade

Static Files

Go in a public subdirectory.

Github/Semver/NPM

Put a README.md markdown file at your git repo root for github.

Put a package.json file with a semantic version number in your git repo root for NPM.

Solution 4:

My question was introduced in April 2011, it's quiet old. During this time, I could improve my experience with Express.js and how to architecture an application written using this library. So, I share here my experience.

Here's my directory structure:

├── app.js   // main entry
├── config   // The configuration of my applications (logger, global config, ...)
├── models   // The model data (e.g. Mongoose model)
├── public   // The public directory (client-side code)
├── routes   // The route definitions and implementations
├── services // The standalone services (Database service, Email service, ...)
└── views    // The view rendered by the server to the client (e.g. Jade, EJS, ...)

App.js

The goal of the app.js file is to bootstrap the expressjs application. It loads the configuration module, the logger module, wait for database connection, ..., and run the express server.

'use strict';
require('./config');
var database = require('./services/database');
var express = require('express');
var app = express();
module.exports = app;

function main() {
  var http = require('http');

  // Configure the application.
  app.configure(function () {
    // ... ... ...
  });
  app.configure('production', function () {
    // ... ... ...
  });
  app.configure('development', function () {
    // ... ... ...
  });

  var server = http.createServer(app);

  // Load all routes.
  require('./routes')(app);

  // Listen on http port.
  server.listen(3000);
}

database.connect(function (err) {
  if (err) { 
    // ...
  }
  main();
});

routes/

The routes directory has a index.js file. Its goal is to introduce a kind of magic to load all other files inside the routes/ directory. Here's the implementation:

/**
 * This module loads dynamically all routes modules located in the routes/
 * directory.
 */
'use strict';
var fs = require('fs');
var path = require('path');

module.exports = function (app) {
  fs.readdirSync('./routes').forEach(function (file) {
    // Avoid to read this current file.
    if (file === path.basename(__filename)) { return; }

    // Load the route file.
    require('./' + file)(app);
  });
};

With that module, creating a new route definition and implementation is really easy. For examples, hello.js:

function hello(req, res) {
  res.send('Hello world');
}

module.exports = function (app) {
  app.get('/api/hello_world', hello);
};

Each route module is standalone.

Solution 5:

I like to use a global "app", rather than exporting a function etc