Running Actionhero

The ActionHero Binary

ActionHero - A multi-transport node.js API Server with integrated cluster
  capabilities and delayed tasks

Binary options:
* help (default)
* start
* start cluster
* generate
* generate action
* generate task
* generate initializer
* generate server
* actions list
* tasks enqueue
* console
* link
* unlink

Descriptions:

* actionhero help
  will display this document

* actionhero version
  will display the version of ActionHero in use

* actionhero start --config=[/path/to/config] --title=[processTitle]
    --daemon
  [config] (optional) path to config.js, defaults to "process.cwd() + '/'
    + config.js". You can also use ENV[ACTIONHERO_CONFIG].
  [title] (optional) process title to use for ActionHeros ID, ps, log, and
    pidFile defaults. Must be unique for each member of the cluster.
    You can also use ENV[ACTIONHERO_TITLE].
    Process renaming does not work on OSX/Windows
  [daemon] (optional) to fork and run as a new background process defaults
    to false

* actionhero start cluster --workers=[numWorkers] --workerTitlePrefix=[title]
    --daemon
  will launch a ActionHero cluster (using node-s cluster module)
  [workers] (optional) number of workers (defaults to # CPUs - 2)
  [title] (optional) worker title prefix (default 'actionhero-worker-')
    set `--workerTitlePrefix=hostname`, your app.id would be like
    `your_host_name-#`
  [daemon] (optional) to fork and run as a new background process defaults
    to false

* actionhero generate
  will prepare an empty directory with a template ActionHero project

* actionhero generate action --name=[name] --description=[description]
  will generate a new action in "actions"
  [name] (required)
  [description] (optional)

* actionhero generate task --name=[name] --description=[description]
    --scope=[scope] --frequency=[frequency]
  will generate a new task in "tasks"
  [name] (required)
  [description] (optional)
  [queue] (required)
  [frequency] (optional)

* actionhero generate initializer --name=[name] --loadPriority=[p]
    --startPriority=[p] --stopPriority=[p]
  will generate a new initializer in "initializers"
  [name] (required)
  [loadPriority] (optional)
  [startPriority] (optional)
  [stopPriority] (optional)

* actionhero generate server --name=[name]
  will generate a new server in "servers"
  [name] (required)

* actionhero actions list
  will list all actions in this server to stdout

* actionhero task enqueue --name=[taskName] --args=[JSON-formatted args]
  will enqueue a task into redis

* actionhero console
  will open an interactive CLI with the API object in scope.
  this is sometimes called a REPL

* actionhero link --name=[pluginName] --overwriteConfig=[overwriteConfig]
  will link the actions, tasks, initializers, etc from a plugin into your
    top-level project normally, you will have first installed the plugin
    via `npm install myPlugin`
  [name] (required)
  [overwriteConfig] (optional) default: false

* actionhero unlink --name=[pluginName]
  will remove the linked actions, tasks, initializers, etc from a plugin in your
    top-level project. Please remove the config files manually
  Remember if your plugin was installed via NPM, also be sure to remove it from your
    package.json or uninstall it with npm uninstall --save
  [name] (required)

#############################################################
## More Help & the ActionHero documentation can be found @ ##
##             http://www.actionherojs.com                 ##
#############################################################

The suggested method to run your ActionHero server is to use the included ./node_modules/.bin/actionhero binary. Note that there is no main.js or specific start script your project needs. ActionHero handles this for you. Your ActionHero project simply needs to follow the proper directory conventions and it will be bootable.

At the time of this writing the ActionHero binary's help contains:

Linking the ActionHero binary

  • If you installed ActionHero globally (npm install actionhero -g) you should have the actionhero binary available to you within your shell at all times.
  • Otherwise, you can reference the binary from either ./node_modules/.bin/actionhero or ./node_modules/actionhero/bin/actionhero.
  • If you installed ActionHero locally, you can add a reference to your path (OSX and Linux): export PATH=$PATH:node_modules/.bin to be able to use simpler commands, IE actionhero start. On windows this can be done by running set PATH=%PATH%;%cd%\node_modules\.bin at command prompt (not powershell).

Environments and Config

By default, ActionHero will use the settings found in the exports.default blocks in /config/. However, you can set environment-specfic overrides or changes. ActionHero inspects process.env.NODE_ENV to load up runtime configuration overrides from exports.#{env} blocks in your configuration files. This is the recommended way to have separate settings for staging and production.

The load order of configs is:

  • default values in /config
  • environment-specific values in /config
  • options passed in to boot with actionhero.start({configChanges: configChanges}, callback)

Programatic Use of ActionHero

var actionheroPrototype = require("actionhero");
var actionhero = new actionheroPrototype();

var timer = 5000;
actionhero.start(params, function(error, api){

  api.log(" >> Boot Successful!");
  setTimeout(function(){

    api.log(" >> restarting server...");
    actionhero.restart(function(error, api){

      api.log(" >> Restarted!");
      setTimeout(function(){

        api.log(" >> stopping server...");
        actionhero.stop(function(error, api){

          api.log(" >> Stopped!");
          process.exit();

        });
      }, timer);
    })
  }, timer);
});

While NOT encouraged, you can always instantiate an ActionHero server yourself. Perhaps you wish to combine ActionHero with an existing project. Here's how! Take note that using these methods will not work for actionCluster, and only a single instance will be started within your project.

Feel free to look at the source of ./node_modules/actionhero/bin/include/start to see how the main ActionHero server is implemented for more information.

You can programmatically control an ActionHero server with actionhero.start(params, callback), actionhero.stop(callback) and actionhero.restart(callback)

From within ActionHero itself (actions, initilizers, etc), you can use api.commands.start, api.commands.stop, and api.commands.restart to control the server.

Signals

> ./node_modules/.bin/actionhero start cluster --workers=2
info: actionhero >> start cluster
notice:  - STARTING CLUSTER -
notice: pid: 41382
info: starting worker #1
info: worker 41383 (#1) has spawned
info: Worker #1 [41383]: starting
info: Worker #1 [41383]: started
info: starting worker #2
info: worker 41384 (#2) has spawned
info: Worker #2 [41384]: starting
info: Worker #2 [41384]: started

# A new terminal
kill -s TTIN `cat pids/cluster_pidfile`

info: worker 41632 (#3) has spawned
info: Worker #3 [41632]: starting
info: Worker #3 [41632]: started

# A new terminal
kill -s KILL `cat pids/cluster_pidfile`

warning: Cluster manager quitting
info: Stopping each worker...
info: Worker #1 [41901]: stopping
info: Worker #2 [41904]: stopping
info: Worker #3 [41906]: stopping
info: Worker #3 [41906]: stopped
info: Worker #2 [41904]: stopped
info: Worker #1 [41901]: stopped
alert: worker 41901 (#1) has exited
alert: worker 41904 (#2) has exited
alert: worker 41906 (#3) has exited
info: all workers gone
notice: cluster complete, Bye!

ActionHero is intended to be run on *nix operating systems. The start and start cluster commands provide support for signaling. (There is limited support for some of these commands in windows).

actionhero start

  • kill / term / int : Process will attempt to "gracefully" shut down. That is, the worker will close all server connections (possibly sending a shutdown message to clients, depending on server type), stop all task workers, and eventually shut down. This action may take some time to fully complete.
  • USR2: Process will restart itself. The process will preform the "graceful shutdown" above, and they restart.

actionhero start cluster

All signals should be sent to the cluster master process. You can still signal the termination of a worker, but the cluster manager will start a new one in its place.

  • kill / term / int: Will signal the master to "gracefully terminate" all workers. Master will terminate once all workers have completed
  • HUP : Restart all workers.
  • USR2 : "Hot reload". Worker will kill off existing workers one-by-one, and start a new worker in their place. This is used for 0-downtime restarts. Keep in mind that for a short while, your server will be running both old and new code while the workers are rolling.
  • TTOU: remove one worker
  • TTIN: add one worker

Shutting Down

When using actionhero start or actionhero start cluster, when you signal ActionHero to stop via the signals above (or from within your running application via api.commands.stop()), actionhero will attempt to gracefully shutdown. This will include running any initializer's stop() method. This will close any open servers, and attempt to allow any running tasks to complete.

Because things sometimes go wrong, actionhero start and actionhero start cluster also have a "emergency stop" timeout. This defaults to 30 seconds, and is configurable via the ACTIONHERO_SHUTDOWN_TIMEOUT environment variable. Be sure that your tasks and actions can complete within that window, or else raise that shutdown limit.

Windows-Specific Notes

  • Sometimes ActionHero may require a git-based module (rather than a NPM module). You will need to have git installed. Depending on how you installed git, it may not be available to the node shell. Be sure to have also installed references to git. You can also run node/npm install from the git shell.
  • Sometimes, npm will not install the actionhero binary @ /node_modules/.bin/actionhero, but rather it will attempt to create a windows executable and wrapper. You can launch ActionHero directly with ./node_modules/actionhero/bin/actionhero

Development Mode

Overview

config.general = {
  developmentMode: true
}
  

ActionHero's development mode is a little different than tools like nodemon in that it tries hard not to restart the server process. Changes to routes, tasks, and actions can simply replace those in memory when they are updated on disk. Other changes, like changes to api.config or initializers are more severe, and will restart the whole application (much like nodemon).

To enable development mode simply set developmentMode: true in your config/api.js.

Note that api.config.general.developmentMode is different from NODE_ENV, which by default is "development" (and is logged out when ActionHero boots). NODE_ENV is used to determine which config settings to use, and has no effect on developmentMode.

Effects of Development Mode

Development mode, when enabled, will poll for changes in your actions, tasks and initializers, and reload them on the fly.

  • this uses fs.watchFile() and may not work on all OSs / file systems.
  • new files won't be loaded in, only existing files when the app was booted will be monitored
  • as deleting a file might crash your application, we will not attempt to re-load deleted files
  • if you have changed the task.frequency of a periodic task, you will continue to use the old value until the task fires at least once after the change
  • changing api.config, initializers, or servers, will attempt to do a "full" reboot the server rather than just reload that component.

Watching custom files

api.watchFileAndAct(path_to_file, function(){
  api.log('rebooting due to config change: ' + path_to_file, info');
  api.commands.restart();
});
  

You can use ActionHero's watchFileAndAct() method to watch additional files your application may have:

Debugging

You can use the awesome node-inspector project to help you debug your ActionHero application within the familiar Chrome Browser's developer tools.

Be sure to run ActionHero with node's --debug flag, ie: node ./node_modules/.bin/actionhero start --debug

// in package.json
"dependencies": {
  "actionhero": "x",
  "node-inspector": "x"
},
  

Start up node-inspector (both node-inspector and ActionHero have the same default port, so you will need to change one of them) ./node_modules/.bin/node-inspector --web-port=1234

That's it! Now you can visit http://0.0.0.0:1234/debug?port=5858 and start debugging. Remember that the way node-debugger works has you first set a breakpoint in the file view, and then you can use the console to inspect various objects. IE: I put a breakpoint in the default status action in the run method:

REPL

actionhero console

Running "console" task
2015-11-14 17:48:01 - notice: *** starting actionhero ***
2015-11-14 17:48:01 - warning: running with fakeredis
2015-11-14 17:48:01 - info: actionhero member 10.0.1.15 has joined the cluster
2015-11-14 17:48:01 - notice: pid: 38464
2015-11-14 17:48:01 - notice: server ID: 10.0.1.15
2015-11-14 17:48:01 - info: ensuring the existence of the chatRoom: defaultRoom
2015-11-14 17:48:01 - info: ensuring the existence of the chatRoom: anotherRoom
2015-11-14 17:48:01 - notice: environment: development
2015-11-14 17:48:01 - notice: *** Server Started @ 2015-11-14 17:48:01 ***
[ AH::development ] > api.id
‘10.0.1.15'

[ AH::development ] > Object.keys(api.actions.actions)
[ ‘cacheTest',
‘randomNumber',
‘showDocumentation',
‘sleepTest',
‘status' ]
  

ActionHero now has a REPL (v9.0.0)! This means you can ‘connect' to a running instance of ActionHero and manually call all the methods on the api namespace. This combined with the new RPC tools make this a powerful debugging and development tool. Running ActionHero console will load up a version of action hero in your terminal where you have access to the api object. This version of the server will boot, initialize, and start, but will skip booting any servers.

The REPL will:

  • source NODE_ENV properly to load the config
  • will connect to redis and load any user-defined initializers
  • will load any plugins
  • will not boot any servers

If you are familiar with rails, this is very similar to rails console

Testing

ActionHero provides test helpers so that you may try your actions and tasks within a headless environment. We do this by including a specHelper initializer which creates a server, testServer when running within the test environment. Via the testServer, you can easily call actions or tasks without making a real request.

We have chosen mocha as our test framework and chai as our assertion tool which are included as dependencies within all new projects (generated with ./node_modules/.bin/actionhero generate). We also use cross-env to set NODE_ENV in a way that works for all operating systems, including Windows. You do not need to use these testing tools, but an example will be provided which makes use of them.

You also don't need to use these test helpers, and you may want to make a ‘real' http or websocket request to test something specific. If this is the case, you can check out how ActionHero tests its own servers for examples.

Getting Started

// package.json from a new actionhero project with `mocha` and `chai` included
{
  "author"      : "YOU <[email protected]>",
  "name"        : "my_actionhero_project",
  "description" : "my actionhero project",
  "version"     : "0.0.1",
  "engines"     : {
    "node": ">=0.10.0"
  },
  "dependencies" : {
    "actionhero" : "12.3.0",
    "ws"         : "latest"
  },
  "devDependencies" : {
    "cross-env": "latest",
    "mocha"  : "latest",
    "chai" : "latest"
  },
  "scripts" : {
    "help"         : "actionhero help",
    "start"        : "actionhero start",
    "actionhero"   : "actionhero",
    "start cluster": "actionhero start cluster",
    "test"         : "cross-env NODE_ENV=test mocha"
  }
}
  

To run a mocha test suite, you invoke the mocha binary, ./node_modules/.bin/mocha. This will tell mocha to look in your ./test folder and run any tests that it can find. There are ways to change the test folder location, only run specific tests, change the reporting format and more which you can learn about on Mocha's website. We asume that you have mocha (and chai) installed to your project by listing it in your package.json. If you used ActionHero generate to create your project, this should already be configured for your.

The majority of the time, you'll be testing actions and other methods you have written, so you'll need to "run" an actionhero server as part of your test suite. Many times you'll want to have ActionHero behave in a slightly unique way while testing (perhaps connect to a special database, don't log, etc). To do this, you can change the behavior of the config files for the test environment. Here's how we tell ActionHero not to write any logs when testing. Note thest test-specific configuration overrides the defaults. To ensure that ActionHero boots with the test environment loaded, the test command you run should explicitly do this, AKA: NODE_ENV=test ./node_modules/.bin/mocha. You can log this in as the test script in your package.json so you can simplify the running of tests with just npm test.

ActionHero comes with a specHelper to make it easier to test tasks and actions. This specHelper is a special server which can check things without needing to make an HTTP, websocket, etc request. If you need to check the true behavior of a server (perhaps how the router works for an HTTP request), you should make a real HTTP request in your test suite, using something like the request library (example).

Example Test

'use strict'

let path = require('path')
var expect = require('chai').expect
var ActionheroPrototype = require(path.join(__dirname, '/../../actionhero.js'))
var actionhero = new ActionheroPrototype()
var api

describe('Action: RandomNumber', () => {
  before((done) => {
    actionhero.start((error, a) => {
      expect(error).to.be.null()
      api = a
      done()
    })
  })

  after((done) => {
    actionhero.stop(() => {
      done()
    })
  })

  var firstNumber = null
  it('generates random numbers', (done) => {
    api.specHelper.runAction('randomNumber', (response) => {
      expect(response.randomNumber).to.be.at.least(0)
      expect(response.randomNumber).to.be.at.most(1)
      firstNumber = response.randomNumber
      done()
    })
  })

  it('is unique / random', (done) => {
    api.specHelper.runAction('randomNumber', (response) => {
      expect(response.randomNumber).to.be.at.least(0)
      expect(response.randomNumber).to.be.at.most(1)
      expect(response.randomNumber).not.to.equal(firstNumber)
      done()
    })
  })
})
  

Say you had an action that was supposed to respond with a randomNumber, and you wanted to write a test for it.

More details on the specHelper methods are below.

If you want to see fuller example of how to create an integration test within ActionHero, please check out the tutorial

Test Methods

new api.specHelper.connection()

  • generate a new connection object for the testServer
  • this connection can run actions, chat, etc.
  • connection.messages will contain all messages the connection has been sent (welcome messages, action responses, say messages, etc)

api.specHelper.runAction(actionName, input, callback)

  • use this method to run an action
  • input can be either a api.specHelper.connection object, or simply a hash of params, IE: {key: 'value'}
  • the callback returns message and connection.
  • example use:
api.specHelper.runAction(cacheTest', {key: ‘key', value: value'}, function(message, connection){
// message is the normal API response;
// connection is a new connection object
})

api.specHelper.getStaticFile(file, callback)

  • request a file in /public from the server
  • the callback returns message and connection where message is a hash:
var message = {
error    : error,    // null if everything is OK
content  : (string), // string representation of the file's body
mime     : mime,     // file mime
length   : length    // bytes
}

api.specHelper.runTask(taskName, params, callback)

  • callback may or may not return anything depending on your task's makeup
api.specHelper.runTask(sendEmailTask', {message: ‘hello' to: evan@test.com'}, function(response){
// test it!
// remember that the task really will be run, so be sure that the test environment is set properly
})

Notes

Be sure to run your tests in the test environment, setting the shell's env with NODE_ENV=test. You can alternatively set this explicitly in your tests with process.env.NODE_ENV = 'test'

If you do not want the specHelper actions to include metadata (data.response.serverInformation, data.response.requesterInformation, and data.response.messageCount) from the server, you can configure api.specHelper.returnMetadata = false in your tests.

Production Notes

Topology Example

// Assume we use the flag `process.env.ACTIONHERO_ROLE` to denote the type of server
// You can set this variable in the ENV of your server or launch each process with the flag:
// Worker => `ACTIONHERO_ROLE='worker' npm start`
// Server => `ACTIONHERO_ROLE='server' npm start`

// config/tasks.js

exports.production = {
    tasks: function(api){

        // default to config for 'server'
        var config = {
          scheduler: false,
          queues: ['*'],
          verbose: true,
          // ...
        };

        if(process.env.ACTIONHERO_ROLE === 'worker'){
            config.scheduler = true;
            config.minTaskProcessors = 1;
            config.maxTaskProcessors = 10;
        }

        return config;
    }
};

// config/servers/web.js

exports.default = {
    servers: {
        web: function(api){
            config = {
                enabled: true,
                secure: false,
                serverOptions: {},
                port: process.env.PORT || 8080
                // ...
            };

            if(process.env.ACTIONHERO_ROLE === 'worker'){
                config.enabled = false;
            }

            return config;
        }
    }
};
  

Here is a common ActionHero production topology:

cluster

Notes:

  • It's best to seperate the "workers" from the web "servers"
    • be sure to modify the config files for each type of server accordingly (ie: turn of all servers for the workers, and turn of all workers on the servers)
  • To accomplish the above, you only need to make changes to your configuration files on each server. You will still be running the same same ActionHero project codebase. See the example:
  • Always have a replica of redis!

Paths and Environments

You can set a few environment variables to affect how ActionHero runs:

  • PROJECT_ROOT: This is useful when deploying ActionHero applications on a server where symlinks will change under a running process. The cluster will look at your symlink PROJECT_ROOT=/path/to/current_symlink rather than the absolute path it was started from
  • ACTIONHERO_ROOT: This can used to set the absolute path to the ActionHero binaries
  • ACTIONHERO_CONFIG: This can be user to set the absolute path to the ActionHero config directory you wish to use. This is useful when you might have a variable configs per server
  • ACTIONHERO_TITLE: The value of api.id, and the name for the pidfile in some boot configurations

Daemon

When deploying ActionHero, you will probably have more than 1 process. You can use the cluster manager to keep an eye on the workers and manage them

  • Start the cluster with 2 workers: ./node_modules/.bin/actionhero start cluster --workers=2

When deploying new code, you can gracefully restart your workers by sending the USR2 signal to the cluster manager to signal a reload to all workers. You don't need to start and stop the cluster-master. This allows for 0-downtime deployments.

You may want to set some of the ENV variables above to help with your deployment.

Number of workers

When choosing the number of workers (--workers=n) for your ActionHero cluster, choose at least 1 less than the number of CPUs available. If you have a "burstable" architecture (like a Joyent smart machine), opt for the highest number of ‘consistent' CPUs you can have, meaning a number of CPUs that you will always have available to you.

You never want more workers than you can run at a time, or else you will actually be slowing down the execution of all processes.

Of course, not going in to swap memory is more important than utilizing all of your CPUs, so if you find yourself running out of ram, reduce the number of workers!

Pidfiles

ActionHero will write its pid to a pidfile in the normal unix way. The path for the pidfile is set in config/api.js with config.general.paths.pid.

Individual ActionHero servers will name their pidfiles by api.id, which is determined by the logic here and here. For example, on my laptop with the IP address of 192.168.0.1, running npm start would run one ActionHero server and generate a pidfile of ./pids/actionhero-192.168.0.1 in which would be a single line containg the process' pid.

When running the cluster, the cluster process first writes his own pidfile to process.cwd() + './pids/cluster_pidfile'. Then, every worker the cluster master creates will have a pid like actionhero-worker-1 in the location defined by config/api.js.

Git-Based Deployment

#!/usr/bin/env bash
# assuming the ActionHero cluster master process is already running

DEPLOY_PATH=/path/to/your/application

cd $DEPLOY_PATH && git pull
cd $DEPLOY_PATH && npm install
# run any build tasks here, like perhaps an asset compile step or a database migration
cd $DEPLOY_PATH && kill -s USR2 `cat pids/cluster_pidfile`
  

To send a signal to the cluster master process to reboot all its workers (USR2), you can cat the pidfile (bash): kill -s USR2 "cat /path/to/pids/cluster_pidfile"

If you want to setup a git-based deployment, the simplest steps would be something like =>

PAAS and Procfile Deployment

When deploying to a Platform as a Service (PAAS) cluster (like Heroku, Flynn, and even some Docker deployments), we can offer a few pieces of advice:

If you are deploying a seperate WEB and WORKER process type, you can define them in a Procfile and make use of environment variable overrides in addition to those defined from the environemnt. You can modify your config files to use these options:

# ./Procfile
web:    SCHEDULER=false \
        MIN_TASK_PROCESSORS=0 \
        MAX_TASK_PROCESSORS=0 \
        ENABLE_WEB_SERVER=true  \
        ENABLE_TCP_SERVER=true  \
        ENABLE_WEBSOCKET_SERVER=true  \
        ./node_modules/.bin/actionhero start

worker: SCHEDULER=true  \
        MIN_TASK_PROCESSORS=5 \
        MAX_TASK_PROCESSORS=5 \
        ENABLE_WEB_SERVER=false \
        ENABLE_TCP_SERVER=false \
        ENABLE_WEBSOCKET_SERVER=false \
        ./node_modules/.bin/actionhero start
  

Be sure not to use NPM in your Procfile defintions. In many deployment scenarios, NPM will not properly pass signals to the ActionHero process and it will be impossible to signal a graceful shutdown. Examples of this behavior can be found here and here

Global Packages

It's probably best to avoid installing any global packages. This way, you won't have to worry about conflicts, and your project can be kept up to date more easily. When using npm to install a local package the package's binaries are always copied into ./node_modules/.bin.

You can add local references to your $PATH like so to use these local binaries:

export PATH=$PATH:node_modules/.bin

Nginx Example

// From `config/servers/web.js`

exports.production = {
  servers: {
    web: function(api){
      return {
        port: '/home/USER/www/APP/current/tmp/sockets/actionhero.sock',
        bindIP: null,
        metadataOptions: {
          serverInformation: false,
          requesterInformation: false
        }
      }
    }
  }
}
  
# The nginx.conf:

#user  nobody;
worker_processes  4;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
  worker_connections 1024;
  accept_mutex on;
}


http {
    include       mime.types;
    default_type  application/octet-stream;
    server_tokens off;
    sendfile        on;
    keepalive_timeout  65;

    set_real_ip_from  X.X.X.X/24;
    real_ip_header    X-Forwarded-For;

    gzip on;
    gzip_http_version 1.0;
    gzip_comp_level 9;
    gzip_proxied any;
    gzip_types text/plain text/xml text/css text/comma-separated-values text/javascript application/x-javascript font/ttf font/otf image/svg+xml application/atom+xml;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" $request_time';

    server {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X_FORWARDED_PROTO https;
        proxy_redirect off;

        listen       80;
        server_name  _;

        access_log  /var/log/nginx/access.log  main;
        error_log   /var/log/nginx/error.log;

        root        /home/XXUSERXX/XXAPPLICATIONXX/www/current/public/;
        try_files /$uri/index.html /cache/$uri/index.html /$uri.html /cache/$uri.html /$uri /cache/$uri @app;

        client_max_body_size 50M;

        location /primus {
            proxy_http_version 1.1;
            proxy_buffering off;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_set_header Host $host;

            proxy_pass http://unix:/home/XXUSERXX/www/XXAPPLICATIONXX/shared/tmp/sockets/actionhero.sock;
        }

        location / {
            proxy_http_version 1.1;
            proxy_buffering off;
            proxy_cache_bypass $http_pragma $http_authorization;
            proxy_no_cache $http_pragma $http_authorization;

            proxy_pass http://unix:/home/XXUSERXX/www/XXAPPLICATIONXX/shared/tmp/sockets/actionhero.sock;
        }
    }

}
  

While ActionHero can be the font-line server your users hit, it's probably best to proxy ActionHero behind a load balancer, nginx, haproxy, etc. This will help you pool connections before hitting node, SSL terminate, serve static assets, etc.

Here is an example nginx config for interfacing with ActionHero, including using sockets (not http) and handing the websocket upgrade path.

  • Note the proxy-pass format to the socket: proxy_pass http://unix:/path/to/socket
  • Note some of the extra work you need to have for the websocket upgrade headers (the primus directive)

Redis High-Availability

Redis is technically optional in ActionHero environments, but you will need it if you want to coordinates tasks across a cluster of workers, handle group chat mechanics between WebSocket clients, or do other cross-cluster operations. In those cases, you'll want your Redis setup to be reliable. There are 2 methods to achieving HA redis: Sentinels and Cluster. A simple architectural wireframe of how to deploy the various options is below The ioredis node package supports both of these connection schemes, and all you need to change is your connection options.

Sentinel Mode

In Sentinel mode, you have your Redis configured in a normal master->slave configuration. However, rather than hard-code your application to know who the master and slaves are, your application connects to the Sentinel processes instead. These Sentinels transparently pipeline your connection to the proper Redis master, and they do this invisibly to ActionHero / your application.

The biggest advantage to this configuration is high-availability. In the event of a master failure, the Sentinel processes reach a consensus, then elect a new master automatically. Since the same process which handles master election also manages the client connections, no requests are lost - the sentinels hold the connection idle and then replay any pending requests on the new master after election. In the configuration shown in the first diagram above, up to 2 Redis data nodes and any 1 Sentinel can fail without the entire system failing.

Note that it is not necessary to run the Sentinel nodes on separate servers. They can be run as parallel processes on the Redis nodes themselves.

To run this configuration, configure ioredis with a list of the Sentinel nodes and the name of the cluster. The driver will automatically connect to an appropriate Sentinel in round-robin fashion, reconnecting to another node if one is down, or fails.

An example of a redis.js config file for sentinels would be:

exports.production = {
  redis: function(api){
    return {
      channel: 'actionhero-myApp',
      rpcTimeout: 5000,

      pkg: 'ioredis',
      port: null,
      host: null,
      password: 'redis-password',
      database: 0,

      options: {
        name: 'myCluster',
        password: 'redis-password',
        db: 0,
        sentinels: [
          { host: '1.2.3.4', port: 26379 },
        ]
      }
    }
  }
}
  

Cluster Mode

In Cluster mode, Redis shards all the keys in data into "slots" which are evenly allocated though all the masters in the cluster. The client can connect to any node in the cluster, and if the requested key belongs on another node, it will proxy the request for you (just like the Sentinel would). The cluster can also take care of master re-election for each shard in the event of a master node failure.

Cluster mode provides similar high-availability to Sentinel mode, but the sharding allows more data to be stored in the cluster overall. However, where Sentinel mode requires a minimum of 3 servers, Cluster mode requires a minimum of 6 to reach a quorom and provide full redundancy.

Also an important note: while you may opt to run "sentinel processes", it's the same codebase as regular redis, just running in "sentinel mode". The same goes if you run redis in "cluster mode".

An example of a redis.js config file for redis cluster would be:

// TODO

Best Practices

As ActionHero is a framework, much of the work for keeping your application secure is dependent on the types of actions and tasks you create. That said, here is a list of general best-practices for ensuring your deployment is as robust as it can be:

General Configuration

  • Be sure to change api.config.general.serverToken to something unique for your application
  • Turn off developer mode in production.
  • Use api.config.general.filteredParams to hide sensitive information from the logs. You probably don't want to log out password, credit_card, and other things of that nature.

Topology

  • Run a cluster via start cluster. This will guarantee that you can reboot your application with 0 downtime and deploy new versions without interruption.
    • You can run 1 ActionHero instance per core (assuming the server is dedicated to ActionHero), and that is the default behavior of start cluster.
    • You don't need a tool like PM2 to manage ActionHero cluster process, but you can.
    • You can use an init script to start cluster at boot, or use a tool like monit to do it for you.
  • Never run tasks on the same ActionHero instances you run your servers on; never run your servers on the same ActionHero instances you run your tasks on
    • Yes, under most situations running servers + tasks on the same instance will work OK, but the load profiles (and often the types of packages required) vary in each deployment. Actions are designed to respond quickly and offload hard computations to tasks. Tasks are designed to work slower computations.
    • Do any CPU-intensive work in a task. If a client needs to see the result of a CPU-intensive operation, poll for it (or use web-sockets)
  • Use a centralized logging tool like Splunk, ELK, SumoLogic, etc. ActionHero is built for the cloud, which means that it expects pids, application names, etc to change, and as such, will create many log files. Use a centralized tool to inspect the state of your application.
    • Log everything. You never know what you might want to check up on. ActionHero's logger has various levels you can use for this.
  • Split out the redis instance you use for cache from the one you use for tasks. If your cache fills up, do you want task processing to fail?
  • Your web request stack should look like: [Load Balancer] -> [App Server] -> [Nginx] -> [ActionHero]
    • This layout allows you to have control, back-pressure and throttling at many layers.
    • Configure Nginx to serve static files whenever possible to remove load from ActionHero, and leave it just to process actions
  • Use a CDN. ActionHero will serve static files with the proper last-modified headers, so your CDN should respect this, and you should not need to worry about asset SHAs/Checksums.
  • Use redis-cluster or redis-sentinel. The ioredis redis library has support for them by default. This allows you to have a High Availability redis configuration.

Crashing and Safety

> ./node_modules./bin/actionhero start cluster --workers 1
2016-04-11T18:51:32.891Z - info: actionhero >> start cluster
2016-04-11T18:51:32.904Z - notice:  - STARTING CLUSTER -
2016-04-11T18:51:32.905Z - notice: pid: 43315
2016-04-11T18:51:32.911Z - info: starting worker #1
2016-04-11T18:51:33.097Z - info: [worker #1 (43316)]: starting
2016-04-11T18:51:33.984Z - info: [worker #1 (43316)]: started
2016-04-11T18:51:33.985Z - notice: cluster equilibrium state reached with 1 workers
2016-04-11T18:51:44.775Z - alert: [worker #1 (43316)]: uncaught exception => yay is not defined
2016-04-11T18:51:44.775Z - alert: [worker #1 (43316)]:    ReferenceError: yay is not defined
2016-04-11T18:51:44.775Z - alert: [worker #1 (43316)]:        at Object.exports.action.run (/app/actionhero/actions/bad.js:14:5)
2016-04-11T18:51:44.775Z - alert: [worker #1 (43316)]:        at /app/actionhero/initializers/ActionProcessor.js:268:31
2016-04-11T18:51:44.775Z - alert: [worker #1 (43316)]:        at /app/actionhero/initializers/ActionProcessor.js:149:9
2016-04-11T18:51:44.776Z - alert: [worker #1 (43316)]:        at /app/actionhero/node_modules/async/lib/async.js:726:13
2016-04-11T18:51:44.776Z - alert: [worker #1 (43316)]:        at /app/actionhero/node_modules/async/lib/async.js:52:16
2016-04-11T18:51:44.776Z - alert: [worker #1 (43316)]:        at iterate (/app/actionhero/node_modules/async/lib/async.js:260:24)
2016-04-11T18:51:44.776Z - alert: [worker #1 (43316)]:        at async.forEachOfSeries.async.eachOfSeries (/app/actionhero/node_modules/async/lib/async.js:281:9)
2016-04-11T18:51:44.776Z - alert: [worker #1 (43316)]:        at _parallel (/app/actionhero/node_modules/async/lib/async.js:717:9)
2016-04-11T18:51:44.776Z - alert: [worker #1 (43316)]:        at Object.async.series (/app/actionhero/node_modules/async/lib/async.js:739:9)
2016-04-11T18:51:44.777Z - alert: [worker #1 (43316)]:        at api.ActionProcessor.preProcessAction (/app/actionhero/initializers/ActionProcessor.js:148:13)
2016-04-11T18:51:44.777Z - notice: cluster equilibrium state reached with 1 workers
2016-04-11T18:51:44.785Z - info: [worker #1 (43316)]: exited
2016-04-11T18:51:44.785Z - info: starting worker #1
2016-04-11T18:51:44.960Z - info: [worker #1 (43323)]: starting
2016-04-11T18:51:45.827Z - info: [worker #1 (43323)]: started
2016-04-11T18:51:45.827Z - notice: cluster equilibrium state reached with 1 workers
  • Let the app crash rather than being defensive prematurely. ActionHero has a good logger, and if you are running within start cluster mode, your server will be restarted. It is very easy to hide uncaught errors, exceptions, or un-resolved promises, and doing so might leave your application in strange state.
  • We removed domains from the project in v13 to follow this philosophy, and rely on a parent process (start cluster) to handle error logging. Domains are deprecated in node.js now for the same reasons we discuss here.
    • For example, if you timeout connections that are taking too long, what are you going to do about the database connection it was running? Will you roll it back? What about the other clients using the same connection pool? How can you be sure which connection in the mySQL pool was in use? Rather than handle all these edge cases… just let your app crash, log, and reboot.
  • As noted above, centralized logging (Splunk et al) will be invaluable here. You can can also employ a tool like BugSnag to collect and correlate errors.

Actions

  • Remember that all params which come in via the web and socket servers are Strings. If you want to typeCast them (perhaps you always know that the param user_id will be an integer), you can do so in a middleware or within an action's params.formatter step.
  • Always remember to sanitize any input for SQL injection, etc. The best way to describe this is "never pass a query to your database which can be directly modified via user input"!
  • Remember that you can restrict actions to specific server types. Perhaps only a web POST request should be able to login, and not a websocket client. You can control application flow this way.
  • Crafting authentication middleware is not that hard

Tasks

  • Tasks can be created from any part of ActionHero: Actions, Servers, Middleware, even other Tasks.
  • You can chain tasks together to create workflows.
  • ActionHero uses the multiWorker from node-resque. When configured properly, it will consume 100% of a CPU core, to work as many tasks at once as it can. This will also fluctuate depending on the CPU difficulty of the job. Plan accordingly.
  • Create a way to view the state of your redis cluster. Are you running out of RAM? Are your Queues growing faster than they can be worked? Checking this information is the key to having a healthy ecosystem. The methods for doing so are available.
  • Be extra-save within your actions, and do not allow an uncaught exception. This will cause the worker to crash and the job to be remain ‘claimed' in redis, and never make it to the failed queue.

Upgrading ActionHero

Overview

Upgrading big ActionHero projects to a new major might require some effort. Every ActionHero version has it's own specific project files which you generate using actionhero generate command. One of the ways to upgrade your project is to generate a new project using the latest ActionHero framework (npm install actionhero && ./node_modules/.bin/actionhero generate). Using that as your starting point you can then carefully copy all your configs, initializers, servers, links and other custom code from your old project, making sure that you are at the same working state as before. It's a good practice to make tests for your actions (or any other component) before you plan to upgrade your ActionHero project.

With good test coverage you can make sure that you have successfully upgraded your project.

ActionHero follows semantic versioning. This means that a minor change is a right-most number. A new feature added is the middle number, and a breaking change is the left number. You should expect something in your application to need to be changed if you upgrade a major version.

Upgrading from v16.x.x to v17.x.x

Full Release Notes: GitHub

Breaking Changes and How to Overcome Them:

  • Localization (i18n)
    • In ./config/i18n.js be sure to enable objectNotation, or else the new locale file will be gibberish to ActionHero
    • As of this release, ActionHero no longer localizes its log messages. This is done to simplify and speed up the logger methods. There is not mitigation path here without overwriting the api.log() method.
      • Any use of % interpolation should be removed from your logger strings. Favor native JS string templates.
    • ActionHero now ships with locale files by default.
      • You will need to acquire the default locale file and copy it into ./locales/en.json within your project.
      • The error reporters have all been changed to use these new locale file and mustache-style syntax. Update your from the default errors file
      • The welcomeMessage and goodbyeMessage are removed from the config files and ActionHero now refrences the locale files for these strings. Update yours accodingly.
  • utils
    • api.utils.recursiveDirectoryGlob has been removed in favor of the glob package. Use this instead.

Upgrading from v15.x.x to v16.x.x

Full Release Notes: GitHub

Breaking Changes and How to Overcome Them:

The only breaking changes are related to the capilization of internal methods:

  • api.Connecton() rather than api.connection()
  • api.GenericServer() rather than api.genericServer()
  • api.ActionProcessor() rather than api.actionProcessor()
  • require("actionhero") not require("actionhero").actionheroPrototype should you be using ActionHero programatically.

Upgrading from v14.x.x to v15.x.x

Full Release Notes: GitHub

Breaking Changes and How to Overcome Them:

`actionhero generateAction --name=[name]`      -> `actionhero generate action --name=[name]`
`actionhero generateInitializer --name=[name]` -> `actionhero generate initializer --name=[name]`
`actionhero generateServer --name=[name]`      -> `actionhero generate server --name=[name]`
`actionhero generateTask --name=[name]`        -> `actionhero generate task --name=[name]`
  
  • The ActionHero binary has had it's commands changed.
    • Any deployment or automation tools you use will need to be updated accordingly.
  • Tasks now use middleware instead of plugins.
    • You will need to convert all uses of task plugins to task middleware.

Upgrading from v13.x.x to v14.x.x

Full Release Notes: GitHub

Breaking Changes and How to Overcome Them:

  • Redis Client Configurations have changed drastically. This allows for greater configuration, but at a complexity cost.
    • The easiest way to upgrade your config/redis.js is to take if from the master branch directly and re-apply your configuration.
    • Move api.config.redis.channel to api.config.general.channel
    • Move api.config.redis. rpcTimeout to api.config.general.rpcTimeout
    • Throughout the code, use api.config.redis.client rather than api.redis.client

Upgrading from v12.x.x to v13.x.x

Full Release Notes: GitHub

Breaking Changes and How to Overcome Them:

  • Pluggins
    • config/plugins.js is removed. Delete yours.
    • Use the new binary command, actionhero link --name=NameOfPlugin to link your plugins in the new method.
    • Linking plugins will likley create new config files you may need to customize.
  • Locales
    • This release introduced Locales. You will need the new locale config file. The easiest way to upgrade your config/i18n.js is to take if from the master branch.
    • Ensure that api.config.i18n.updateFiles is true so that your locale files can be generated for the first time.
  • Errors
    • config/errors.js has been completely redone to take advantage of connection.localize. The easiest way to upgrade your config/errors.js is to take if from the master branch.
  • Grunt Removed
    • Grunt is removed from the project. The old ActionHero grunt commands have been moved into the ActionHero binary.
  • Redis configuration
    • package is a reserved keyword in JavaScript. We now use the key pkg in the redis config.

Upgrading from v11.x.x to v12.x.x

Full Release Notes: GitHub

Breaking Changes and How to Overcome Them:

  • Redis configuration
    • Switch from using the redis npm pacakge to ioredis. Change this in your package.json.
  • ioredis handles passwords slightly differently. Read the ioredis documentation to learn more.
  • Stats Removed
    • The api.stats subsection has been removed from actionhero
    • If you need the stats subsection, you can get get it via plugin

Upgrading from v10.x.x to v11.x.x

Full Release Notes: GitHub

Breaking Changes and How to Overcome Them:

  • Action Syntax changed
      run: function(api, data, next){
        data.response.randomNumber = Math.random();
        next(error);
      }
              
    • Where data contains:
    • data = {
        connection: connection,
        action: 'randomNumber',
        toProcess: true,
        toRender: true,
        messageCount: 123,
        params: { action: 'randomNumber', apiVersion: 1 },
        actionStartTime: 123,
        response: {},
      }
              
    • You will need to change all of your actions to use data.connection rather than connection directly.
    • You will need to change all of your actions to use data.response rather than connection.response directly.
  • Middleware syntax has changed to match action's data pattern. You will need to change your middleware accordingly.
  • Removed connection._originalConnection.
  • Websockets:
    • The params of websocket connections should NOT be sticky. All actions will start with connection.params = {}. If you rely on the old behavior, you will need to change your client code.
  • Action Processor:
    • Removed duplicate callback prevention in ActionProcessor. This belongs on the user/implementer to handle.