ExpressWebJS

# Queues

While developing your application, you may have some tasks, such as sending email to client, that take too long to perform during a typical web request. ExpressWebJs allows you to easily create queued jobs that will be processed in the background. By moving time intensive tasks to a queue, your application can respond to web requests with blazing speed and provide a better user experience to your clients.

ExpressWebJs queue configuration options are stored in your application's Config/queue.ts configuration file. In this file you will find connection configurations for queue driver that are included with the framework.

Currently, ExpressWebJs supports only rabbitmq. A null queue driver is also included which discards queued jobs.

So install rabbitmq

 npm install amqplib

# Connections Vs. Queues

Before getting started with ExpressWebJs queues, it is important to understand the distinction between "connections" and "queues". In your Config/queue.ts configuration file, there is a connections configuration array. This option defines the connections to backend queue services such as rabbitmq. However, any given queue connection may have multiple "queues" which may be thought of as different stacks or piles of queued jobs.

"use strict";
import env from "expresswebcorets/lib/ENV";

export default {
  /*
    |--------------------------------------------------------------------------
    | Default Queue Connection Name
    |--------------------------------------------------------------------------
    |
    | ExpressWebJs queue API supports an assortment of back-ends via a single
    | API, giving you convenient access to each back-end using the same
    | syntax for every one. Here you may define a default connection.
    |
    | `run npm install amqplib` to get started
    */

  default: env("QUEUE_CONNECTION", null),

  /*
    |--------------------------------------------------------------------------
    | Queue Connections
    |--------------------------------------------------------------------------
    |
    | Here you may configure the connection information for rabbitmq server that
    | is used by your application. A default configuration has been added.
    | ExpressWebJs currently supports rabbitmq for now.
    |
    | Driver: "rabbitmq"
    |
    */

  connections: {
    rabbitmq: {
      driver: "rabbitmq",
      connection: "default",
      queue: env("RABBITMQ_QUEUE", "default"),
      retry_after: 90,
      block_for: null,
      host: env("RABBITMQ_HOST"),
      port: env("RABBIT_PORT"),
    },
  },
};

The .env values are

QUEUE_CONNECTION=rabbitmq
RABBITMQ_QUEUE=myQueue
RABBITMQ_HOST=amqp://localhost
RABBIT_PORT=5672

The connection configuration example in the queue configuration file contains a queue attribute. This is the default queue that jobs will be dispatched to when they are sent to a given connection. In other words, if you dispatch a job without explicitly defining which queue it should be dispatched to, the job will be placed on the queue that is defined in the queue attribute of the connection configuration:

import sendMailJob from "App/Jobs/sendMail_job";

// This job is sent to the default connection's default queue...
new sendMailJob().dispatch(data);

// This job is sent to the default connection's "emails" queue...
sendMailJob.onQueue("emails").dispatch(data);

Some applications may not need to ever push jobs into multiple queues, instead preferring to have one simple queue. However, pushing jobs to multiple queues can be especially useful for applications that wish to prioritize or segment how jobs are processed.

# RabbitMq

In order to use the rabbitmq queue driver, you should configure a Rabbitmq connection in your App/Config/queue.js configuration file.

  /*
    |--------------------------------------------------------------------------
    | Default Queue Connection Name
    |--------------------------------------------------------------------------
    |
    | ExpressWebJs queue API supports an assortment of back-ends via a single
    | API, giving you convenient access to each back-end using the same
    | syntax for every one. Here you may define a default connection.
    |
    | `run npm install amqplib` to get started
    */

  default: process.env.QUEUE_CONNECTION || null,

  /*
    |--------------------------------------------------------------------------
    | Queue Connections
    |--------------------------------------------------------------------------
    |
    | Here you may configure the connection information for rabbitmq server that
    | is used by your application. A default configuration has been added.
    | ExpressWebJs currently supports rabbitmq for now.
    |
    | Driver: "rabbitmq"
    |
    */

  connections: {
    rabbitmq: {
      driver: "rabbitmq",
      connection: "default",
      queue: process.env.RABBITMQ_QUEUE || "default",
      retry_after: 90,
      block_for: null,
      host: process.env.RABBITMQ_HOST,
      port: process.env.RABBIT_PORT,
    },
  },
};

# Creating Jobs

By default, all of the queueable jobs for your application are stored in the App/Jobs directory. If the App/Jobs directory doesn't exist, it will be created when you run the make-job Maker command:

 ts-node maker make-job sendMailJob

The generated class will extend the ShouldQueue class, indicating to ExpressWebJs that the job should be pushed into the queue to run asynchronously.

# Class Structure

Job classes are very simple, normally containing only a handle method that is invoked when the job is processed by the queue. To get started, let's take a look at an example job class. Here, we want to send an email to a clients once they register in our application:

"use strict";
import ShouldQueue from "expresswebcorets/lib/Queue/shoudQueue";

class SendMailJob extends ShouldQueue {
  constructor() {
    super();
    this.signature = "SendMailJob_job";
    this.queueSignature(this.signature);
  }
  /**
   * Execute the job.
   * @return void
   */
  handle(data: any) {
    // Process email sending ...
  }
}

export default SendMailJob;

# Dispatching Jobs

Once you have written your job class, you may dispatch it using the dispatch method on the job itself. The arguments passed to the dispatch method will be given to the job's handle method:

Example:

import { Request, Response, NextFunction } from "Elucidate/HttpContext";
import HttpResponse from "Elucidate/HttpContext/ResponseType";
import sendMailJob from "App/Jobs/sendMail_job";
import clientModel from "App/Model/Client_model";

class ClientController
{
    store = async (req: Request, res: Response, next: NextFunction) => {
        let client = clientModel.query().insert({
            firstName: req.body.firstname,
            lastName: req.body.lasttname,
            email: req.body.email,
        });

        // ...

        new sendMailJob().dispatch(client);
    }
}

//In your sendMailJob class handle method
handle(data){
    console.log(data.firstName);
    console.log(data.lastName);
    console.log(data.email);
}

If you would like to conditionally dispatch a job, you may use the dispatchIf method:

new sendMailJob().dispatchIf(condition, client);

# Dispatching To A Particular Queue

By pushing jobs to different queues, you may "categorize" your queued jobs. Keep in mind, this does not push jobs to different queue "connections" as defined by your queue configuration file, but only to specific queues within a single connection. To specify the queue, use the onQueue method when dispatching the job:

import { Request, Response, NextFunction } from "Elucidate/HttpContext";
import HttpResponse from "Elucidate/HttpContext/ResponseType";
import sendMailJob from "App/Jobs/sendMail_job";
import clientModel from "App/Model/Client_model";

class ClientController
{
    store = async (req: Request, res: Response, next: NextFunction) => {
        let client = clientModel.query().insert({
            firstName: req.body.firstname,
            lastName: req.body.lasttname,
            email: req.body.email,
        });

        // ...

        new sendMailJob().onQueue('email').dispatch(client);
    }
}

# Running The Queue Worker

The queue-work Command

ExpressWebjs includes a Maker command that will start a queue worker and process new jobs as they are pushed onto the queue. You may run the worker using the queue-work Maker command. Note that once the queue-work command has started, it will continue to run until it is manually stopped or you close your terminal:

 ts-node maker queue-work

If you are running the queue with a name, then you need to add the name:

 ts-node maker queue-work [QUEUE-NAME]

# Supervisor

In production, you need a way to keep your queue-work processes running. A queue-work process may stop running for a variety of reasons, such as an exceeded worker timeout.

For this reason, you need to configure a process monitor that can detect when your queue:work processes exit and automatically restart them.

← Task Scheduling →