# Queues

# Introduction

While developing your application, you may have some tasks, such as sending email to client, that take too long to perform during a typical web request. ExpressWebJs allows you to easily create queued jobs that will be processed in the background. By moving time intensive tasks to a queue, your application can respond to web requests with blazing speed and provide a better user experience to your clients.

ExpressWebJs queue configuration options are stored in your application's Config/Queue.ts configuration file. In this file you will find connection configurations for queue driver that are included with the framework.

Currently, ExpressWebJs supports only rabbitmq and redis. A null queue driver is also included which discards queued jobs.

# Connections Vs. Queues

Before getting started with ExpressWebJs queues, it is important to understand the distinction between "connections" and "queues". In your Config/Queue.ts configuration file, there is a connections configuration array. This option defines the connections to backend queue services such as rabbitmq or redis. However, any given queue connection may have multiple "queues" which may be thought of as different stacks or piles of queued jobs.

import { env, queueProviders } from "expresswebcorets/lib/Env";
import * as bullmq from "bullmq";

export default {
  /*
    |--------------------------------------------------------------------------
    | Default Queue Connection Name
    |--------------------------------------------------------------------------
    |
    | ExpressWebJs queue API supports a variety of back-ends via a single
    | API, giving you convenient access to each back-end using the same
    | syntax for every one. Here you may define a default connection.
    |
    */

  default: env("QUEUE_CONNECTION", null),

  /*
    |--------------------------------------------------------------------------
    | Queue Connections
    |--------------------------------------------------------------------------
    |
    | Here you may configure the connection information for rabbitmq server that
    | is used by your application. A default configuration has been added.
    | ExpressWebJs currently supports rabbitmq and redis for now. 
    |
    | Currently Supported Drivers: "rabbitmq","redis".
    | Make sure to install the provider for each driver.
    | For rabbitmq, install provider via `npm i rabbitmq`
    | For redis, install provider via `npm i bull`
    |
    */

  connections: {
    rabbitmq: {
      driver: "rabbitmq",
      provider: null,
      provider_name: queueProviders.RABBITMQ,
      connection: "default",
      queue: env("RABBITMQ_QUEUE", "default"),
      retry_after: 90,
      block_for: null,
      host: env("RABBITMQ_HOST"),
      port: env("RABBITMQ_PORT"),
    },
    redis: {
      driver: "redis",
      provider: bullmq,
      provider_name: queueProviders.BULLMQ,
      connection: "default",
      queue: env("REDIS_QUEUE", "default"),
      password: env("REDIS_PASSWORD"),
      host: env("REDIS_HOST"),
      port: env("REDIS_PORT"),
    },
  },
};

The .env values for RabbitMQ

QUEUE_CONNECTION=rabbitmq
RABBITMQ_QUEUE=myQueue
RABBITMQ_HOST=amqp://localhost
RABBIT_PORT=5672

The .env values for Redis

REDIS_CLIENT=default
REDIS_QUEUE=myQueue
REDIS_HOST=127.0.0.1
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0

The connection configuration example in the queue configuration file contains a queue attribute. This is the default queue that jobs will be dispatched to when they are sent to a given connection. In other words, if you dispatch a job without explicitly defining which queue it should be dispatched to, the job will be placed on the queue that is defined in the queue attribute of the connection configuration:

import sendMailJob from "App/Jobs/sendMail_job";

// This job is sent to the default connection's default queue...
new sendMailJob().dispatch(data);

// This job is sent to the default connection's "emails" queue...
sendMailJob.onQueue("emails").dispatch(data);

Some applications may not need to ever push jobs into multiple queues, instead preferring to have one simple queue. However, pushing jobs to multiple queues can be especially useful for applications that wish to prioritize or segment how jobs are processed.

# RabbitMq

In order to use the rabbitmq queue driver, you should first download the provider

Once the provider is downloaded, you can now configure a Rabbitmq connection in your App/Config/Queue.ts configuration file.

  connections: {
     rabbitmq: {
      driver: "rabbitmq",
      provider: rabbitmq,
      provider_name: queueProviders.RABBITMQ,
      connection: "default",
      queue: env("RABBITMQ_QUEUE", "default"),
      retry_after: 90,
      block_for: null,
      host: env("RABBITMQ_HOST"),
      port: env("RABBITMQ_PORT"),
    },
};

# Redis with bull

In order to use the redis queue driver, you should first download the provider

Once the provider is installed, you can now configure a Redis connection in your App/Config/Queue.ts configuration file.

  connections: {
     redis: {
      driver: "redis",
      provider: bull,
      provider_name: queueProviders.BULL,
      connection: "default",
      queue: env("REDIS_QUEUE", "default"),
      password: env("REDIS_PASSWORD"),
      host: env("REDIS_HOST"),
      port: env("REDIS_PORT"),
    },
};

# Redis with bullmq

In order to use the redis queue driver, you should first download the provider

Once the provider is installed, you can now configure a Redis connection in your App/Config/Queue.ts configuration file.

  connections: {
     redis: {
      driver: "redis",
      provider: bullmq,
      provider_name: queueProviders.BULLMQ,
      connection: "default",
      queue: env("REDIS_QUEUE", "default"),
      password: env("REDIS_PASSWORD"),
      host: env("REDIS_HOST"),
      port: env("REDIS_PORT"),
    },
};

# Creating Jobs

By default, all of the queueable jobs for your application are stored in the App/Jobs directory. If the App/Jobs directory doesn't exist, it will be created when you run the make-job Maker command:

The generated class will extend the ShouldQueue class, indicating to ExpressWebJs that the job should be pushed into the queue to run asynchronously.

# Class Structure

Job classes are very simple, normally containing only a handle method that is invoked when the job is processed by the queue. To get started, let's take a look at an example job class. Here, we want to send an email to a clients once they register in our application. We will be using Redis queue.

import ShouldQueue from "expresswebcorets/lib/Queue/shouldQueue";

class SendMailJob extends ShouldQueue {
  constructor() {
    super("SendMailJob_job");
  }
  /**
   * Execute the job.
   * @return void
   */
  handle<T>(data: T) {
    // Process email sending ...
  }
}

export default SendMailJob;
⚠️ We imported bull, because thats the supported redis provider for ExpressWebJs. If you are using rabbitmq, you will need to import rabbitmq

# Dispatching Jobs

Once you have written your job class, you may dispatch it using the dispatch method on the job itself. The arguments passed to the dispatch method will be given to the job's handle method:

Example:

import { Request, Response, NextFunction } from "Elucidate/HttpContext";
import HttpResponse from "Elucidate/HttpContext/ResponseType";
import sendMailJob from "App/Jobs/sendMail_job";
import clientModel from "App/Model/Client_model";

export class ClientController{
  public async store(req: Request, res: Response){
    const client = await clientModel.query().insert({
            firstName: req.body.firstname,
            lastName: req.body.lastname,
            email: req.body.email,
      });
    // ...
    new sendMailJob().dispatch(client); 👈 You dispatch your job
    }
}

//In your sendMailJob class handle method
handle(data){
  console.log(data.firstName);
  console.log(data.lastName);
  console.log(data.email);
}

If you would like to conditionally dispatch a job, you may use the dispatchIf method:

new sendMailJob().dispatchIf(condition, client);

# BullMQ options

You can still passing bullmq job options when dispatching your job.

import { JobOptions } from "bullmq";

new lookUpJob().dispatch<IUser, JobOptions>(user, {
  repeat: {
    every: 10000,
    limit: 20,
  },
});

This repeats job every 10 seconds but no more than 20 times

# Dispatching To A Particular Queue

By pushing jobs to different queues, you may "categorize" your queued jobs. Keep in mind, this does not push jobs to different queue "connections" as defined by your queue configuration file, but only to specific queues within a single connection. To specify the queue, use the onQueue method when dispatching the job:

import { Request, Response, NextFunction } from "Elucidate/HttpContext";
import HttpResponse from "Elucidate/HttpContext/ResponseType";
import sendMailJob from "App/Jobs/sendMail_job";
import clientModel from "App/Model/Client_model";

export class ClientController {
  public async store(req: Request, res: Response){
    const client = await clientModel.query().insert({
        firstName: req.body.firstname,
        lastName: req.body.lastname,
        email: req.body.email,
    });

    // ...
    new sendMailJob().onQueue("email").dispatch(client); 👈 dispatch your job to email queue
  };
}

# Running The Queue Worker

The queue-work Command

ExpressWebjs includes a Maker command that will start a queue worker and process new jobs as they are pushed onto the queue. You may run the worker using the queue-work Maker command. Note that once the queue-work command has started, it will continue to run until it is manually stopped or you close your terminal:

If you are running the queue with a name, then you need to add the name:

# Supervisor

In production, you need a way to keep your queue-work processes running. A queue-work process may stop running for a variety of reasons, such as an exceeded worker timeout.

For this reason, you need to configure a process monitor that can detect when your queue:work processes exit and automatically restart them.