Using Redis as a Queue in Nodejs

When scaling out a website, sometimes you will have cpu intensive tasks or tasks that the user should not wait on when making a API request. For these kind of activities, one option is to create a Queue. You send a message to one server and have a worker (workers) read that request and process.

For example, say the user sends a request to complete a todo (like at Habitica ;)) and you then need to send webhooks that are subscribed to this action. This is a great time to set up a queue and worker.

For more ideas on queue and how to scale, check out this repo.

Using Redis Simple Message Queue

For this tutorial, let’s get started with a repo I found rightly called Redis Simple Message Queue.

You will first need to install Redis on your system, once you have that, let’s build the producer. The producer will send request to Redis.

Create a new directory and a file for the producer.

mkdir redis-simple-queue
cd redis-simple-queue
touch producer.js

Next, let’s init a new npm project and install rsmq

npm init -y
npm install rsmq --save

Creating the Producer

Now, in the producer.js, let’s add the following to connect to Redis. We first import the new library, then connect to Redis assumed on our localhost.

const RedisSMQ = require("rsmq");
const rsmq = new RedisSMQ({
  host: "127.0.0.1",
  port: 6379, ns: "rsmq"
});

Then, we should create a queue that will listen to our webhook requests. Note that you can have multiple queues for all sorts of processes.

rsmq.createQueue({ qname: 'webhook-queue'}, (err, resp) => {
	if (resp === 1) console.log('queue created');
});

Alright! Almost done here. Now that we are connect, let’s send a message.

rsmq.sendMessage({
    qname: 'webhook-queue',
    message: 'Hello World',
  }, (err, resp) => {
  	if (resp) console.log('Message sent. ID:', resp);
  });

This is a good example of using a producer, but for our scenario at the beginning, we would be calling this send message inside of our Express.js route request.

The full code for our producer is here.

const RedisSMQ = require("rsmq");
const rsmq = new RedisSMQ({
  host: "127.0.0.1",
  port: 6379, ns: "rsmq"
});


rsmq.createQueue({ qname: 'webhook-queue'}, (err, resp) => {
	if (resp === 1) console.log('queue created');

  rsmq.sendMessage({
    qname: 'webhook-queue',
    message: 'Hello World',
  }, (err, resp) => {
  	if (resp) console.log('Message sent. ID:', resp);
  });
});

Building a Worker

Okay, now we are ready to create a worker to process. First create a new file.

touch worker.js

Then, let’s add the connection logic again.

const RedisSMQ = require("rsmq");
const rsmq = new RedisSMQ({
  host: "127.0.0.1",
  port: 6379, ns: "rsmq"
});

 
Now, we add a little snippet to recieve the message. And you can see where we will then handle the logic for the request.

rsmq.receiveMessage({
  qname: 'webhook-queue',
}, (err, resp) => {
	if (!resp.id) return;
	console.log('Message received.', resp);
  // Do long logic here
});

We are almost done. Now that we have processed the request, we need to remove the request from the queue so no other worker processes it. The reason we don’t automatically remove the queue when we send it is for retries. So, if one work fails, we can just resend the request.

rsmq.deleteMessage({
      qname: 'webhook-queue',
      id: resp.id,
  }, (err, resp) => {
  	if (resp === 1) console.log('Message deleted.');
  });

And the full code is here.

const RedisSMQ = require("rsmq");
const rsmq = new RedisSMQ({
  host: "127.0.0.1",
  port: 6379, ns: "rsmq"
});

rsmq.receiveMessage({
  qname: 'webhook-queue',
}, (err, resp) => {
	if (!resp || !resp.id) return;
	console.log('Message received.', resp);

  // Do long logic here

  rsmq.deleteMessage({
      qname: 'webhook-queue',
      id: resp.id,
  }, (err, resp) => {
  	if (resp === 1) console.log('Message deleted.');
  });
});

Conclusion

This is a very short introduction to queues, but I’m hoping you can see the power of queue already. There is much more to explore: how to use multiple workers, how to do retries and timeouts and how to orchestrate all of this. We leave this to explore in the future :D.

Installing Redis on Mac OSX with Homebrew

Below is a quick recipe on install Redis with the great Homebrew.

brew install redis

That’s it for install! Now let’s get Redis started. You can verify the install with this command.

brew info redis

 
You can run Redis with this command.

brew services start redis

Now, let’s check redis

redis-cli ping

Done! Easy enough. You can use this command to view service status with Brew.

brew services list