Node.js Sleep using Async/Await

Are you also downloading lots of data from an API and need a simple way to throttle? Here is a quick snippet on how to use sleep (a function common in many languages).

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function work() {
  console.log('Start sleeping');
  await sleep(5000);
  console.log('Five seconds later');
}

work();

 

Json to DataFrame with Pandas

This week I need to grab a JSON file (from my Mongo database) and put this info in a Dataframe. Pandas makes this supprisingly easy, of course. Here is the code.

import pandas as pd
import json

with open('./datajson') as data_file:
    data = json.load(data_file)
    df = pd.io.json.json_normalize(data)
    print(df)

We simply load the json file, parse the json with the build in Python json library and then use the json_normalize to convert the documents. Super easy.

Dockerizing a Node.js App

In this tutorial, I’ll quickly show you how to Dockerize a Nodejs app. I want cover what docker is or why – hopefully that’s why you found this post. Let’s get to it.

Creating the Project

Start by creating a new project

mkdir nodejs-docker
cd nodejs-docker/
npm init -y
touch index.js
touch Dockerfile

The only thing new above should be the Dockerfile. This is where we will declare the commands for docker to run.

And here is the docker file with coments on the steps.

FROM node:8 # Tell Docker to start from a Node:8 image (this is maintained by node)

WORKDIR /home/user/nodejs-docker # Add the absolute path to our project directory

COPY . . # Copy all of the current directory (the one above) to the Docker directory

RUN npm install --production # Install everything we need from production

CMD ["node", "index.js"] # Run the app!

Building and Running

Now that we have the project set up, let’s build and run the app.

Here is the build command. This builds our current directory (find that Dockerfile) and creates and image.

docker build -t nodejs-docker .

We can view images on our machine with this.

docker image ls

Finally, to run the command we do the following command:

docker run -p 8080:80 nodejs-docker

This command runs our docker image and forward the outside port 8080 to port 80 in the docker container. To get this to work, you will need to build something like an Express app to listen to port 80.

And, that’s about it! If you like to explore further, I recommend checking out this blog: https://nodesource.com/blog/containerizing-node-js-applications-with-docker and the official docs: https://docs.docker.com/get-started/

RxJava2 Completable Example

When using RxJava to listen to events, especially API responses from Retrofit, we typically use the Observable class. Similarly to using a Single, we don’t always need the onNext.

In fact, sometimes we don’t need any return info example for the complete event. For example, if we are posting a task to an API and we just want to know that the request was successful. For these casses we can use the Completable class.

Say we have this retrofit interface:

public interface APIClient {
    @POST("/task")
    Completable createTask(@Body String taskName);
}

Then we can simply use a Completable to let us know if the request was successful.

apiClient.createTask("a new task")
    .subscribe(() -> {
        // handle complete
    }, throwable -> {
        // handle error
    });

As with a Single, this helps us save a bit of boilerplate code throughout our code base.

RxJava2 Single Example

When using RxJava to listen to events, especially API responses from Retrofit, we typically use the Observable class. However, sometimes we don’t need the onNext call because we simply want the final result. For that, we can use a Single.

Say we have this retrofit interface:

public interface APIClient {
    @GET("/me")
    Single getMe();
}

Then we can simply use a Single to let us know when the user is fetched.

apiClient.getMe()
    .subscribe(user -> {
        // user returned
    }, throwable -> {
        // handle error
    });

It’s not a huge difference, but when you are repeating this code over a code base it can help to remove that extra function. It also makes the code more readable with the intent being to only handle the returned data.

Sending Email in Node.JS with Amazon SES

Amazone Simple Email Service (SES) is a cheap and easy way to send transactional emails to your users. I use this on many projects to do “lifecycle marketing” (activity updates, reminders) and sending items like password reset. In this tutorial, we will set up Amazon SES and send a quick email.

Prereqs

Start by setting up your aws keys.
https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-node-credentials-shared.html?shortFooter=true

Getting Setup

First let’s create a new directiory and set up a npm project.

mkdir amazon-ses
cd amazon-ses/
npm init -y

 Then, install the aws-sdk library and create a new file for our email program.

npm install aws-sdk --save
touch send-email.js

Sending the Email

We start off by configuring our AWS library and creating a new instance of AWS.SES which allows us to send the emails.

const AWS = require('aws-sdk');

AWS.config.update({ region: 'us-west-2' });

const SES = new AWS.SES({ apiVersion: '2010-12-01' });

Now, to send the email we simply use the sendEmail and pass it a list of params. We also call .promise() so we can have a promise, of course 😀

const fromEmailAddress = 'myadmin@admin.com';
const testEmailAddress = 'test@test.com';

const params = {
  Destination: {
    CcAddresses: [
      testEmailAddress,
    ],
    ToAddresses: [
      testEmailAddress,
    ],
  },
  Message: {
    Body: {
      Html: {
       Charset: 'UTF-8',
       Data: '<div>Html!</div>',
      },
      Text: {
       Charset: 'UTF-8',
       Data: 'test email',
      }
     },
     Subject: {
      Charset: 'UTF-8',
      Data: 'Test email'
     }
    },
  Source: fromEmailAddress,
  ReplyToAddresses: [
    testEmailAddress,
  ],
};

const sendPromise = SES.sendEmail(params).promise();
sendPromise.then((data) =&gt; {
    console.log(data.MessageId);
  }).catch((err) =&gt; {
    console.error(err, err.stack);
  });

The params are rather large, but it will get the job done!

Using Redis as a Queue in Nodejs

When scaling out a website, sometimes you will have cpu intensive tasks or tasks that the user should not wait on when making a API request. For these kind of activities, one option is to create a Queue. You send a message to one server and have a worker (workers) read that request and process.

For example, say the user sends a request to complete a todo (like at Habitica ;)) and you then need to send webhooks that are subscribed to this action. This is a great time to set up a queue and worker.

For more ideas on queue and how to scale, check out this repo.

Using Redis Simple Message Queue

For this tutorial, let’s get started with a repo I found rightly called Redis Simple Message Queue.

You will first need to install Redis on your system, once you have that, let’s build the producer. The producer will send request to Redis.

Create a new directory and a file for the producer.

mkdir redis-simple-queue
cd redis-simple-queue
touch producer.js

Next, let’s init a new npm project and install rsmq

npm init -y
npm install rsmq --save

Creating the Producer

Now, in the producer.js, let’s add the following to connect to Redis. We first import the new library, then connect to Redis assumed on our localhost.

const RedisSMQ = require("rsmq");
const rsmq = new RedisSMQ({
  host: "127.0.0.1",
  port: 6379, ns: "rsmq"
});

Then, we should create a queue that will listen to our webhook requests. Note that you can have multiple queues for all sorts of processes.

rsmq.createQueue({ qname: 'webhook-queue'}, (err, resp) => {
	if (resp === 1) console.log('queue created');
});

Alright! Almost done here. Now that we are connect, let’s send a message.

rsmq.sendMessage({
    qname: 'webhook-queue',
    message: 'Hello World',
  }, (err, resp) => {
  	if (resp) console.log('Message sent. ID:', resp);
  });

This is a good example of using a producer, but for our scenario at the beginning, we would be calling this send message inside of our Express.js route request.

The full code for our producer is here.

const RedisSMQ = require("rsmq");
const rsmq = new RedisSMQ({
  host: "127.0.0.1",
  port: 6379, ns: "rsmq"
});


rsmq.createQueue({ qname: 'webhook-queue'}, (err, resp) => {
	if (resp === 1) console.log('queue created');

  rsmq.sendMessage({
    qname: 'webhook-queue',
    message: 'Hello World',
  }, (err, resp) => {
  	if (resp) console.log('Message sent. ID:', resp);
  });
});

Building a Worker

Okay, now we are ready to create a worker to process. First create a new file.

touch worker.js

Then, let’s add the connection logic again.

const RedisSMQ = require("rsmq");
const rsmq = new RedisSMQ({
  host: "127.0.0.1",
  port: 6379, ns: "rsmq"
});

 
Now, we add a little snippet to recieve the message. And you can see where we will then handle the logic for the request.

rsmq.receiveMessage({
  qname: 'webhook-queue',
}, (err, resp) => {
	if (!resp.id) return;
	console.log('Message received.', resp);
  // Do long logic here
});

We are almost done. Now that we have processed the request, we need to remove the request from the queue so no other worker processes it. The reason we don’t automatically remove the queue when we send it is for retries. So, if one work fails, we can just resend the request.

rsmq.deleteMessage({
      qname: 'webhook-queue',
      id: resp.id,
  }, (err, resp) => {
  	if (resp === 1) console.log('Message deleted.');
  });

And the full code is here.

const RedisSMQ = require("rsmq");
const rsmq = new RedisSMQ({
  host: "127.0.0.1",
  port: 6379, ns: "rsmq"
});

rsmq.receiveMessage({
  qname: 'webhook-queue',
}, (err, resp) => {
	if (!resp || !resp.id) return;
	console.log('Message received.', resp);

  // Do long logic here

  rsmq.deleteMessage({
      qname: 'webhook-queue',
      id: resp.id,
  }, (err, resp) => {
  	if (resp === 1) console.log('Message deleted.');
  });
});

Conclusion

This is a very short introduction to queues, but I’m hoping you can see the power of queue already. There is much more to explore: how to use multiple workers, how to do retries and timeouts and how to orchestrate all of this. We leave this to explore in the future :D.

Installing Redis on Mac OSX with Homebrew

Below is a quick recipe on install Redis with the great Homebrew.

brew install redis

That’s it for install! Now let’s get Redis started. You can verify the install with this command.

brew info redis

 
You can run Redis with this command.

brew services start redis

Now, let’s check redis

redis-cli ping

Done! Easy enough. You can use this command to view service status with Brew.

brew services list

 

Quick start with Sails.js – Opinionated Javascript

While Node.Js has done create with unopionated frameworks that allow for conversation and new trials, sometimes you need some direction. During my development time I’ve seen the benefits to using something like Rails or Laravel with best practices built in. Of course, they come with a lot of bloat you may not need. However, if you are looking for some great practices and most features bootstrap, NodeJs can now deliver with Sail.js.

Here is a quick guide to getting set up. Navigate to your terminal and install the Sails command line tool.

npm install sails -g

Then, create a new project.

sails new sails-project

You will have to option to start and empty project (allowing for more configuration) or start with a full template. Let’s choose option 1 (full app).

Now, let’s boot up.

cd new-project
sails lift

Now visit http://localhost:1337/ and you will be set!

Check out the official docs to learn more.

CSS Grid in 5 Minutes

CSS Grid is a method to allow us to easily build grids in CSS. Let’s take a look at how easy this is.

Let’s start with HTML. This gives us a wrapper and 4 small grid items.

<div class="grid">
<div class="grid-item"></div>
<div class="grid-item"></div>
<div class="grid-item"></div>
<div class="grid-item"></div>
</div>

Then, the CSS. We declare the wrapper to be a grid, to repeat columns with 1fr (1 fraction of the space 4 times or 1/4) and each should have a gap of 1em.

.grid {
  display: grid;
  grid-template-columns: repeat(4, 1fr);
  grid-gap: 1em;
}

.grid-item {
  background-color: orange;
  min-height: 250px;
}

And that’s it! Here is the final result

See the Pen Your First CSS Grid in 5 Minutes by Keith (@thehollidayinn) on CodePen.