Integrate elastic beanstalk health check with slack using cloud watch alarm

Get notified on Slack through Cloud Watch Alarms whenever your elastic beanstalk application becomes unhealthy.

Click on the image to zoom

In this article I will explain step by step how you can get the entire ecosystem working.

Elastic Beanstalk ( EB ) let’s you deploy your application code and provision auto scaling group, elastic load balancing, security groups automatically for you. EB makes developer’s life easy and developers can focus on writing the core business functionalities without worrying about creating and maintaining the infrastructure on AWS.

EB let’s you integrate with Cloud Watch Alarms on various metrics of your EB application, for example , health status, healthy host counts etc.

Slack on the other hand, is the most popular collaboration tool used by many organizations to carry out operational tasks. We are going to marry Slack with Cloud Watch alarm to make a smart and automated monitoring system for our elastic beanstalk application. Whenever the health of elastic beanstalk application goes in degraded or severe, Cloud Watch alarm goes into alarm state and invokes a lambda function through SNS topic. Lambda function posts a message to a pre-selected slack channel using web hook.

Elastic Beanstalk + Cloud Watch Alarm + Slack = HAPPINESS !

Part – One ( Create The Slack App ) 

First thing first. We will first create our new slack app and make sure it can receive messages from external sources through web hooks.

Step#1

Log into https://api.slack.com . If you don’t have any slack account, you can create one for free. Once you logged into the site, you will see a menu “Your Apps” on top right corner as shown in below screen shot.

Click on the image to zoom

Click on that link which will take you to your apps dashboard as in below screen shot.

Click on the image to zoom

 

Click on “Create an App” button on the page which will bring the wizard to create a new app.

Type the name you want for your new slack app and then choose the workspace from the dropdown. Then hit “Create App” button.

Click on the image to zoom

Step#2

Your app should be created and you should see a page like below. Now you need to create a web hook to receive messages from external sources. On the left panel, under “Features”, you will see a link “Incoming Webhooks”. Click on the link. You should see Incoming Webhooks page. Click on “Activate Incoming Webhooks” toggle to turn it on.

Click on the image to zoom

You will be asked to select a channel in which you want to receive messages from external sources such as cloud watch in this tutorial.

Click on the image to zoom

Once you authorize, your web hook url should be created and details will be shown on screen.

Click on the image to zoom

Step#3

Now as the web hook is created, let’s test it out. On the web hook page, you will see a curl command and “Copy” button next to that. Hit the “Copy” button which will copy the entire curl command to clipboard.

Open the terminal window ( on Mac) or PowerShell ( On Windows) and paste the command and hit enter.

Click on the image to zoom

If everything goes fine, you should receive a message “Hello, World!” on your slack channel.

Part – Two ( Create The Beanstalk Application) 

Step#1

I am not going to deep dive how to create a beanstalk application from scratch.  I will write a separate blog on this topic soon. For now, you may follow below article to create and deploy a spring boot application.

https://aws.amazon.com/blogs/devops/deploying-a-spring-boot-application-on-aws-using-aws-elastic-beanstalk/

In this tutorial, I am using spring boot application. You can use any other language/framework you are familiar with which beanstalk supports. For example, PHP, Python or NodeJS.

For this tutorial I created one sample spring boot project which you can clone and build on your machine. Follow below commands which will clone, build and run the spring boot application running on port 5000.

 git clone https://github.com/just4give/springboot-aws-ebs.git && cd springboot-aws-ebs

 ./gradlew clean build

 java -jar build/libs/springboot-aws-ebs-0.0.1-SNAPSHOT.jar
Click on the image to zoom

Once your spring boot app is launched, issue below curl command which should return HTTP status code 200.

 curl -X GET -i http://localhost:5000/health 

Terminate the application by pressing Ctrl+D or closing the terminal window.  Note that the application jar file is created under build/libs which you will deploy to beanstalk.

Step#2

Now upload this jar file to the beanstalk application you created in step 1 following the link above.  I have created a beanstalk application using same jar as shown in below screen shot.

Click on the image to zoom

Now let’s test the deployed application. This application has one API /health which should return HTTP status 200. Copy the beanstalk application url and replace your application url in below curl command.

 curl -X GET -i http://springbootdev.us-west-2.elasticbeanstalk.com/health

The curl should return HTTP status code 200

Click on the image to zoom

Step#3

Now we will modify the health check url of beanstalk application to use this API.

Click on the image to zoom

Click on the “Configuration” menu from left pane. Then click “Modify” link of “Monitoring” card.

Type “/health” for health check path and hit “save” button.

Click on the image to zoom

Part – Three ( Create Lambda function and Cloud Watch alarm) 

Step#1

In this step, we will create one SNS topic. Log into AWS console and navigate to SNS dashboard (Services -> Application Integration -> Simple Notification Service). On the dashboard, click on “Create Topic” link. This will open a wizard. Type “springboot-sns” as the topic name and hit “Create” button.

Click on the image to zoom

We will not create any subscription right now. We will come back to this topic later and create a subscription with lambda endpoint.

Step#2

In this step we will create a cloud watch alarm for our spring boot application. Navigate to cloud watch (Services -> Management Tools -> Cloud Watch)

On the cloud watch console, click on the “Create Alarm” button which will open a wizard to create new alarm.  On “1. Select Metric” tab, click on Environment Metric under ElasticBeanstalk, then select your spring boot application environment.

Click on the image to zoom
Click on the image to zoom

 

Then hit “Next” button.

On “Define Alarm” tab, type name for the alarm as “springboot-env-health”.

Click on the image to zoom

Scroll down and configure two alarms as below. One for state ALARM and another for state OK. Select the SNS topic we created in last step from the dropdown.

Click on the image to zoom

Step#3

In this step, we will create our lambda function which will push message to the slack channel through the webhook url. Navigate to lambda console ( Services -> Compute -> Lambda )

On lambda dashboard, hit “Create Function” button.

Click on the image to zoom

Choose “Author from scratch”. Type “springbootslack” as name, choose NodeJS 6.10 as runtime and choose appropriate role. If you don’t have any existing role, choose “Create a custom role” from dropdown which will take you to IAM console where you can create a new role. This role will be assumed by your lambda function.

Finally hit “Create Function” button. Then copy and paste below code in lambda inline code editor.

Note: Change the path value at line#65. You will get the value from web hook url you created in part one.

Now hit “Save” button.


var https = require('https');
var util = require('util');

exports.handler = function(event, context) {
console.log(JSON.stringify(event, null, 2));
console.log('From SNS:', event.Records[0].Sns.Message);

var postData = {
    "channel": "#aws-sns",
    "username": "AWS SNS via Lamda :: Cloud",
    "text": "*" + event.Records[0].Sns.Subject + "*",
    "icon_emoji": ":aws:"
};

var message = JSON.parse(event.Records[0].Sns.Message);
var severity = "good";

if(message.NewStateValue =='OK'){
    severity = "#36a64f";
}else if(message.NewStateValue =='ALARM'){
    severity = "#ff0000";
}else{
    severity = "warning";
}

console.log('reason',message.NewStateReason );
//console.log('reason2',JSON.parse(message).NewStateReason );
postData.attachments = [
  {
    "color": severity,
    "text": message.NewStateReason,
    "fields": [
       {
        "title": "EnvironmentName",
        "value": message.Trigger.Dimensions[0].value,
        "short": true
       },
       {
        "title": "Metric",
        "value": message.Trigger.MetricName,
        "short": true
       },
       {
        "title": "Threshold",
        "value": message.Trigger.Threshold,
        "short": true
       },
       {
        "title": "Period",
        "value": message.Trigger.Period,
        "short": true
       }
     ],
    "footer": "crazykoder.com",
    "footer_icon": "https://platform.slack-edge.com/img/default_application_icon.png",
    "ts": parseInt((new Date(message.StateChangeTime).getTime() / 1000).toFixed(0))
    }
   ];

var options = {
   method: 'POST',
   hostname: 'hooks.slack.com',
   port: 443,
   path: '/services/XXXXX/YYYYYYYY'
};

var req = https.request(options, function(res) {
   res.setEncoding('utf8');
   res.on('data', function (chunk) {
   context.done(null);
   });
});

req.on('error', function(e) {
   console.log('problem with request: ' + e.message);
});

req.write(util.format("%j", postData));
   req.end();
};

Step#4

You have successfully created your lambda function. Before moving forward, let’s first test out the lambda function.

Click on the image to zoom

Click on “Test” button next to “Save” which will bring up the test wizard. Type the event name “DymmyTest” and then copy & paste below JSON inside bottom text area. Click on “Create”.


{
"Records": [
  {
   "EventSource": "aws:sns",
   "EventVersion": "1.0",
   "EventSubscriptionArn": "arn:aws:sns:us-east-1:xxxxx:slack-notification:71e7e166-eb12-46ad-954d-xxxxxx",
   "Sns": {
   "Type": "Notification",
   "MessageId": "5be3bf2d-031c-5824-9a96-ec538105f18b",
   "TopicArn": "arn:aws:sns:us-east-1:378123733884:slack-notification",
   "Subject": "ALARM: \"cloudwatch-alarm\" in US East (N. Virginia)",
   "Message": "{\"AlarmName\":\"cloudwatch-alarm\",\"AlarmDescription\":\"OK or INFO\",\"AWSAccountId\":\"xxxxxx\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [15.0 (22/05/18 20:37:00)] was greater than or equal to the threshold (5.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2018-05-22T20:38:49.939+0000\",\"Region\":\"US East (N. Virginia)\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"EnvironmentHealth\",\"Namespace\":\"AWS/ElasticBeanstalk\",\"StatisticType\":\"Statistic\",\"Statistic\":\"AVERAGE\",\"Unit\":null,\"Dimensions\":[{\"name\":\"EnvironmentName\",\"value\":\"ereg3-lov-np-test\"}],\"Period\":60,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanOrEqualToThreshold\",\"Threshold\":5.0,\"TreatMissingData\":\"\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
   "Timestamp": "2018-05-22T20:38:49.969Z",
   "SignatureVersion": "1",
   "Signature": "NEcGzhvdHgNM9MlkXQ8kTLZ5mRoNMvwC7ccCQrrU3pZU2wQRLBTHnGYuGJ00w51FIYpXyD2FU0I+4E99wZorNmVJDXFypZRfA7qImQDzth//4Of5CcCvY8Tvp1gpw4w2HoIzQLYRSqq7d/znn5Aj2OM5+MUoVVCvmoPKQVfTBT0adn925ke3IeQXII6TKvV0I+tg2+IEzbykIE/CM7N5VRswVH7LCl/rpR9jaifKdvoiGEQSEWjTFlJmyade7amNTLxszIsrFzw83aowAeTYAVjEv3JA3Tr441b9wrhmWMbr/vZ2utGl04a1/j4aec1AnnIIJJ4mZ9+8N1U66Yar4Q==",
   "SigningCertUrl": "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-eaea6120e66ea12e88dcd8bcbddca752.pem",
    "UnsubscribeUrl": "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:378123733884:ereg3-slack-notification:71e7e166-eb12-46ad-954d-0c1955a6f5bb",
   "MessageAttributes": {}
   }
  }
]
}

Click on the image to zoom

Now, on the lambda function page, you should see “DummyTest” selected next to “Test” button. If not, select that from the dropdown and hit “Test” button. If you have followed all the steps, you should receive a message on your slack channel like below.

Step#5

Now as you created your lambda function and tested it successfully, it’s time to hook it up to the SNS topic we created in step#1. Navigate to the SNS console and open the topic we created earlier.

On the topic detail page, click on “Create subscription” button which will bring up another wizard.

Click on the image to zoom

Choose “AWS Lambda” as protocol and then choose the lambda function “springbootslack” we created earlier. Leave version as default. Click on “Create subscription” button.

Part – Four ( Testing ) 

Step#1

Congratulations! You have successfully wired up all the pieces together and it’s to test out the entire system we just built. To test it, we need to make our spring boot application unhealthy. There are many ways you can achieve this but I am going to show you a very easy and clean approach without needing you to re-deploy the jar file.

Open your beanstalk application on aws console and click on “Configuration” from left pane. Then click on “Modify” link of “Software” card.

Go towards the bottom of the page. You will see some environment  properties of you beanstalk application. Add following property and hit “Apply”.

Name – ENDPOINTS_HEALTH_MAPPING_UP

Value –  INTERNAL_SERVER_ERROR

 

Click on the image to zoom

Above change will effect /health API and will return HTTP 500 instead of HTTP 200.  So this will make health checks to fail and will trigger cloud watch alarm. You should receive a message on your slack channel after 5 minutes notifying you that your spring boot application became unhealthy.

Now, go back to the Configuration -> Software and change the value from INTERNAL_SERVER_ERROR to OK. Click on “Apply” button. This change will restore /health API causing it to return HTTP 200. Wait for 5-7 minutes and you will be notified on slack that your application health has changed from alarm to ok.

Click on the image to zoom

CONGRATULATION!!! 

You have automated your application health monitoring. No need to periodically check your application, you will be notified right on your smart phone whenever health status changes from OK to ALARM or vice versa.

If you like this tutorial, please hit like button and share with others. If you have any issue, please let me know.

 

3

Leave a Reply

Your email address will not be published. Required fields are marked *