Build Resilient Microservices Using Spring Retry and Circuit Breaker Pattern

“Everything fails all the time” — Werner Vogels
This is sad but true, everything fails specially in Microservice architecture with many external dependencies. Modern applications have tens of microservices which communicate with each other over REST. Anytime any microservice may go down causing entire operation to fail. At a broad level we can classify these failures in two categories

  • Transient — where application will heal itself in a matter of seconds such as network glitch.
  • Non-Transient — where application suffer for a longer period, minutes or hours such as database connection, unavailability due to high traffic or throttling limit.
    To improve the resilience of our microservice architecture we should consider following two patterns.


  1. Retry
  2. Circuit Breaker

For transient failures, we don’t want to fail the request immediately rather would prefer to retry few times. There may a temporary network glitch and next attempt may be successful. While implementing Retry Pattern you should be careful how many retries you want. May be you can limit to 3 retries for each REST call as an example. But if the failure is not transient and you keep on doing 3 retries for each REST call, pretty soon you will make further damage to the microservice which is already suffering. You should stop sending further request to the service after certain number of failures and resume sending requests after a while. You are right, I am talking about Circuit Breaker Pattern.
I have been after this for a while and recently implemented these two patterns in Spring boot microservice using Spring-Retry.
Concept is very simple, microservice A will make REST call to microservice B. If it fails, it will automatically retry 3 times. If Service B is still unable to process, fallback method will be called. After certain number of fallback method is execute in a given time frame, circuit will be opened. As a result Service A will not make any further REST call to Service B, reducing cloud resource usage, network bandwidth usage. Once reset time is over, circuit will be closed automatically allowing REST calls to Service B again.


Above log indicates for each request, our service retried 3 times (“called ShakyExternalService api/customer/name”) before executing the fallback method ( “returning name from fallback method”). Once fallback method is called 3 times in a period of 15 seconds, circuit was opened and further request to the api was served directly from fallback without trying to make API call. In the log you can see last 3 lines where fallback was executed directly.

Checkout my article on Media ( ) for source code and further details.

Deploy Microsoft Bot framework to AWS Lambda

I got very excited when I first saw Microsoft Bot Framework ( Microsoft has done a great job! You can deploy same bot to multiple channels such as MS Team, Slack, Facebook. But all the documents/tutorial I read recommend to deploy bot backend code to Azure. I have been working with AWS for few years but never used Azure before. So was looking for a solution to deploy the Nodejs backend to AWS lambda as I am more comfortable. I still need to create Azure account to create the bot and deploy to channels but that’s OK ( who knows in future I may migrate from AWS Lambda to Azure Function 🙂 )

I googled but did not find a solution which just works out of the box. So I decided to poke the framework code a bit and get it deployed to lambda.

Find out full story and source code on Media

Keep ngrok running forever on Raspberry Pi

Click to enlarge

If you are a Raspberry Pi developer you often need to access web server running on your Raspberry Pi from outside of your home network. For example on your mobile when you are not connected to home Wifi. There are different ways to let the traffic go to your Raspberry Pi from internet such as modify your home router settings and open port forward to your Pi or use HTTP tunnel to route traffic to your Pi. I prefer the later and ngrok is a great utility to achieve that.

But If you are not a paid user of ngrok then your ngrok session expires in 8 hours. You need to restart the ngrok process to obtain new address which is a pain if you want to access your web server from anywhere anytime. It would have been more convenient if you can run ngrok forever without worrying about session expiration every 8 hours.

Here I am going to show you how you can keep your ngrok session running forever.

Please note, this is a hack and probably you should be good if you are prototyping something for your own such as home automation etc. Please consider paid subscription of ngrok if you are thinking to use it in production environment or commercial use.

1. Install Ngrok on Raspberry pi

SSH into your Raspberry Pi or open terminal on your Raspberry pi and execute following commands one by one.

cd /home/pi

sudo wget

unzip unzip

sudo chmod +x ./ngrok

./ngrok help

If everything goes fine, you should see ngrok help instruction printed on your terminal.

2. Setup software

Copy both and file to /home/pi directory. Grant execution permission on both the files by running following commands on terminal

sudo chmod +x
sudo chmod +x

3. Setup cron job

We are going to setup our cron job to execute the shell script every hour at 59 minutes such as 10:59, 11:59.  If you want to change the schedule, please check _run_ngrok method in

Execute following command to open crontab editor

crontab -e

Then add below two lines at the end of the file, save and exit.

59 * * * * sh /home/pi/ &
@reboot sh /home/pi/ &

First line will execute the script at every 59 minutes and the second line to execute the script if your Pi is rebooted.

When the script is executed, it will create HTTP tunnel on the port you specified in the python program. Now it’s up to you what you are going to do with the ngrok address. I usually store it in a dynamoDB table so that I can find the latest address of my Raspberry Pi and connect to it from web. I am sure you will figure out something which works best for you.

I recommend you to have a look at the python code and read all the comments to better understand the changes you may need to do to get it working on your Raspberry Pi


This is a proof of concept to keep ngrok running for a long time without paying the monthly subscription. If you get a better hack to achieve the same please do share.

Happy coding !!!

Let Raspberry Pi’s talk using AWS IoT Shadow

During Christmas break I was planning to implement a smart door bell. A Raspberry Pi in the porch with a camera attached to it. Pi will capture the image of the person, recognize the face and if face matches, it will signal another Pi inside the house which will unlock the door. Sounds very cool right? But immediately I realized I need a way to communicate between two Raspberry Pis which are not physically connected.

I did some hobby projects with AWS IoT and thought that would be a very good candidate to act as a message hub (basically MQTT hub). The Pi at porch will send a signal by updating to AWS IoT Thing shadow. The other Pi inside house will listen for any delta changes in that Thing shadow. When there is a change and the command is to open the door, it will turn the door lock or the latch.

At the time of writing this blog, I have not finished that project but I established the communication between two Raspberry Pis. I attached a switch to one and a LED to another. Pressing the switch on A will turn the LED on on another Pi.

I thought this might be very helpful if want to do something similar and I created a video tutorial on this.

Click to zoom
Click to zoom



  "Version": "2012-10-17",
  "Statement": [
      "Effect": "Allow",
      "Action": [
      "Resource": [

Beginners Guide To Ethereum Smart Contract

Few months back I built a project, a proof of concept to streamline peer to peer vehicle sale using Ethereum blockchain and IoT ( check out my recent projects for more details on that ) which inspired many college students.

Since then I have been asked by many students how to get started with blockchain development. Some of them asked how to install required softwares to write smart contract in solidity, some of them asked how to build a web application (a.k.a  DApp) and interact with Ethereum blockchain. Instead of answering individual questions, I decided to create a video tutorial  to guide then step by step to write and deploy their first smart contract and Ethereum DApp.

Watch below youtube video.

Checkout my Github repo to download entire source code




AWS API Gateway Multi-Stage Deployment with Lambda Alias

Click on the image to zoom

The Cloud Computing World is moving to Serverless architecture for obvious reasons. There is no need to keep your servers running and paying for 24X7 where you application may be serving your customers 50% of a day. You should pay only for the time your application actually processing your customer’s requests.

When we talk about Serveless Architecture on AWS, the very first thing would come to your mind is Apigateway with Lambda ! You can build your REST APIs without running EC2 instances. That’s very cool. But when I first adopted this stack, I was wondering how I can handle my CI/CD. For example, during development phase I like to update my lambda functions several times without effecting my PROD REST APIs which are being used by my customers. I deployed two Apigateway stages?—?dev and prod but problem was they both point to same lambda function. As a result, when I was updating my lambda function during development, it was effecting prod APIs as well.

Click on the image to zoom

Coming from Elastic Beanstalk where I can create multiple environments for an application and controlling code promotion from DEV to PROD so that PROD APIs are updated only when I decide to promote code from DEV to PROD, I was looking for similar mechanism using Apigateway and Lambda where I can control when to publish lambda code to PROD.

After some research I found we can achieve same behavior using Lambda alias with versioning and Apigateway stage variables.

When you use versioning in AWS Lambda, you can publish one or more versions of your Lambda function. As a result, you can work with different variations of your Lambda function in your development workflow. Each version is immutable with unique ARN. In addition, you can create aliases for your lambda function pointing to different immutable version. You can think alias as logical name of you environment specific lambda function which underneath points to different versions. For more information on this topic read this.

Now instead of directly assigning a lambda function in Apigateway integration request, you can assign lambda alias where alias is a variable. The variable will be resolved from a value from stage variable. For example you configure lambda function in Apigateway as myLamnda:${stageVariables.lambdaAlias} and define lambdaAlias stage variable in each of your Apigateway stages.

Click on the image to zoom
Click on the image to zoom
Click on the image to zoom

With above configurations, when you hit APIs from prod stage ( say /prod/todos), Apigateway will send the request to myLambda:PROD alias for fulfillment. myLambda:PROD points to immutable version. In your development environment (say /dev/todos) Apigateway (stage=dev) will point to myLambda:DEV alias which points to $LATEST version of your lambda function. As a result when you update code for your lambda function, it will update $LATEST version and immediately you can test your API in dev stage of Apigateway (/dev/todos). But it will not impact your prod API as it points to myLambda;PROD alias which behind the scene points to a immutable version.

Click on the image to zoom

Now when time comes to promote your lambda code to production, you create a version of you lambda code (from $LATEST) and update PROD alias to point to new version. That’s it !

One important note?—?when you are using lambda aliases, you need to grant permission to Apigateway for each aliases.

Click on the image to zoom

If you are interested in giving it a try by yourself, I wrote some terraform scripts to help you get started. All the AWS resources you need for this exercise such as lambda function, IAM role, aliases, api gatway resources, states will be created for you just by executing terraform commands from your terminal. Visit my github repo here for detail instructions.


Integrate elastic beanstalk health check with slack using cloud watch alarm

Get notified on Slack through Cloud Watch Alarms whenever your elastic beanstalk application becomes unhealthy.

Click on the image to zoom

In this article I will explain step by step how you can get the entire ecosystem working.

Elastic Beanstalk ( EB ) let’s you deploy your application code and provision auto scaling group, elastic load balancing, security groups automatically for you. EB makes developer’s life easy and developers can focus on writing the core business functionalities without worrying about creating and maintaining the infrastructure on AWS.

EB let’s you integrate with Cloud Watch Alarms on various metrics of your EB application, for example , health status, healthy host counts etc.

Slack on the other hand, is the most popular collaboration tool used by many organizations to carry out operational tasks. We are going to marry Slack with Cloud Watch alarm to make a smart and automated monitoring system for our elastic beanstalk application. Whenever the health of elastic beanstalk application goes in degraded or severe, Cloud Watch alarm goes into alarm state and invokes a lambda function through SNS topic. Lambda function posts a message to a pre-selected slack channel using web hook.

Elastic Beanstalk + Cloud Watch Alarm + Slack = HAPPINESS !

Part – One ( Create The Slack App ) 

First thing first. We will first create our new slack app and make sure it can receive messages from external sources through web hooks.


Log into . If you don’t have any slack account, you can create one for free. Once you logged into the site, you will see a menu “Your Apps” on top right corner as shown in below screen shot.

Click on the image to zoom

Click on that link which will take you to your apps dashboard as in below screen shot.

Click on the image to zoom


Click on “Create an App” button on the page which will bring the wizard to create a new app.

Type the name you want for your new slack app and then choose the workspace from the dropdown. Then hit “Create App” button.

Click on the image to zoom


Your app should be created and you should see a page like below. Now you need to create a web hook to receive messages from external sources. On the left panel, under “Features”, you will see a link “Incoming Webhooks”. Click on the link. You should see Incoming Webhooks page. Click on “Activate Incoming Webhooks” toggle to turn it on.

Click on the image to zoom

You will be asked to select a channel in which you want to receive messages from external sources such as cloud watch in this tutorial.

Click on the image to zoom

Once you authorize, your web hook url should be created and details will be shown on screen.

Click on the image to zoom


Now as the web hook is created, let’s test it out. On the web hook page, you will see a curl command and “Copy” button next to that. Hit the “Copy” button which will copy the entire curl command to clipboard.

Open the terminal window ( on Mac) or PowerShell ( On Windows) and paste the command and hit enter.

Click on the image to zoom

If everything goes fine, you should receive a message “Hello, World!” on your slack channel.

Part – Two ( Create The Beanstalk Application) 


I am not going to deep dive how to create a beanstalk application from scratch.  I will write a separate blog on this topic soon. For now, you may follow below article to create and deploy a spring boot application.

In this tutorial, I am using spring boot application. You can use any other language/framework you are familiar with which beanstalk supports. For example, PHP, Python or NodeJS.

For this tutorial I created one sample spring boot project which you can clone and build on your machine. Follow below commands which will clone, build and run the spring boot application running on port 5000.

 git clone && cd springboot-aws-ebs

 ./gradlew clean build

 java -jar build/libs/springboot-aws-ebs-0.0.1-SNAPSHOT.jar
Click on the image to zoom

Once your spring boot app is launched, issue below curl command which should return HTTP status code 200.

 curl -X GET -i http://localhost:5000/health 

Terminate the application by pressing Ctrl+D or closing the terminal window.  Note that the application jar file is created under build/libs which you will deploy to beanstalk.


Now upload this jar file to the beanstalk application you created in step 1 following the link above.  I have created a beanstalk application using same jar as shown in below screen shot.

Click on the image to zoom

Now let’s test the deployed application. This application has one API /health which should return HTTP status 200. Copy the beanstalk application url and replace your application url in below curl command.

 curl -X GET -i

The curl should return HTTP status code 200

Click on the image to zoom


Now we will modify the health check url of beanstalk application to use this API.

Click on the image to zoom

Click on the “Configuration” menu from left pane. Then click “Modify” link of “Monitoring” card.

Type “/health” for health check path and hit “save” button.

Click on the image to zoom

Part – Three ( Create Lambda function and Cloud Watch alarm) 


In this step, we will create one SNS topic. Log into AWS console and navigate to SNS dashboard (Services -> Application Integration -> Simple Notification Service). On the dashboard, click on “Create Topic” link. This will open a wizard. Type “springboot-sns” as the topic name and hit “Create” button.

Click on the image to zoom

We will not create any subscription right now. We will come back to this topic later and create a subscription with lambda endpoint.


In this step we will create a cloud watch alarm for our spring boot application. Navigate to cloud watch (Services -> Management Tools -> Cloud Watch)

On the cloud watch console, click on the “Create Alarm” button which will open a wizard to create new alarm.  On “1. Select Metric” tab, click on Environment Metric under ElasticBeanstalk, then select your spring boot application environment.

Click on the image to zoom
Click on the image to zoom


Then hit “Next” button.

On “Define Alarm” tab, type name for the alarm as “springboot-env-health”.

Click on the image to zoom

Scroll down and configure two alarms as below. One for state ALARM and another for state OK. Select the SNS topic we created in last step from the dropdown.

Click on the image to zoom


In this step, we will create our lambda function which will push message to the slack channel through the webhook url. Navigate to lambda console ( Services -> Compute -> Lambda )

On lambda dashboard, hit “Create Function” button.

Click on the image to zoom

Choose “Author from scratch”. Type “springbootslack” as name, choose NodeJS 6.10 as runtime and choose appropriate role. If you don’t have any existing role, choose “Create a custom role” from dropdown which will take you to IAM console where you can create a new role. This role will be assumed by your lambda function.

Finally hit “Create Function” button. Then copy and paste below code in lambda inline code editor.

Note: Change the path value at line#65. You will get the value from web hook url you created in part one.

Now hit “Save” button.

var https = require('https');
var util = require('util');

exports.handler = function(event, context) {
console.log(JSON.stringify(event, null, 2));
console.log('From SNS:', event.Records[0].Sns.Message);

var postData = {
    "channel": "#aws-sns",
    "username": "AWS SNS via Lamda :: Cloud",
    "text": "*" + event.Records[0].Sns.Subject + "*",
    "icon_emoji": ":aws:"

var message = JSON.parse(event.Records[0].Sns.Message);
var severity = "good";

if(message.NewStateValue =='OK'){
    severity = "#36a64f";
}else if(message.NewStateValue =='ALARM'){
    severity = "#ff0000";
    severity = "warning";

console.log('reason',message.NewStateReason );
//console.log('reason2',JSON.parse(message).NewStateReason );
postData.attachments = [
    "color": severity,
    "text": message.NewStateReason,
    "fields": [
        "title": "EnvironmentName",
        "value": message.Trigger.Dimensions[0].value,
        "short": true
        "title": "Metric",
        "value": message.Trigger.MetricName,
        "short": true
        "title": "Threshold",
        "value": message.Trigger.Threshold,
        "short": true
        "title": "Period",
        "value": message.Trigger.Period,
        "short": true
    "footer": "",
    "footer_icon": "",
    "ts": parseInt((new Date(message.StateChangeTime).getTime() / 1000).toFixed(0))

var options = {
   method: 'POST',
   hostname: '',
   port: 443,
   path: '/services/XXXXX/YYYYYYYY'

var req = https.request(options, function(res) {
   res.on('data', function (chunk) {

req.on('error', function(e) {
   console.log('problem with request: ' + e.message);

req.write(util.format("%j", postData));


You have successfully created your lambda function. Before moving forward, let’s first test out the lambda function.

Click on the image to zoom

Click on “Test” button next to “Save” which will bring up the test wizard. Type the event name “DymmyTest” and then copy & paste below JSON inside bottom text area. Click on “Create”.

"Records": [
   "EventSource": "aws:sns",
   "EventVersion": "1.0",
   "EventSubscriptionArn": "arn:aws:sns:us-east-1:xxxxx:slack-notification:71e7e166-eb12-46ad-954d-xxxxxx",
   "Sns": {
   "Type": "Notification",
   "MessageId": "5be3bf2d-031c-5824-9a96-ec538105f18b",
   "TopicArn": "arn:aws:sns:us-east-1:378123733884:slack-notification",
   "Subject": "ALARM: \"cloudwatch-alarm\" in US East (N. Virginia)",
   "Message": "{\"AlarmName\":\"cloudwatch-alarm\",\"AlarmDescription\":\"OK or INFO\",\"AWSAccountId\":\"xxxxxx\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [15.0 (22/05/18 20:37:00)] was greater than or equal to the threshold (5.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2018-05-22T20:38:49.939+0000\",\"Region\":\"US East (N. Virginia)\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"EnvironmentHealth\",\"Namespace\":\"AWS/ElasticBeanstalk\",\"StatisticType\":\"Statistic\",\"Statistic\":\"AVERAGE\",\"Unit\":null,\"Dimensions\":[{\"name\":\"EnvironmentName\",\"value\":\"ereg3-lov-np-test\"}],\"Period\":60,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanOrEqualToThreshold\",\"Threshold\":5.0,\"TreatMissingData\":\"\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
   "Timestamp": "2018-05-22T20:38:49.969Z",
   "SignatureVersion": "1",
   "Signature": "NEcGzhvdHgNM9MlkXQ8kTLZ5mRoNMvwC7ccCQrrU3pZU2wQRLBTHnGYuGJ00w51FIYpXyD2FU0I+4E99wZorNmVJDXFypZRfA7qImQDzth//4Of5CcCvY8Tvp1gpw4w2HoIzQLYRSqq7d/znn5Aj2OM5+MUoVVCvmoPKQVfTBT0adn925ke3IeQXII6TKvV0I+tg2+IEzbykIE/CM7N5VRswVH7LCl/rpR9jaifKdvoiGEQSEWjTFlJmyade7amNTLxszIsrFzw83aowAeTYAVjEv3JA3Tr441b9wrhmWMbr/vZ2utGl04a1/j4aec1AnnIIJJ4mZ9+8N1U66Yar4Q==",
   "SigningCertUrl": "",
    "UnsubscribeUrl": "",
   "MessageAttributes": {}

Click on the image to zoom

Now, on the lambda function page, you should see “DummyTest” selected next to “Test” button. If not, select that from the dropdown and hit “Test” button. If you have followed all the steps, you should receive a message on your slack channel like below.


Now as you created your lambda function and tested it successfully, it’s time to hook it up to the SNS topic we created in step#1. Navigate to the SNS console and open the topic we created earlier.

On the topic detail page, click on “Create subscription” button which will bring up another wizard.

Click on the image to zoom

Choose “AWS Lambda” as protocol and then choose the lambda function “springbootslack” we created earlier. Leave version as default. Click on “Create subscription” button.

Part – Four ( Testing ) 


Congratulations! You have successfully wired up all the pieces together and it’s to test out the entire system we just built. To test it, we need to make our spring boot application unhealthy. There are many ways you can achieve this but I am going to show you a very easy and clean approach without needing you to re-deploy the jar file.

Open your beanstalk application on aws console and click on “Configuration” from left pane. Then click on “Modify” link of “Software” card.

Go towards the bottom of the page. You will see some environment  properties of you beanstalk application. Add following property and hit “Apply”.




Click on the image to zoom

Above change will effect /health API and will return HTTP 500 instead of HTTP 200.  So this will make health checks to fail and will trigger cloud watch alarm. You should receive a message on your slack channel after 5 minutes notifying you that your spring boot application became unhealthy.

Now, go back to the Configuration -> Software and change the value from INTERNAL_SERVER_ERROR to OK. Click on “Apply” button. This change will restore /health API causing it to return HTTP 200. Wait for 5-7 minutes and you will be notified on slack that your application health has changed from alarm to ok.

Click on the image to zoom


You have automated your application health monitoring. No need to periodically check your application, you will be notified right on your smart phone whenever health status changes from OK to ALARM or vice versa.

If you like this tutorial, please hit like button and share with others. If you have any issue, please let me know.


Build your next app on aws server-less architecture

Don’t spin up servers anymore to deploy your applications. Servers are costly and involves regular maintenance and security patching to keep your servers safe from hackers which means additional pay checks for IT support. This adds up operational cost and burden on startups where fund is limited most of the cases.

Moreover during initial phase of a startup product  ( or a new product from a popular giant) you don’t know how much traffic your product is going to attract. Your product may get occasional traffic through out the day but as you have provisioned your servers, you are paying for 24 hours, 7 days.

Click on the image to zoom

Server-less architecture gives you the flexibility to pay only for the resources used by your application without running a fleet of EC2 containers, Elastic Load Balancers when most of the time they sit idle.

Use Amazon API Gateway which is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. Amazon API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, authorization and access control, monitoring, and API version management. Amazon API Gateway has no minimum fees or startup costs. You pay only for the API calls you receive and the amount of data transferred out.

Use AWS Lambda for your backend which lets you run code without provisioning or managing servers. You pay only for the compute time you consume – there is no charge when your code is not running. Lambda supports variety of popular programming languages such as NodeJS, Java, Python, C# and Go.

Use AWS DynamoDB for your storage which is a fully managed NoSQL database for all applications that need consistent, single-digit millisecond latency at any scale. It is a fully managed cloud database and supports both document and key-value store models.

Use AWS S3 to host your static web assets which is highly available and durable object storage. Your assets are automatically replicated to multiple availability zones within same region for high availability. Use AWS CloudFront to serve your static web content from S3 bucket. Amazon CloudFront is a global content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to your viewers with low latency and high transfer speeds. CloudFront offers a simple, pay-as-you-go pricing model with no upfront fees. CloudFront can server up to 10,000 request per seconds.

OfficeBot ( ) was built entirely on server-less architecture as described in above diagram.