Asynchronous tasks management made simple: AWS SQS with AWS Lambda

Background

In the fast-paced realm of modern technology, we often find ourselves dealing with tasks that can’t be done in the blink of an eye.

Visualize running a popular e-commerce platform where customers place orders or a social media application where users upload and process media files. These tasks take time, and your users won’t wait indefinitely. This is where the magical combination of AWS SQS and Lambda steps in to save the day.

Let’s have a real-world journey to explore how AWS SQS and Lambda can supercharge your application’s performance! We will implement this collaboration using Golang.

Sponsored

We are what we repeatedly do. Excellence, then, is not an act, but a habit. Try out Justly and start building your habits today!

The Brave Amazon Warriors: AWS SQS and Lambda

AWS SQS: Our first hero

It’s like a reliable courier service that transports messages between different parts of our application. Whether it’s payment processing, order confirmation, or inventory updates, SQS is our messenger. Think of it as a conveyor belt for tasks.

AWS Lambda: Our second hero

Lambda is like a team of skilled workers ready to spring into action. When a message arrives via SQS, Lambda is triggered to execute the necessary code.

These workers can be assigned specific tasks like,

• Processing payments
• Sending emails or
• Updating inventory

The beauty is that Lambda workers automatically adapt to the workload, like a team of superheroes ready to save the day.

The Battle Plan: AWS SQS and Lambda in Action

Step 1: Create an SQS Queue

Our first mission is to create an SQS queue to handle incoming messages.
The messages will be placed into the SQS queue from any executing program(It can be Lambda, too).

Visit the AWS Management Console, navigate to SQS Service, and click on Create Queue. If you want to provide SQS access to specific users, modify the access policy accordingly.

By that, We have our messaging system ready in place.

P.S. You can choose between Standard and FIFO queues as per the requirement.

Step 2: Crafting the Lambda Functions(SQS message sender and processor)

Our Lambda functions are like specialized blacksmiths who craft different components of our e-commerce machinery.

Here, we will create two Lambda functions. One is to place a message in a Queue and another is to process a message from Queue.

Let’s create our message sender Lambda function first.

• Create a Lambda function “AddMessageToQueue” to add a message in the SQS Queue

In the AWS Management Console, navigate to the AWS Lambda service, and click Create Function.

P.S. — We have chosen Go 1.x as an environment for our Lambda function, feel free to opt for the environment of your choice!

• Add Code to the Lambda function and point out the Lambda handler

sqs_message_sender/main.go

package main

import (
 "context"
 "fmt"
 "strconv"

 "github.com/aws/aws-lambda-go/lambda"
 "github.com/aws/aws-sdk-go/aws"
 "github.com/aws/aws-sdk-go/aws/credentials"
 "github.com/aws/aws-sdk-go/aws/session"
 "github.com/aws/aws-sdk-go/service/sqs"
)

const (
 AWS_ACCESS_KEY_ID     = "AKXXXXXXXXXXXXXXXXXX"
 AWS_SECRET_ACCESS_KEY = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
 AWS_REGION            = "us-east-1"
 AWS_SQS_URL           = "https://sqs.us-east-1.amazonaws.com/XXXXXXXXXXXX/sample.fifo"
)

func Handler(ctx context.Context) (string, error) {
 fmt.Println("I'm going to push a message to the SQS queue")

 // SQS message-sending logic goes here
 AddMessageToQueue(1, 2)

 return "Message sent successfully", nil
}

func main() {
 lambda.Start(Handler)
}

func AddMessageToQueue(productId, quantity int) {
 awsSession, err := GetAWSSession()
 if err != nil {
  panic(err)
 }

 // prepare SQS client
 sqsClient := sqs.New(awsSession)

 // prepare message attributes to push into queue
 messageAttributes := map[string]*sqs.MessageAttributeValue{
  "ProductId": {
   DataType:    aws.String("String"),
   StringValue: aws.String(strconv.Itoa(productId)),
  },
  "Quantity": {
   DataType:    aws.String("String"),
   StringValue: aws.String(strconv.Itoa(quantity)),
  },
 }

 // send message to the queue
 _, err = sqsClient.SendMessage(&sqs.SendMessageInput{
  MessageGroupId:         aws.String("productId" + fmt.Sprint(productId)),
  MessageDeduplicationId: aws.String("productId" + fmt.Sprint(productId) + "_" + fmt.Sprint(quantity)),
  MessageAttributes:      messageAttributes,
  MessageBody:            aws.String("Going to process queue for item : " + fmt.Sprint(productId)),
  QueueUrl:               aws.String(AWS_SQS_URL),
 })

 if err != nil {
  panic(err)
 }
}

func GetAWSSession() (*session.Session, error) {
 fmt.Println(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
 credential := credentials.NewStaticCredentials(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, "")

 _, err := credential.Get() /* get AWS credentials */
 if err != nil {
  return nil, err
 }

 cfg := aws.NewConfig().WithRegion(AWS_REGION).WithCredentials(credential) /* configure AWS account with credentials and region */

 session, err := session.NewSession(cfg)

 if err != nil {
  return nil, err
 }

 return session, nil
}

Create a zip of the binary of the above file and upload it as a code for the Lambda function.

Create a binary — GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o main main.go

Create a zip file — zip main.zip main

• Test the Lambda function by adding a message to the SQS Queue

Let’s move toward creating the SQS message processor Lambda.

• Create a Lambda function “ProcessQueueMessage”
  This function will be responsible for receiving messages from the SQS queue and will proceed further.

• Add code to the Lambda function and point out the lambda handler

sqs_message_processor/main.go

package main

import (
 "context"
 "fmt"

 "github.com/aws/aws-lambda-go/events"
 "github.com/aws/aws-lambda-go/lambda"
)

func main() {
 lambda.Start(ListenAndProcessSQSMessage)
}

// listens event from SQS(whenver any new message is placed in SQS Queue)
func ListenAndProcessSQSMessage(c context.Context, sqsEvent events.SQSEvent) {
 fmt.Println("Lambda is triggered by SQS")

 fmt.Println("sqsEvent : ", sqsEvent)

 data := make(map[string]interface{})

 for _, item := range sqsEvent.Records {
  for k, v := range item.MessageAttributes {
   data[k] = *v.StringValue
  }
 }

 fmt.Println("data : ", data) // map[ProductId:1 Quantity:2]
}

Again, create a binary of the above file, generate a zip of it, and upload it to the Lambda function.

Make sure you create a separate project for deploying the ProcessQueueMessage Lambda, because of two reasons:

1. Lambda uses the executable file name, for pointing out the lambda handler(main.go in our case) -> that we have already used for AddMessageToQueue Lambda function

2. For ProcessQueueMessage, in case of adding the lambda handler code into other file than of main.go, for example hello.go. It will be unable to find out the handler unless we add it inside main(). But inside a package main there can only be one main() method as per Go’s nature.

Therefore, it’s not possible for Golang in straight forward way, it’s possible for other languages though.
Refer: https://github.com/aws/aws-lambda-go/issues/182

Now, the Lambda function can be assigned a specific task, like processing payments, sending order confirmations, or updating inventory. Which is automatically executed when a message arrives in the SQS queue.

Step 3: Battle-Ready Permissions to Lambda function

It’s time to ask our brave soldiers to access the tool as per the requirements on the battlefield.

Whoa! I’m talking about pointing out our message processor Lambda function(ProcessQueueMessage) to tackle the order from the SQS Queue, whenever the Queue receives any message from the message sender Lambda function(AddMessageToQueue).

The error above shows that the lambda role is lacking permission to access SQS Service.
We need to make sure our Lambda functions have the right permissions to access the SQS queue. This is like equipping our blacksmiths with the right tools for the job.

Let’s now add the required permissions to the Lambda function role so that Lambda can easily access the SQS service.

Permission is all set. It’s time to add the lambda trigger to the SQS Queue.

Yayy! It worked.

Step 4: Victory Through Testing

With everything set up, we test our system to ensure it’s battle-ready. It’s like watching our army in training to defend our kingdom.

We send a test message to the SQS queue and observe our Lambda functions swing into action.

Let’s run AddMessageToQueue which will place the message to Queue and Queue will trigger the ProcessQueueMessage Lambda function.

Check Cloudwatch logs by navigating to the Cloudwatch service. You will see the logs as below:

AddMessageToQueue has placed our message with ProductId and Quantity to the SQS queue and the queue has passed it to the lambda function ProcessQueueMessage.

We can further process the passed message data from ProcessQueueMessage. For example to update the inventory in our case.

Congratulations!🎉, You have just learned a trick to optimize your project architecture by moving asynchronous operations to SQS.

Benefits of using SQS with AWS Lambda

As we journey through this process, we experience several real-world benefits.

1. Scalability

Just like during a massive shopping festival, our system can handle a surge in orders without breaking a sweat. We can dynamically adjust the number of SQS queues and Lambda functions to match our application’s needs.

2. Reliability

Messages in SQS are stored redundantly, and Lambda is a reliable, serverless service that takes care of error recovery and scaling. Our customers can trust that their orders will be processed without a hitch.

3. Simplified Architecture

We can focus on our core business because we don’t need to worry about server management and message queue maintenance. AWS handles these operational tasks, simplifying our architecture.

4. Cost Optimization

We’ve optimized our costs by configuring our resources efficiently while meeting our application’s demands. P.S. AWS Lambda works on the principle Pay as you go, that too as per the execution duration.

Finishing Thoughts

Background processing is an essential part of almost every development we do. It’s better to optimize the execution flow by separating the instant and time-consuming tasks to provide a delightful user experience.

So, whether you’re…

• Building a popular e-commerce website or
• Processing user-generated content on a social media platform or
• Managing any task that benefits from background processing or
• Building your favorite game application to reward users based on their scores

Consider the dynamic duo of AWS SQS and Lambda to simplify your architecture and supercharge your asynchronous task management.

This isn’t just a tech solution, it’s a real-world game-changer! You can adapt this collaboration for any of the cloud providers or any programming language you find flexible to use.

Similar Articles