diff --git a/sqs-lambda-lmi-esm-provisioned-sam/README.md b/sqs-lambda-lmi-esm-provisioned-sam/README.md new file mode 100644 index 000000000..7ec0f1023 --- /dev/null +++ b/sqs-lambda-lmi-esm-provisioned-sam/README.md @@ -0,0 +1,111 @@ +# Amazon SQS + Lambda Managed Instances (LMI) with Provisioned Mode ESM + +This pattern deploys an Amazon SQS Standard Queue connected to an AWS Lambda function via an Event Source Mapping (ESM) configured with **Provisioned Mode** — an ESM feature that pre-allocates dedicated polling resources for predictable, high-throughput message processing. + +Learn more about this pattern at Serverless Land Patterns: [https://serverlessland.com/patterns/sqs-lambda-lmi-esm-provisioned-sam](https://serverlessland.com/patterns/sqs-lambda-lmi-esm-provisioned-sam) + +Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. + +## Requirements + +* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured +* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +* [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed + +## Build Instructions + +1. From the command line, use AWS SAM to build the AWS resources for the pattern as specified in the template.yml file: + ``` + sam build + ``` + +## Deployment Instructions + +1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository: + ``` + git clone https://github.com/aws-samples/serverless-patterns + ``` +2. Change directory to the pattern directory: + ``` + cd sqs-lambda-lmi-esm-provisioned-sam + ``` +1. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file: + ``` + sam deploy --guided + ``` +1. During the prompts: + * Enter a stack name + * Enter the desired AWS Region + * Accept the default parameter values or tune them for your workload + * Allow SAM CLI to create IAM roles with the required permissions. + + Once you have run `sam deploy --guided` mode once and saved arguments to a configuration file (samconfig.toml), you can use `sam deploy` in future to use these defaults. + +1. Note the outputs from the SAM deployment process. These contain the resource names and/or ARNs which are used for testing. + +## How it works + +1. A producer (any AWS service, SDK client, or CLI) sends messages to the SQS Standard Queue. +2. The ESM's provisioned event pollers continuously long-poll the queue using up to 10 SQS API calls per second per poller. +3. When messages are available, pollers assemble batches (up to `BatchSize` messages, waiting up to `MaxBatchingWindowInSeconds`) and invoke the Lambda function concurrently. +4. Lambda scales the number of active pollers between `MinimumPollers` and `MaximumPollers` based on queue depth, adding up to **1,000 concurrent executions per minute**. +5. If a message fails processing after 3 attempts (`maxReceiveCount`), it is moved to the Dead Letter Queue. +6. A CloudWatch Alarm fires as soon as any message lands in the DLQ. + +## Testing + +1. Get the queue URL from the stack outputs: + +```bash +aws cloudformation describe-stacks \ + --stack-name \ + --query "Stacks[0].Outputs" +``` + +2. Send test messages: + +```bash +QUEUE_URL= + +# Send a single message +aws sqs send-message \ + --queue-url $QUEUE_URL \ + --message-body '{"orderId": "123", "amount": 99.99}' + +# Send a batch of 10 +for i in $(seq 1 10); do + aws sqs send-message \ + --queue-url $QUEUE_URL \ + --message-body "{\"orderId\": \"$i\", \"amount\": $((RANDOM % 100))}" +done +``` + +3. Check Lambda logs: + +```bash +sam logs --stack-name --tail +``` + +4. Inspect the ESM status and poller count: + +```bash +ESM_ID= + +aws lambda get-event-source-mapping --uuid $ESM_ID +``` + +## Cleanup + +1. Delete the stack + ```bash + aws cloudformation delete-stack --stack-name STACK_NAME + ``` +2. Confirm the stack has been deleted + ```bash + aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'STACK_NAME')].StackStatus" + ``` +---- +Copyright 2025 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +SPDX-License-Identifier: MIT-0 diff --git a/sqs-lambda-lmi-esm-provisioned-sam/example-pattern.json b/sqs-lambda-lmi-esm-provisioned-sam/example-pattern.json new file mode 100644 index 000000000..594c92274 --- /dev/null +++ b/sqs-lambda-lmi-esm-provisioned-sam/example-pattern.json @@ -0,0 +1,68 @@ +{ + "title": "Amazon SQS to AWS Lambda with Lambda Managed Instances (LMI) Provisioned Mode ESM", + "description": "This pattern shows how to process Amazon SQS messages using AWS Lambda Managed Instances (LMI) with a Provisioned Mode Event Source Mapping for faster scaling and higher throughput.", + "language": "Python", + "level": "300", + "framework": "AWS SAM", + "introBox": { + "headline": "How it works", + "text": [ + "A producer sends messages to an Amazon SQS Standard Queue. The Event Source Mapping (ESM) is configured with Provisioned Mode, which pre-allocates dedicated event pollers between a configurable minimum and maximum. Each poller handles up to 1 MB/s throughput, 10 concurrent Lambda invocations, and 10 SQS API calls per second.", + "The Lambda function runs on Lambda Managed Instances (LMI) — an EC2-backed execution model that supports multiconcurrency, allowing each execution environment to handle multiple SQS batches simultaneously. LMI instances run inside a private VPC with outbound access via a NAT Gateway.", + "Partial batch failure reporting (ReportBatchItemFailures) ensures only failed messages are returned to the queue. After 3 failed attempts, messages are moved to a Dead Letter Queue. A CloudWatch Alarm fires as soon as any message lands in the DLQ.", + "This pattern deploys one SQS Standard Queue, one Dead Letter Queue, one Lambda function with an LMI Capacity Provider, one Event Source Mapping with Provisioned Mode, a VPC with private subnets and NAT Gateway, and one CloudWatch Alarm." + ] + }, + "gitHub": { + "template": { + "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/sqs-lambda-lmi-esm-provisioned-sam", + "templateURL": "serverless-patterns/sqs-lambda-lmi-esm-provisioned-sam", + "projectFolder": "sqs-lambda-lmi-esm-provisioned-sam", + "templateFile": "template.yaml" + } + }, + "resources": { + "bullets": [ + { + "text": "Using Lambda with Amazon SQS", + "link": "https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html" + }, + { + "text": "Lambda Managed Instances (LMI) overview", + "link": "https://docs.aws.amazon.com/lambda/latest/dg/managed-instances.html" + }, + { + "text": "Event Source Mapping Provisioned Mode for SQS", + "link": "https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#sqs-esm-provisioned-mode" + }, + { + "text": "Reporting batch item failures for SQS", + "link": "https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting" + } + ] + }, + "deploy": { + "text": [ + "sam build", + "sam deploy --guided" + ] + }, + "testing": { + "text": [ + "See the GitHub repo for detailed testing instructions." + ] + }, + "cleanup": { + "text": [ + "Delete the stack: aws cloudformation delete-stack --stack-name STACK_NAME." + ] + }, + "authors": [ + { + "name": "Serda Kasaci Yildirim", + "image": "https://drive.google.com/file/d/1rzVS1hrIMdqy6P9i7-o7OBLNc0xY0FVB/view?usp=sharing", + "bio": "Serda is a Solutions Architect at Amazon Web Services (AWS) based in Vienna with 3 years at AWS, specializing in serverless technologies and Event-Driven Architecture.", + "linkedin": "serdakasaci" + } + ] +} diff --git a/sqs-lambda-lmi-esm-provisioned-sam/src/handler.py b/sqs-lambda-lmi-esm-provisioned-sam/src/handler.py new file mode 100644 index 000000000..84e61491a --- /dev/null +++ b/sqs-lambda-lmi-esm-provisioned-sam/src/handler.py @@ -0,0 +1,19 @@ +import json +import logging + +logger = logging.getLogger() +logger.setLevel(logging.INFO) + + +def lambda_handler(event, context): + batch_item_failures = [] + + for record in event.get("Records", []): + try: + body = json.loads(record["body"]) + logger.info("Processing message", extra={"message_id": record["messageId"], "body": body}) + except Exception as e: + logger.error("Failed to process record %s: %s", record["messageId"], e) + batch_item_failures.append({"itemIdentifier": record["messageId"]}) + + return {"batchItemFailures": batch_item_failures} diff --git a/sqs-lambda-lmi-esm-provisioned-sam/template.yaml b/sqs-lambda-lmi-esm-provisioned-sam/template.yaml new file mode 100644 index 000000000..f27dc55af --- /dev/null +++ b/sqs-lambda-lmi-esm-provisioned-sam/template.yaml @@ -0,0 +1,365 @@ +AWSTemplateFormatVersion: '2010-09-09' +Transform: AWS::Serverless-2016-10-31 +Description: > + Amazon SQS (Standard) + Lambda Managed Instances (LMI) with Provisioned Mode ESM. + This pattern demonstrates how to use Lambda's Provisioned Mode for SQS Event Source + Mappings to achieve faster scaling and higher throughput for message processing workloads. + +Globals: + Function: + Runtime: python3.13 + Architectures: + - arm64 + Timeout: 30 + MemorySize: 2048 + Environment: + Variables: + POWERTOOLS_SERVICE_NAME: sqs-lmi-processor + LOG_LEVEL: INFO + +Parameters: + MinPollers: + Type: Number + Default: 2 + MinValue: 2 + MaxValue: 200 + Description: > + Minimum number of provisioned event pollers (2-200). Each poller handles up to + 1 MB/s throughput, 10 concurrent invokes, and 10 SQS API calls/sec. + + MaxPollers: + Type: Number + Default: 10 + MinValue: 2 + MaxValue: 2000 + Description: > + Maximum number of provisioned event pollers (2-2000). Lambda scales between + MinPollers and MaxPollers based on queue depth. + + BatchSize: + Type: Number + Default: 10 + MinValue: 1 + MaxValue: 10000 + Description: Maximum number of messages per Lambda invocation batch. + + MaxBatchingWindowSeconds: + Type: Number + Default: 5 + MinValue: 0 + MaxValue: 300 + Description: > + Maximum time (seconds) Lambda waits to fill a batch before invoking. + Set to 0 for lowest latency; higher values improve throughput efficiency. + + VpcCidr: + Type: String + Default: 10.0.0.0/16 + Description: CIDR block for the VPC + + SubnetAZ1Cidr: + Type: String + Default: 10.0.1.0/24 + Description: CIDR for private subnet in AZ1 + + SubnetAZ2Cidr: + Type: String + Default: 10.0.2.0/24 + Description: CIDR for private subnet in AZ2 + +Resources: + + # ----------------------------------------------------------------------- + # VPC — private network for LMI EC2 instances + # ----------------------------------------------------------------------- + VPC: + Type: AWS::EC2::VPC + Properties: + CidrBlock: !Ref VpcCidr + EnableDnsSupport: true + EnableDnsHostnames: true + Tags: + - Key: Name + Value: !Sub "${AWS::StackName}-vpc" + + # Private subnet in AZ1 + SubnetAZ1: + Type: AWS::EC2::Subnet + Properties: + VpcId: !Ref VPC + CidrBlock: !Ref SubnetAZ1Cidr + AvailabilityZone: !Select [0, !GetAZs ""] + Tags: + - Key: Name + Value: !Sub "${AWS::StackName}-subnet-az1" + + # Private subnet in AZ2 + SubnetAZ2: + Type: AWS::EC2::Subnet + Properties: + VpcId: !Ref VPC + CidrBlock: !Ref SubnetAZ2Cidr + AvailabilityZone: !Select [1, !GetAZs ""] + Tags: + - Key: Name + Value: !Sub "${AWS::StackName}-subnet-az2" + + # ----------------------------------------------------------------------- + # NAT Gateway — allows LMI instances to reach AWS APIs (SQS, Lambda, etc.) + # without exposing them to inbound internet traffic + # ----------------------------------------------------------------------- + EIP: + Type: AWS::EC2::EIP + Properties: + Domain: vpc + + InternetGateway: + Type: AWS::EC2::InternetGateway + + VPCGatewayAttachment: + Type: AWS::EC2::VPCGatewayAttachment + Properties: + VpcId: !Ref VPC + InternetGatewayId: !Ref InternetGateway + + # Public subnet for NAT Gateway (NAT GW must live in a public subnet) + PublicSubnet: + Type: AWS::EC2::Subnet + Properties: + VpcId: !Ref VPC + CidrBlock: 10.0.0.0/24 + AvailabilityZone: !Select [0, !GetAZs ""] + MapPublicIpOnLaunch: true + Tags: + - Key: Name + Value: !Sub "${AWS::StackName}-public-subnet" + + PublicRouteTable: + Type: AWS::EC2::RouteTable + Properties: + VpcId: !Ref VPC + + PublicRoute: + Type: AWS::EC2::Route + DependsOn: VPCGatewayAttachment + Properties: + RouteTableId: !Ref PublicRouteTable + DestinationCidrBlock: 0.0.0.0/0 + GatewayId: !Ref InternetGateway + + PublicSubnetRouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref PublicSubnet + RouteTableId: !Ref PublicRouteTable + + NatGateway: + Type: AWS::EC2::NatGateway + Properties: + AllocationId: !GetAtt EIP.AllocationId + SubnetId: !Ref PublicSubnet + + # Private route table — routes outbound traffic through NAT Gateway + PrivateRouteTable: + Type: AWS::EC2::RouteTable + Properties: + VpcId: !Ref VPC + + PrivateRoute: + Type: AWS::EC2::Route + Properties: + RouteTableId: !Ref PrivateRouteTable + DestinationCidrBlock: 0.0.0.0/0 + NatGatewayId: !Ref NatGateway + + SubnetAZ1RouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref SubnetAZ1 + RouteTableId: !Ref PrivateRouteTable + + SubnetAZ2RouteTableAssociation: + Type: AWS::EC2::SubnetRouteTableAssociation + Properties: + SubnetId: !Ref SubnetAZ2 + RouteTableId: !Ref PrivateRouteTable + + # Security group for LMI EC2 instances — no inbound, outbound to AWS APIs only + LMISecurityGroup: + Type: AWS::EC2::SecurityGroup + Properties: + GroupDescription: Security group for LMI EC2 instances + VpcId: !Ref VPC + SecurityGroupEgress: + - IpProtocol: tcp + FromPort: 443 + ToPort: 443 + CidrIp: 0.0.0.0/0 + Description: HTTPS outbound for AWS API calls + Tags: + - Key: Name + Value: !Sub "${AWS::StackName}-lmi-sg" + + # ----------------------------------------------------------------------- + # LMI Capacity Provider — manages the EC2 fleet that runs the function + # ----------------------------------------------------------------------- + CapacityProvider: + Type: AWS::Serverless::CapacityProvider + Properties: + CapacityProviderName: !Sub "${AWS::StackName}-capacity-provider" + VpcConfig: + SubnetIds: + - !Ref SubnetAZ1 + - !Ref SubnetAZ2 + SecurityGroupIds: + - !Ref LMISecurityGroup + InstanceRequirements: + Architectures: + - arm64 # matches the function architecture below + ScalingConfig: + MaxVCpuCount: 40 # max vCPUs across all EC2 instances; tune for your workload + AverageCPUUtilization: 70 # scale up when avg CPU exceeds 70% + Tags: + Pattern: sqs-lmi-provisioned-esm + + # ----------------------------------------------------------------------- + # Dead Letter Queue — receives messages that fail after all retries + # ----------------------------------------------------------------------- + DeadLetterQueue: + Type: AWS::SQS::Queue + Properties: + QueueName: !Sub "${AWS::StackName}-dlq" + MessageRetentionPeriod: 1209600 # 14 days + Tags: + - Key: Pattern + Value: sqs-lmi-provisioned-esm + + # ----------------------------------------------------------------------- + # Main SQS Standard Queue + # ----------------------------------------------------------------------- + MessageQueue: + Type: AWS::SQS::Queue + Properties: + QueueName: !Sub "${AWS::StackName}-queue" + VisibilityTimeout: 90 # Must be >= 6x Lambda timeout + MessageRetentionPeriod: 86400 # 1 day + RedrivePolicy: + deadLetterTargetArn: !GetAtt DeadLetterQueue.Arn + maxReceiveCount: 3 + Tags: + - Key: Pattern + Value: sqs-lmi-provisioned-esm + + # ----------------------------------------------------------------------- + # Lambda Function — runs on LMI (EC2-backed, multiconcurrency) + # ----------------------------------------------------------------------- + ProcessorFunction: + Type: AWS::Serverless::Function + Properties: + FunctionName: !Sub "${AWS::StackName}-processor" + CodeUri: src/ + Handler: handler.lambda_handler + Description: Processes SQS messages via LMI Provisioned Mode ESM + # Wire function to the LMI capacity provider. + # NOTE: CapacityProviderConfig must be set on first deploy — cannot be + # added or removed after the function resource has been created. + CapacityProviderConfig: + Arn: !GetAtt CapacityProvider.Arn + # Memory-to-vCPU ratio per execution environment: 2, 4, or 8 GiB/vCPU + # Must not exceed the function's total memory (512 MB here = 0.5 GiB, + # so set to 2 which is the minimum supported ratio) + ExecutionEnvironmentMemoryGiBPerVCpu: 2 + # Max concurrent invocations per execution environment (multiconcurrency) + # Each env can handle multiple SQS batches simultaneously + PerExecutionEnvironmentMaxConcurrency: 10 + Policies: + - SQSPollerPolicy: + QueueName: !GetAtt MessageQueue.QueueName + Tags: + Pattern: sqs-lmi-provisioned-esm + + # ----------------------------------------------------------------------- + # Event Source Mapping — SQS -> Lambda with Provisioned Mode (LMI) + # + # Key settings: + # ProvisionedPollerConfig — enables LMI Provisioned Mode + # FunctionResponseTypes — partial batch failure reporting + # ScalingConfig is NOT set — mutually exclusive with ProvisionedPollerConfig + # ----------------------------------------------------------------------- + SQSEventSourceMapping: + Type: AWS::Lambda::EventSourceMapping + Properties: + FunctionName: !GetAtt ProcessorFunction.Arn + EventSourceArn: !GetAtt MessageQueue.Arn + BatchSize: !Ref BatchSize + MaximumBatchingWindowInSeconds: !Ref MaxBatchingWindowSeconds + # Report partial batch failures so only failed messages return to queue + FunctionResponseTypes: + - ReportBatchItemFailures + # LMI Provisioned Mode — pre-allocates dedicated polling resources + # Enables 3x faster scaling and up to 16x higher processing capacity + # vs. on-demand mode. Cannot be combined with ScalingConfig.MaximumConcurrency. + ProvisionedPollerConfig: + MinimumPollers: !Ref MinPollers + MaximumPollers: !Ref MaxPollers + + # ----------------------------------------------------------------------- + # CloudWatch Alarm — alert when messages land in the DLQ + # ----------------------------------------------------------------------- + DLQAlarm: + Type: AWS::CloudWatch::Alarm + Properties: + AlarmName: !Sub "${AWS::StackName}-dlq-messages" + AlarmDescription: Messages arrived in the Dead Letter Queue + Namespace: AWS/SQS + MetricName: NumberOfMessagesSent + Dimensions: + - Name: QueueName + Value: !GetAtt DeadLetterQueue.QueueName + Statistic: Sum + Period: 60 + EvaluationPeriods: 1 + Threshold: 1 + ComparisonOperator: GreaterThanOrEqualToThreshold + TreatMissingData: notBreaching + +Outputs: + VpcId: + Description: VPC ID + Value: !Ref VPC + + SubnetAZ1Id: + Description: Private Subnet AZ1 ID + Value: !Ref SubnetAZ1 + + SubnetAZ2Id: + Description: Private Subnet AZ2 ID + Value: !Ref SubnetAZ2 + + LMISecurityGroupId: + Description: LMI Security Group ID + Value: !Ref LMISecurityGroup + + CapacityProviderArn: + Description: LMI Capacity Provider ARN + Value: !GetAtt CapacityProvider.Arn + + QueueUrl: + Description: SQS Queue URL for sending test messages + Value: !Ref MessageQueue + + QueueArn: + Description: SQS Queue ARN + Value: !GetAtt MessageQueue.Arn + + DLQUrl: + Description: Dead Letter Queue URL + Value: !Ref DeadLetterQueue + + ProcessorFunctionArn: + Description: Lambda Processor Function ARN + Value: !GetAtt ProcessorFunction.Arn + + EventSourceMappingId: + Description: ESM UUID — use this to inspect or update the mapping + Value: !Ref SQSEventSourceMapping