Amazon SNS & SQS Simplified

A overview of Amazon SNS & SQS with introduction, creating configuring & testing sample workflow using Amazon SNS, SQS & S3 through AWS Console.

Amazon SNS

Amazon Simple Notification Service (SNS) is a highly available, durable, secure, fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems, and serverless applications. Amazon SNS provides topics for high-

throughput, push-based, many-to-many messaging. Using Amazon SNS topics, your publisher systems can fan out messages to a large number of subscriber endpoints for parallel processing, including Amazon SQS queues, AWS Lambda functions, and HTTP/S webhooks. Additionally, SNS can be used to fan out notifications to end users using mobile push, SMS, and email. 

Amazon SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. SQS eliminates the complexity and overhead associated with managing and operating message oriented middleware, and empowers developers to focus on differentiating work. Using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available. Get started with SQS in minutes using the AWS console, Command Line Interface or SDK of your choice, and three simple commands.
SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

What is Dead Letter Queue?

A dead-letter queue is a queue that other (source) queues can target for messages that can't be processed (consumed) successfully. In this tutorial you learn how to create an Amazon SQS source queue and to configure a second queue as a dead-letter queue for it. For more information, see Amazon SQS Dead-Letter Queues.

Important

When you designate a queue to be a source queue, a dead-letter queue is not created automatically. You must first create a normal standard or FIFO queue before designating it a dead-letter queue.

The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a standard queue must also be a standard queue.

 Simple Workflow - Amazon SNS & SQS:

Assume a scenario you are placing an order for a notebook, SNS sends an email to designated email address about the order request plus action to be taken and also adds the message to the orders content queue in SQS.

Setup SNS Topic

• Login to AWS Console & setup an SNS Topic named Order and two subscription for Email & Amazon SQS
• Create a Subscription using Email protocol, configure your email id and verify the email address by clicking the Subscription link received to your email
• Create a Subscription using Amazon SQS protocol, copy the ARN of the ContentQ that you will be creating under Setup SQS Queues

Setup SQS Queues

• Create a standard queue named ContentQ , leave everything to default and change message retention period to 1 day (range is 1 to 14 days).
• Go to Permission tab of the newly created Q enable Allow Everybody for All SQS Actions (SQS:*)
• Redrive policy will be empty at this point in time
• Create a standard queue named ContentRedriveQ , leave everything to default and change message retention period to 5 day (range is 1 to 14 days).
• No change to permission and redrive policy will be empty
• Go to ContentQ, enable Dead Letter Queue and select ContentRedriveQ, leave Maximum receives as 1.
• Testing SQS 
• Select ContentQ -> Queue Actions -> Send a Message -> Type a message and Click Send
• Select ContentQ -> Queue Actions -> View / Delete Messages -> Click Start polling for messages.  This action reads the message from the queue but does not send a confirmation of successful processing of the message. This turns to be failed or unprocessed category, which then falls into Dead Letter Queue. Now you will be able to see this message under ContentRedriveQ, which is mapped as a Dead Letter Queue of ContentQ.

Testing SNS & SQS Flow

• Go to Amazon SNS ->Topic -> Orders -> Publish Message -> Enter Subject & Message Body, leave rest to default and click `Publish Message`.
• This message will be delivered to your configured email via Email protocol subscription to Order topic and will be delivered to SQS - ContentQ via SQS protocol subscription to Order topic.
• Select ContentQ -> Queue Actions -> View / Delete Messages -> Click Start polling for messages.  This action reads the message from the queue but does not send a confirmation of successful processing of the message. This turns to be failed or unprocessed category, which then falls into Dead Letter Queue. Now you will be able to see this message under ContentRedriveQ, which is mapped as a Dead Letter Queue of ContentQ.

Enhanced Workflow: Amazon S3, SNS & SQS Events:

Now lets enhance the above workflow a bit by triggering message as part of S3 upload / put event. You will see how well the messages fan-out from SNS Topic to email & SQS queues.
Message fan-out is nothing but broadcasting messages from one to many.

Setup S3 Event

• Create an S3 bucket named ordersbucket, go to Properties -> Events -> Add Notification -> Enter Details for New Event
  Event Name: ForSNSNotificaiton, Events: PUT, Send to: SNS Topic, SNS: Orders and try to save
  You will get the following error as your S3 bucket do not have permission to SNS Topic
  Unable to validate the following destination configurations. Permissions on the destination topic do not allow S3 to publish notifications from this bucket. (arn:aws:sns:us-east-1:<account id>:Orders)
• To grant permission to the S3 bucket events to trigger SNS Topic, go to SNS Topic -> Orders -> Edit Access Policy ->
  replace the blue text by green 

"Condition": {
        "StringEquals": {
          "AWS:SourceOwner": "<account id>"
        }

"Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3::: ordersbucket"
        } 

• Now go back & Click Save for S3, it should work.
• This configuration itself triggers the S3 event, which would sent the below email to your configured email and a message to ContentQ

{  "Type" : "Notification",  "MessageId" : "f0c66789-543d-5527-9bfd-328a83cbd237",  "TopicArn" : "arn:aws:sns:us-east-1:544638597657:Orders",  "Subject" : "Amazon S3 Notification",  "Message" : "{\"Service\":\"Amazon S3\",\"Event\":\"s3:TestEvent\",\"Time\":\"2019-03-14T04:16:54.038Z\",\"Bucket\":\"ordersbucket\",\"RequestId\":\"37475229D2AF3B2E\",\"HostId\":\"csuC6wL1zh8now9VybrB8LUju2Nc1z1sF1C1TN0HU15tHR6KvI2x4lIBqrM9pMXiz5wgSQkgyYczk=\"}",  "Timestamp" : "2019-03-14T04:16:54.145Z",  "SignatureVersion" : "1",  "Signature" : "LBIZeIodCe6Y1Irx8ZBLifqrEPbUw+tEFAwVygDszoMDyKZSrMD5kwKsJ0kZzjuaXvOeYhdITIuwgWMNnrRJpLhH9EtlhbHV0g/GT/pDaNZb52JV6vRB8zO0de8DC2AVDgQ7TyxS7Vx6TuqBPuxsRX0mdD1H+UPxc3+1ory7UAXggcT0h7zKVQkT7BrT+9dJs8+RfUQ/1YODYNZCR0qJMHQIqUDbx4KR0KtuobZ+wTwT60hJgUYcM/13VL7cZgckMNGuYv8qNJc4hEwb591V8C5nnvx7JEksLJkP91PfQJsCzoGaGvh+UhDWmjVI6fHMZNo+zmyMe8I0sEw==",  "SigningCertURL" : "https://sns.us-east-1.amazonaws.com/SimpleNotificationService-6aad65c2f991xxxxxxxxda11f913f9.pem",  "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:<accountid>:Orders:925ec4b2-1333-4b69-b82b-569c1eb7dd94"}

Testing S3, SNS & SQS Flow

• To test the whole flow upload a file into ordersbucket, which would trigger S3 PUT Event -> SNS Topic - Orders and watch the rest to happen seamlessly
• This message will be delivered to your configured email with subject - Amazon S3 Notification via Email protocol subscription to Order topic and will be delivered to SQS - ContentQ via SQS protocol subscription to Order topic.

• Select ContentQ -> Queue Actions -> View / Delete Messages -> Click Start polling for messages.  This action reads the message from the queue but does not send a confirmation of successful processing of the message. This turns to be failed or unprocessed category, which then falls into Dead Letter Queue. Now you will be able to see this message under ContentRedriveQ, which is mapped as a Dead Letter Queue of ContentQ.

Cleanup

Let's clean the resources that we have created as part of this demo

• Delete S3 Bucket named ordersbucket
• Delete SNS Topic Subscriptions that you created for email protocol & SQS
• Delete SNS Topic Orders
• Delete SQS - ContentQRedrive & ContentQ
• You will get a waring that there are messages in the Queue, if you still want to delete. You can either choose to delete or purge the messages and then come back for deletion.