Amazon Kinesis Workflow

Let's build Amazon Kinesis sample workflow with Amazon Kinesis Agent on EC2 Instance, Amazon Kinesis Stream, Amazon Kinesis Firehouse and Amazon S3 bucket.

Read Amazon Kinesis Data Streams key-concepts before proceeding further.

Amazon Kinesis Workflow

Step by Step Instructions:

• Go to AWS console; choose a region say - N.Virginia
• Create Kinesis data stream as data_stream with 1 shard and leave rest as default
• Create a Kinesis Forehose delivery stream named data_delivery with 
• source as Kinesis data stream - data_stream, created in previous step
• destination as S3 bucket - kinesis-destination
  Note: You can choose to e
• Create an EC2 instance and setup JDK, JAVA_HOME and follow the instructions in Amazon Kinesis Agent Setup guide.
• create a log file as /opt/applogs/web.log add some text
• configure /etc/aws-kinesis/agent.log file with kinesis endpoint, firehose endpoint, file to be streamed and kinesisstream name 
• kinesis.endpoint - kinesis.us-east-1.amazonaws.com
• firehose.endpoint - firehose.us-east-1.amazonaws.com
• filepattern - /opt/applogs*
• kinesisstream - data_stream
• sudo service aws-kinesis-agent <start / stop> start or stop the kinesis streaming agent.
  Once you start the agent, you will see that dat flowing through Kinesis Stream to Kinesis firehouse and then finally getting stored in the destination bucket

Cleanup:

• Delete S3 Bucket - kinesis-destination
• Delete Kinesis Forehose delivery stream named data_delivery 
• Delete Role - firehose_delivery_role
• Delete Kinesis data stream - data_stream
• Delete EC2 instance & the corresponding role, if you have created them as part of this workflow