September 28, 2018

LAB 13: Create a pipeline using AWS CodePipeline for Lambda deploy

This Lab has the step by step instructions to create a CI CD pipeline for a Serverless Application Deployment - Lambda using AWS CodeCommit, AWS CodeBuild, AWS Lambda, AWS CodeDeploy & AWS CodePipeline

Ensure to complete LAB 9 through LAB 12 including pre-requisite as stated in AWS Developer Tools CI CD Lab blog.

Login to AWS Console as `codedeploy_user`, go to AWS CodePipeline and create a pipeline with the following configuration

Create a Pipeline














Add Approve & Execute Stage

Once the above pipeline is created, edit to add 2 more stages as ApproveChangeSet and ExecuteChangeSet


Select the ARN of the SNS topic you have created earlier in this Lab series

Run the Pipeline

We have 5 stages as Source, Build, Staging, ApproveChangeSet and ExecuteChangeSet.
After the Build & Staging (after creating the CloudFormation ChangeSet), the pipeline is waiting on the manual approval.

Approval process

Based on our email configured in SNS Topic: Subscription, email notification will be sent as follows and the user can launch the URL provided for Approve or Reject and approve it.




Once the Manual approval is completed, ExecuteChangeSet stage will be executed and the Lambda function is created through CloudFormation template.






September 27, 2018

LAB 12: AWS CI CD Create Service Role for CloudFormation Deploy & Lambda

In this Lab, you will be creating a Service Role to do the CloudFormation deployment and Lambda Execution 

Ensure to complete LAB 9 through LAB 11 including pre-requisite as stated in AWS CI CD - 3rd Workflow - Serverless Application Deployment Lab blog
Login to AWS Console as `codedeploy_user` & go to AWS IAM dashboard -> Roles -> Create Role, select AWS Service as the type of trusted entity and proceed with the below configuration to create the role named CloudFormationServiceRole




Create an in-line policy CFN_Lambda_CICD_Policy with the policy document from Github.

Create Lambda Service Role


Create an in-line policy policy_LambdaWorker with the policy document from Github.


LAB 11 - AWS CI CD - Setup a Build Project in AWS CodeBuild for Lambda

This Lab has the step by step instructions to create & configure a Build using AWS  CodeBuild, which will compile the latest commit of the Lambda function, generate the artifact and uploaded to the named S3 bucket.

Ensure to complete LAB 9 through LAB 10 including pre-requisite as stated in AWS CI CD - 3rd Workflow - Serverless Application Deployment Lab blog.
Please note that for the purpose of this example, the Serverless Application is using demo-4-codedeploy as the bucket name. Make sure to edit that with your S3 bucket name before you proceed further.

Login to AWS Console as `codedeploy_user` & go to AWS CodeBuild from AWS Console, click Create Project and proceed with the configuration as follows

Build Source


Build Environment


Storying Build Artifacts


Enabling Build Logs




Save the Build & Run. Verify that the artifact is uploaded to S3 Bucket.

buildspecl.yml

Here is the closer look of buildspec.yml used in this example. In addition to this, you can environment variables as stated below

<add buildspec.yml>


Debugging CodeBuild

  • AWS CodeBuild is a managed service, which uses pre-defined container images for handling builds. Hence you will not have access to the container images running your build.
  • Local Build Support is enabled for AWS CodeBuild since May 2018, which you can use for debugging issues locally.
  • Enabling Logging to CloudWatch or S3 is key to monitor the build progress from AWS Management Console and for debugging in case of failures

LAB 10: Setup Repository in AWS CodeCommit for Lambda

This Lab has the step by step instructions to setup a Repository in AWS CodeCommit, add the pre-developed Lambda Application Code and commit the Changeset
Ensure that you have already went through the AWS CI CD using AWS Developer Tools & create IAM User, Group and S3 bucket before we move forward.
  1. Login to AWS Console as `codedeploy_user` & go to AWS CodeCommit dashboard
  2. Create an empty repository ascodecommit_lambda and you may Skip SNS Topic as of now and view the Connect to your repository screen

  1. 3. Get the clone URL (say git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/codecommit_lambda), paste run it on your terminal. It will ask for username and password, enter the Git Credentials you have generated in Lab-9You will get a warning that you appear to have cloned an empty repository and you are good to proceed.

  2. Download the Serverless Application source code from here and commit to your Git Repo using the commands as in below command line reference
  3. $ git clone <git URL>

                $ git init
     
                $ git status
  4. $ git add .
  5.                   $ git commit -m "Adding Serverless Application to CodeCommit Repo"
                      $ git push
                      $ git status

Now your source code is in git and may use the below command to commit the changes, if any, to the source code
  • git status - To view the changes 
  • git add . - to add all changes to the local repo and use git add <folder[/file]> to add selected folder or file to the local repo
  • git rm . - to add all changes to the local repo and use git rm <folder[/file]> to add selected folder or file to the local repo
  • git commit -m “what changes are getting in”
  • git commit --amend - will open the editor for you to modify the commit. Commit can be modified from command line as well using git commit --amend -m “New Commit”
  • git push - push the changes to the remote
  • git log -2 - gives the last two commit
  • git  branch -lvv - gives the local branches with remote mapping
  • git checkout <branchname> - to checkout from a remote branch
  • git branch -D <branchname> - to delete a local branch
  • git reset --hard <branchname> - to do a hard reset to the head of the revision from the remote - This is helpful if your local branch is messed-up and you want to overwrite everything from remote.

LAB 9 - AWS CI CD - IAM - Creating CodeDeploy Group & User

  1. In this Lab, we will be creating an IAM User under a new Group from IAM, which will be used hence forth for this demo to access AWS CI CD Services (AWS CodeCommit, AWS CodeBuild, AWS CodeDeploy & AWS  CodePipeline both programmatically & through AWS Console.

  2. Login to AWS Console as Root User, go to IAM -> Users and enter the details as follows to create codedeploy_user and set the Custom Password, which you can use later to login via IAM Console
    [Note: You will have to uncheck Required password reset option if you do not want to create a new password at the next sign-in.]




  3. Next in permissions screen, Create a Group as codedeploy_group  attach policy by clicking Permissions tab to add the following permissions
  4. AWSCodeCommitFullAccess
    IAMSelfManageServiceSpecificCredentials
    CloudWatchFullAccess
    AWSCodeDeployFullAccess
    IAMReadOnlyAccess
    AWSCodePipelineFullAccess
    AWSCodeBuildAdminAccess
  5. And add an in-line custom policy for the following hand picked policies                          {
    "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Stmt1537596486000",
                "Effect": "Allow",
                "Action": [
                    "iam:CreateRole",
                    "iam:AttachRolePolicy",
                    "iam:CreatePolicy",
                    "iam:CreatePolicyVersion",
                    "iam:DeletePolicy",
                    "iam:DeletePolicyVersion",
                    "iam:PutRolePolicy",
                    "events:PutRule",
                    "events:PutTargets",
                    "cloudformation:DescribeStackEvents",
                    "cloudformation:GetStackPolicy",
                    "cloudformation:DeleteStack ",
                    "apigateway:GET"
                ],
                "Resource": [
                    "*"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject",
                    "s3:PutObjectAcl",
                    "s3:GetObject",
                    "s3:GetObjectAcl",
                    "s3:DeleteObject"
                ],
                "Resource": "arn:aws:s3:::demo-4-codedeploy/*"
            }
        ]
    }

  6. Please note that the following 3 policies were required for CodeCommit as stated in prerequisite section of Connect to your repository and the remaining policies are required for the respective services.
    AWSCodeCommitFullAccess   
    IAMSelfManageServiceSpecificCredentials  
    IAMReadOnlyAccess 
    Note: This gives access to all repositories in your CodeCommit. If you want to restrict access, you need to write Custom Policies. Same applies to all services where unrestricted access is given. 

  7. Now add codedeploy_user to codedeploy_group 
  8. and then make sure to download the user credentials as this is the last time you will be able to access it Credentials, else you will have to regenerate the keys later.

  9. Get back to the user codedeploy_user and create Git credentials for this user Under HTTPS Git Credentials for AWS CodeCommit.
  10. Click Generate button and download credentials  

AWS CI CD - 3rd Workflow - Serverless Application Deployments

3rd Workflow in AWS CI CD using AWS Developer Tools Lab Series is about Serverless Application Deployment that can be taken-up independently as the material have been scripted from scratch with step by step instructions.  AWS CodeCommit, AWS CodeBuild, AWS CodeDeploy & AWS CodePipeline

Serverless Application is one or more Lambda functions, which is basically a compute service, and has triggering events like CloudWatch, S3 or DynamoDB etc, associated with a Lambda function. Deploying of Lambda functions along with all of the associated resources, as a bundle is termed as serverless deployments


Serverless Application is one or more Lambda functions, which is basically a compute service, and has triggering events like CloudWatch, S3 or DynamoDB etc, associated with a Lambda function. Deploying of Lambda functions along with all of the associated resources, as a bundle is termed as serverless deployments


Serverless Application is one or more Lambda functions, which is basically a compute service, and has triggering events like CloudWatch, S3 or DynamoDB etc, associated with a Lambda function. Deploying of Lambda functions along with all of the associated resources, as a bundle is termed as serverless deployments
Serverless Application Deployment


  • In this workflow, a simple Lambda function using python is sourced from Codecommit repository, configured to ‘Codebuild’ service, for compiling and unit testing phase and then based on the buildpsec.yml(which is input for codebuild) the code repo for Lambda, is packaged and zipped into the relevant S3 bucket location configured
  • Based on the application framework chosen, JUnit, Unittest or other tests can be configured to run in CodeBuild stage
  • ‘aws cloudformation package’ specified in build phase of buildspec.yml converts the SAM template into CloudFormation for Create or replace stack
  • SAM template is basically an extension of Cloudformation template and only difference between these two is the ‘AWS Transform: serverless’ in SAM template, which defines the Lambda specific instructions
  • Next, a ‘Manual approval’ stage with the SNS topic is configured. The approver will get notified via email with a link to ‘Approve or Reject’
  • Once approved, the Cloudformation ‘Execute change set’ action is triggered from next stage, which actually creates all resources bundled with Lambda
  1. LAB 9: Prerequisites Setup
    1. Create IAM Group & User
    2. Create CloudWatch Log group named codedeploy_logs
    3. Create S3 Bucket named demo-4-codedeploy
          Cross-verify the configurations defined in the LAB 9 against the below mind-map.
  2. LAB 10: Setup Repository in AWS CodeCommit
       
    Cross-verify the configurations defined in the LAB 10 against the below mind-map.
  3. LAB 11: Setup a Build in AWS CodeBuild
        Cross-verify the configurations defined in the LAB 11 against the below mind-map.
  4. LAB 12: Create a pipeline using AWS CodeDeploy
        Cross-verify the configurations defined in the LAB 12 against the below mind-map.
  5. LAB 13: Create a pipeline using AWS CodePipeline
        Cross-verify the configurations defined in the LAB 13 against the below mind-map.
  6. View AWS CI CD - Serverless Application Deployments Workflow

AWS CI CD - Reference Links

Quick reference guide to the materials used in CI CD Lab using AWS Developer Tools!

MindMap 

  1. View AWS CI CD - Application Deployments Workflow
  2. View AWS CI CD - Infrastructure Rollouts Workflow
  3. View AWS CI CD - Serverless Application Deployments Workflow

Application Source Code

Please refer slide deck used to present CI / CD using AWS Developer Tools during AWS Community Day Bengaluru 2018.

September 26, 2018

LAB 8: AWS CI CD - Add Infra Source and Deploy for CloudFormation to Existing Pipeline

This Lab will focus on taking the pipeline you have created in Lab 5 Create a pipeline using AWS CodePipeline CodePipeline to the next level where you will spin-up the EC2 instance using CloudFormation Template and deploy the Java Web application on it.
get the source for spinning up EC2 from codecommit_infra_demo repo and create CodeDeploy application to deploy the CloudFormation template.

Login to AWS Console as `codedeploy_user`, go to AWS CodePipeline and edit the existing pipeline with the following configuration

Edit the Pipeline - codepipeline_demo

Add New ActionSource


Add New Deploy under Staging



Run the Pipeline