Blog on DevOps, Cloud Computing, SCM: Migration Story: Agile to FDD in light of AWS @ AWS Community Day 2017

AWS User Group Bengaluru had organized the first ever AWS Community Day in Bengaluru on 28th Oct 2017 which was an all-day event running two parallel tracks - proven use cases / success stories and workshops.

I also had an opportunity to present one of our companies most successful use-case "Migration Story: Agile to FDD in-light of AWS"

While moving from Agile to Feature Driven Development (FDD) with geographically distributed development centre, it is customary to have dedicated light environment per feature and robust automation to build & deploy at your own will!

The migration story was on Agile to FDD from a cloud based supply chain leader, GTNexus an Infor Company, that unfolds how the AWS Services came as a blessing in disguise to provide a highly scalable, Elastic & Cost Effective Solution to facilitate on-demand miniature development environments and independent Build & Deployment framework.

Outline

How GTNexus used Agile Scrum and the paradigm shift in the branching strategy when moved from scrum to Feature Driven Development
Development Cycle & the Timeline
High-level architecture of a full-fledged test environment hosted in DataCenter and the need for on-demand, scalable, miniature development environment for feature based testing in AWS.
Highly Elastic Build & Deployment Framework to cater the on-demand build & deployment needs of FTEs
While we were talking, our we had ~ 500 on-demand FTEs across 4 regions viz - Mumbai, US West - Oregon, Europe - Frankfurt & US East - North Virginia.
How useful was the move to AWS in terms of Scalability, Elasticity, Cost Efficiency and Security.

Miniature FTE in AWS

On the pursuit of light environment for Feature Based Testing, re-designed the QA Environment with a Single Windows & Linux VM Images with application services installed and housing the OS specific Data Services (say SQL Server in Windows; DB2, Riak/KVS & memsql in Linux).

After the light environment is setup in private cloud / Data Center, the next step is to setup this light Feature Test Environment in AWS. Windows & Linux VM Images are exported from Data Center & imported into AWS. A Feature Test Environment in AWS comprises of a VPC, Internet Gateway, Internet facing subnet, two EC2 instances that are replicated from the Data Center VMs. These two EC2 instances are stored as pre-fabricated AMIs for creating on-demand FTEs using CloudFormation templates.

Elasticity in FTEs

With a FrontEnd application developed using AWS SDK for Python, Engineer aka the Feature Owner is free to create new FTE by providing the following inputs like

Who owns this FTE?
Where is User’s office located?
Which branch should the FTE be based off?
What is the feature development Id as in Jira?

and creates the cloud formation template on the fly, which in turn creates the Windows & Linux Instance from the region specific AMIs in the low latency region close to user’s office location and creates RecordSets.
The feature owner can trigger the build & deployment the against the chosen branch to get his changes into the new FTE.

With the FrontEnd application, Engineers are free to create FTEs at their own will, trigger build & deployment as needed, Run Code validation suite like JUnit Tests & SQL Static Code Check.

When the feature development is complete, feature owner promotes the feature branch to the integration branch and delete the FTE Environment.

And thats how the elasticity is in place for Feature Test Environments.

Elasticity in Build & Deploy Infra

Yes, the build & deployment framework for FTEs are also scalable & elastic in nature and that’s implemented using Jenkins + EC2 plugin. With this plugin, if Jenkins notices that the build or deployment cluster is overloaded, it'll start instances using EC2 API and automatically connect them as Jenkins slaves. When the load goes down, excessive EC2 instances will be terminated. This setup allows you to maintain a small in-house cluster. The FrontEnd application for creating FTEs have an interface to the Jenkins which helps to trigger build & deployment at the click of a button.

The Build & Deployment infrastructure is of three pre-fabricated AMIs