Ozlo provides chat app
Create a cost-effective solution to increase the throughput of the builds
Ozlo provides an chat app which can be described as "The friendly assistant who helps you get things done — find restaurants, discover recipes, order delivery, buy movie tickets and more!" They collect data across various sources such as Facebook, Youtube, Yelp etc. then helps to answers questions based on geo-location and user preferences.
Ozlo had a single huge test suite that validates the integrity of the all application components. Each pull request triggers build. As the team size continue to grow, builds were backlogged affecting developer productivity. The ask was to create a cost-effective solution to increase the throughput of the builds.
The trivial solution to add more slave nodes would turn out to be expensive since they need high resource machines to satisfy the build requirements. We proposed to build a AWS ECS cluster which can provide containers that can act as Jenkins slave. We moved the share data on to AWS EFS and mounted the volume to the slave container, which saves time rather than waiting for files to download to complete. All of this was completed using best practices policies and procedures and in partnership with nClouds partners and advisors. It was also important to ensure that Ozlo's team were in the loop and understood the process and timeframes.
We created an AWS EFS filesystem and referred it from the job which creates shared files, so that the AWS EFS is populated with all versions of shared files. We created a AWS CloudFormation template (using Troposphere) creates an AWS ECS cluster, Docker registry and 2 Auto Scaling groups 1 with a single reserved instance and 3 spot instances. We reserve 95% for CPU for each build job. We have defined scaling policies based on CPUReservation on AWS ECS cluster and scale up and down the AWS Spot Instance cluster. This enables slavepool to automatically scale up to 4x (configurable) based on demand for running multiple jobs.
The wait time for build results has gone down over 3 times in case of multiple parallel builds. Of course, the max instances of autoscaling group can be adjusted to even more throughput. All this is done while keeping the costs low by allocating more machines only when there is demand.
You can also email us directly at firstname.lastname@example.org for your inquiries or use the form below