nClouds recently kicked off the 5C’s Series of webinars that presents customer case studies on CI, CD, Containers, Cloud and Culture, in collaboration with co-sponsors, Amazon Web Services (AWS). In our first session, I was joined by Jarrod Sinclair, Chief Architect, Prodea; Doug Cliche, Solution Architect, AWS; and Alan Shimel, Editor-in-Chief, DevOps.com. We explored key lessons from Prodea’s work-in-progress journey adopting a DevOps culture.
nClouds first engaged with Prodea’s Arrayent group in 2016, prior to Prodea’s acquisition of Arrayent, when it took three to six months for their team to release new features to production and roll out to customers. Fast forward a year to current state and it’s clear that they’ve benefited from implementing a DevOps process: greater than 10x gains in delivery speed, improved quality, and a culture of confidence and collaboration. What’s more, recognizing Prodea’s rapid innovation, Entrepreneur.com lists Prodea’s Arrayent as one of “25 Innovative IoT Companies and Products You Need to Know.”
There are innumerable takeaways from the hour-long discussion. I spotlight five in this post, and encourage you to watch the webinar replay for additional valuable insights.
1. Start small. But make the project real, and success visible.
Hopefully everyone recognizes by now, DevOps is a journey that involves changes to process, technology and your organization’s culture. It’s not one-and-done, so strap in for the ride. But you have to start somewhere, and the best bet is to start small, with a project that is manageable, measurable, and winnable. You take that first small success and spotlight it because most DevOps initiatives require support at all levels — bottom up, top down, and middle out.
Arrayent’s journey started with one question, “Is it possible to cloud-enable our platform?” With hundreds of thousands of lines of code in their environment, they knew they wanted to containerize with Docker. So they started small to demonstrate the “art of the possible.” The team took a week to create their core stack as a local environment with Docker Compose. Then, in a gutsy move, they gave instructions on running the local environment to their CEO. Within 20 minutes he was running the environment on his laptop. The team had their jumping off point and executive buy-in.
2. Fail fast. Recover, learn and pivot faster.
You’re on a journey, so you’re setting goals and then showing steady progress toward those goals. Will it be a straight line? Probably not. But it’s not all or nothing, either.
At the start of their journey, Arrayent had a hard time delivering quality updates quickly and routinely. That led to a culture characterized by lack of trust in the process, continuous fire fighting with deployments and rollbacks, and little time for innovation.
In one case, they tried to migrate their environment to the cloud, but because it failed initially, the team adopted a “we tried this before and it didn’t work” mindset. For a time, this prevented them from embracing new technologies and creating competitive advantage.
But they made incremental steps. First, by “eating their own dog food.” They built their production environment and implemented it first internally — in QA. That gave the team the ability to experiment, fail fast, recover. Ultimately, it gave the team confidence in the tools, process, and infrastructure.
Once the team realized that there was an opportunity for dramatic gains in speed and efficiency on cloud, they left their past failure behind and focused on learning from their mistakes instead of succumbing to them.
3. Automate. It’s how you go fast with quality.
It’s kind of crazy, maybe predictable, but many companies moving to cloud first use it like an on-premises environment. The platform, the infrastructure, the tools. They miss the many new capabilities they can tap that benefit dev and ops teams directly. These impact speed, quality and cost. And when you release your team from grunt work because your new tools do it for them, you get happier, more productive team members focused on higher value tasks.
Arrayent assembled a great set of tools that automated many tasks, with Ansible, Jenkins and AWS at the core. They were able to rapidly spin up new environments, and then knock them down, getting speed and productivity.
They used EC2 Container Service (ECS) for container management, eliminating the need to install, operate and scale their own cluster management infrastructure. The team essentially doesn’t have to worry about Docker container management.
The team updated their CI/CD pipeline to make testing part of the deployment. They leveraged the Robot Framework to get immediate feedback about whether their environments were deployed correctly, and then built an automated test process that allowed them to run a full regression on every pull request. And with Amazon Route 53 they can spin up new configurations, start working, connect to the new environment, test, and turn it off.
4. Fix the root. No band-aids.
Adopting DevOps in any organization requires buy-in at various levels. There’s risk, as with any corporate initiative, of sugarcoating issues and looking for quick fixes that improve optics. That’s not the solution. Find the root cause. Do the more painful fix. Assume permission to make fundamental changes, as needed.
Plus, you’ll find that the 80-20 rule applies here. When you fix the root you have the biggest impact first. You’re not fixing symptoms, you’re fixing root cause. And you’ll solve bottlenecks in the process.
Arrayent found transparency to be essential to identifying root cause. They created a Chatbot to promote collaboration among and across the team, and bring visibility and transparency to resolving issues.
5. Culture wins. It’s your competitive advantage.
DevOps culture is founded in collaboration. It’s inherently about knocking down silos, not just in dev and ops, but with all stakeholders across the software delivery lifecycle. And, DevOps promotes a culture of feedback and transparency. Ultimately, these elements are essential to successful DevOps transformation.
Among the biggest challenges Arrayent faced during their DevOps journey was convincing the QA department to move away from their legacy environment. Instead of compromising and adopting a “one foot in, one foot out” policy, they adopted a “burn the ships” mantra that shut down their old environment and got everyone on the cloud. There was no retreat, everyone was invested in going forward.
Bold moves like “burn the ships” are calculated, of course. There must be confidence, not just among leadership, but with the broader team. At Arrayent, that confidence grew organically. By delivering quality, consistently. By eliminating rollbacks, completely. They created a blameless culture, enabled by feedback and consistency across the dev, test and production environments. It evolved into a fun and exciting place to work. And the ability to adapt to change and innovate rapidly made the company shine — and led ultimately to getting acquired.
Next steps: continuous learning
To learn more about Prodea’s DevOps journey, watch the 5C’s Series webinar, “Faster Innovation with Confidence”. Or, read the Prodea case study.
