Test OpsWorks Cookbooks locally using Test Kitchen and Docker

The challenge

• Testing cookbooks on OpsWorks takes a really long time and the feedback loop is really slow
• Most of the OpsWorks cookbooks don’t run locally unless OpsWorks agent is running.

In this blog post I’ll walk you through how to simulate a OpsWorks environment locally using Test Kitchen and Docker ( boot2docker ). Hopefully, this will help you feel more confident about the making changes and getting faster feedback.  You can check out the repo opsworks-local. It has everything you need to get you going.

git clone --recursive git@github.com:nclouds/opsworks-local.git

Launch a sample PHP application using AWS CloudFormation.

Note: I am using a same template from the AWS examples. The only thing I have changed is to lock down the Chef version to 11.10.

Click the animated gif image below to see workflow.

Animated CloudFormation Template Workflow

Go to the Stack in OpsWorks and select “Permissions”. Check ssh and sudo for your user. ( If you don’t already have SSH keys added to OpsWorks, you can add them by going to “My Settings” and paste in your public key)

OpsWorks Permissions

Get the public IP address for the PHP app for the instance and ssh into it to grab the metadata needed to run the deployments locally.

ssh jgiri@54.166.161.54
Last login: Thu Apr  9 03:49:40 2015 from 76.126.94.248
 This instance is managed with AWS OpsWorks.
   ######  OpsWorks Summary  ######
   Operating System: Amazon Linux AMI release 2014.03
   OpsWorks Instance: php-app1
   OpsWorks Instance ID: 115a5dad-e4e2-40db-b380-71ab2d3217fe
   OpsWorks Layers: PHP App Server
   OpsWorks Stack: opsworkstest
   EC2 Region: us-east-1
   EC2 Availability Zone: us-east-1a
   EC2 Instance ID: i-7491f388
   Public IP: 54.166.161.54
   Private IP: 10.185.136.48
 Visit https://aws.amazon.com/opsworks for more information.
[jgiri@php-app1 ~]$ sudo su -
Last login: Thu Apr  9 03:50:52 UTC 2015 on pts/0
[root@php-app1 ~]# /opt/aws/opsworks/current/bin/opsworks-agent-cli get_json

Now copy the output into the file php.json in your working directory.

├── Berksfile
├── cloudformation
│   ├── OpsWorks
│   └── OpsWorks.json
├── cookbooks
│   └── opsworks_agent
│       ├── Berksfile
│       ├── CHANGELOG.md
│       ├── Gemfile
│       ├── LICENSE
│       ├── Thorfile
│       ├── chefignore
│       ├── metadata.rb
│       ├── recipes
│       │   └── default.rb
│       └── templates
│           └── default
│               ├── client.yml.erb
│               └── instance-agent.yml.erb
└── opsworks-cookbooks
└── php.json

Run kitchen converge

kitchen converge 
## Get the Docker container info once the Kitchen converge finishes 
docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                                          NAMES
5e42dc05521c        f374dfd02e9f        "/usr/sbin/sshd -D -   7 minutes ago       Up 7 minutes        0.0.0.0:49156->22/tcp, 0.0.0.0:49157->80/tcp   dreamy_banach
➜  opsworks-local git:(master) ✗ boot2docker ip
192.168.59.103

You should now be able to hit the app on https://192.168.59.103:49157   ( See .kitchen.yml to change the IP )

PHP App Congratulations!

Here is how the Berksfile looks like:

source "https://supermarket.chef.io"
def opsworks_cookbook(name, branch='release-chef-11.10')
  cookbook name, github: 'aws/opsworks-cookbooks', branch: branch, rel: name
end
opsworks_cookbooks = Dir['opsworks-cookbooks' + '/*'].select { |f| File.directory?(f)  }.map { |f| File.basename(f)  }
opsworks_cookbooks.each do |cb|
  cookbook(cb, path: File.join('/Users/jt/nclouds/demo_ops/opsworks-cookbooks', cb))
end
cookbook "local_opsworks", path: "cookbooks/local_opsworks"

OpsWorks has many cookbooks. You can identify all dependent cookbook for PHP app and reference them individually, I am lazy  . I simply just added a submodule for opsworks-cookbooks and included all the cookbooks in the Berksfile: local_opsworks is needed to install the OpsWorks, or else most of the OpsWorks deploy cookbooks fail. The Opsworks agent needs to be running.

Let’s also take a look at .kitchen.yml

---
<% require 'json'
php_layer = JSON.parse(File.read('php.json'))
%>
driver:
  name: docker
  socket: tcp://192.168.59.103:2376
provisioner:
  name: chef_zero
platforms:
  - name: ubuntu-14.04
    driver_config:
      forward:
      - 80
suites:
  - name: default
    run_list:
     - opsworks_agent
     - mysql::client
     - dependencies
     - opsworks_ganglia::client
     - mod_php5_apache2
     - deploy::default
     - deploy::php
    attributes:  { "opsworks" : <%=php_layer['opsworks'].to_json%>, "deploy": <%=php_layer['deploy'].to_json%> }
    cookbook_path: cookbooks

• We are reading the php.json file for the attributes.
• You can get the socket information using boot2docker socket command
• opsworks_agent is custom recipe which install the agent
• All the other recipes are needed to deploy PHP, you can always get a list of the recipes from the OpsWorks Layer UI or from the command line. I removed some cookbooks like EBS and ssh from the list, they are not needed for the local environment.

  aws opsworks describe-layers --region us-east-1  --layer-id 8d19ac5f-40ba-33-8b67-5b974805c03a | jq   '.Layers[].DefaultRecipes| .Setup[], .Deploy[]'
  opsworks_initial_setup
  ssh_host_keys
  ssh_users
  mysql::client
  dependencies
  ebs
  opsworks_ganglia::client
  mod_php5_apache2
  deploy::default
  deploy::php

• Attributes: we are passing in attributes opsworks and deploy, which are needed for the deployment

A few other useful tips about OpsWorks

• All the AWS OpsWorks cookbooks are hosted here: https://github.com/aws/opsworks-cookbooks
• Every time one makes changes to a cookbook, one must run “update_custom_cookbooks” to upload the changes to server.
• When OpsWorks finishes, one can finds that final cookbooks are saved in:  /var/chef/cookbooks/ ( This is the result of the command: berks install. This can be helpful for debugging.)
• You can also run the update_custom_cookbooks command from command line. I feel like it’s much faster to do this from command line, as it takes less time.

#Note: Get the json from the last run /var/lib/aws/opsworks/chef/latest.json, the only thing important for updating the custom cookbook is the revision variable

# The next line is long and a single line but is shown here as multiple lines

• /opt/aws/opsworks/current/bin/chef-client 
   -j /var/lib/aws/opsworks/chef/2015-04-01-02-51-31-01.json 
   -c /var/lib/aws/opsworks/client.stage1.rb 
   -o opsworks_custom_cookbooks::update,opsworks_custom_cookbooks::load,opsworks_custom_cookbooks::execute

• Run the deployment on the instance. I also find this to be much faster then running it from the UI. One just needs to make sure that if you made any changes in the UI, like new environment variables, those won’t be available in the JSON unless you add them to the json manually.

# The next line is long and a single line but is shown here as multiple lines

• /opt/aws/opsworks/current/bin/chef-client 
   -j /var/lib/aws/opsworks/chef/2015-04-09-04-23-18-01.json 
   -c /var/lib/aws/opsworks/client.stage2.rb 
   -o deploy::default,opsworks_stack_state_sync,deploy::php,test_suite,opsworks_cleanup