Automating Development Environments with Vagrant and Chef

What is development environments automation? Why use it? What problem it solves?

Imagine you are a developer just assigned to a new project. There are some new cool ideas to be realized and you are given some good challenges in this project. You are full of excitement to begin coding. Now you just need to set up the environment.

There is a proper guide in the project wiki for doing this, you start to follow it and end up with following activities:

• Upgrade your local MySQL server and client.
• Add that project-specific setting to the MySQL conf.
• Create your local project database.
• Create your local project database users and add permissions.
• import database contents from a shared test environment (and you’re being told to not take the latest contents, its broken at the moment..).
• Realize that the MySQL database had to be created as UTF-8 encoded. Delete the database and perform again the previous three steps.
• Upgrade your existing Apache HTTP Server.
• Re-configure your existing Apache HTTP Server installation for the new project. There is a nice project-specific httpd.conf file available to copy and paste, but you need to remove and install modules for that to work, as well as override some personal settings.
• Copy some additional configuration files and static content files for Apache HTTP Server.
• and so on and so on.

So you take a day or two for environment setup. You encounter various issues. Some of the issues are easily solved by that nice senior guy of the new project. Other issues he has never seen. The strangest issue you encounter sounds familiar to him, but he does not remember exactly what they did to overcome this issue. The lady who probably knows more has left the project, works in some other country now, and is unavailable at the moment.

Why is every environment setup for any project a unique artwork? What if you could just get it with a single click?  Why not let the computer do the step-by-step setup automatically for you (and you having a lunch or a table football match with nice colleagues at the same time), and still start being productive in half an hour instead of three and half days? This is exactly where development environments automation, when done properly, can help.

So, development environment automation can make local development environment setups automatic and repeatable. This would save lot of time in situations like these:

• you are starting work in a new project and need a particular local development environment set up
• you are switching back and forth between multiple projects that each have a different local development environment
• your local development environment has broken and it seems to take more than an hour to figure out why and fix it
• you need a temporary environment for a quick demo or a proof-of-concept experiment

Tools

You can always script development environment automation in your favorite programming language, but that may turn out a long and hard task. Another way would be to build the entire solution in Docker way, that also takes care of development environment automation. Yet third way is to use Vagrant, which is a tool especially meant for that. Since I like automation and since I happened to have some free time, I started exploring development environment automation the Vagrant way to create a setup of a particular project I would possibly be starting in the future.

Once I got started, I encountered a mess of related tools and technologies that were totally new for me: Ruby, Chef, ChefDK, Chef Solo, ChefSpec, Vagrant, various Vagrant plugins, Berkshelf, Knife, Test Kitchen, RSpec, Serverspec – what are all these? Do I need to learn them all? Do I need to learn a new programming language? Several weeks later, the mess was sorted out and I had a working solution together with automated tests. Here is a brief description of the solution:

The basis for the entire solution is Vagrant, of course. Vagrant and all the other tools are based on Ruby programming language. However, i did not have to learn all of Ruby, because the tools generally provide a Ruby-based DSL for configuring them. (I still got quite excited about Ruby)

I used Chef Solo as provisioning engine. This is a version of Chef that works without a Chef server.

I started with a generic public vanilla Ubuntu 14.04 box from the Hashicorp’s Atlas box catalog ) Then i had my Chef cookbook build up everything on top of the box.

I used some public open-source Chef cookbooks for setting up some widely used open-source components: Tomcat, ActiveMQ, MySQL, Java. I used Berkshelf to manage cookbook dependencies. I also used the Berkshelf plugin for Vagrant to handle the dependencies fully automatically – I just had to declare them. This series of blog posts was the most helpful source of information for me: http://misheska.com/blog/2013/06/16/getting-started-writing-chef-cookbooks-the-berkshelf-way/

Since I’m a fan of test-driven development, I developed unit tests for this solution as well. I used Chefspec, which is a Chef-specific extension of Rspec – a unit-testing framework for Ruby. Here is an example of a Chefspec unit test: it ‘creates tomcat-users.xml file with correct admin user and roles

do
expect(chef_run).to render_file(users_file).with_content(/<role.*rolename=\"manager-gui\"/)
expect(chef_run).to render_file(users_file).with_content(/<role.*rolename=\"manager-script\"/)
expect(chef_run).to render_file(users_file).with_content(/<role.*rolename=\"manager-jmx\"/)
expect(chef_run).to render_file(users_file).with_content(/<role.*rolename=\"manager-status\"/)
expect(chef_run).to render_file(users_file).with_content(/<user.*username=\"#{test_tomcat_admin_username}\"/)
expect(chef_run).to render_file(users_file).with_content(/<user.*password=\"#{test_tomcat_admin_password}\"/)
expect(chef_run).to render_file(users_file).with_content(/<user.*roles=.*manager-gui/)
expect(chef_run).to render_file(users_file).with_content(/<user.*roles=.*manager-jmx/)
expect(chef_run).to render_file(users_file).with_content(/<user.*roles=.*manager-status/)
end

Since I also love automation in general, I also wrote some smoke (integration) tests using Serverspec, which is another extension of Rspec. I used the Test Kitchen tool that comes with Chef Development Kit to automatically build and integration-test the solution. Here is an example of a Serverspec integration test:

</pre>
<pre>describe "Smoke test" do
describe command("curl ") do
its(:stdout) { should match /.*<img.*alt=\"Snap It Up\".*/ }
its(:exit_status) { should eq 0 }
end
end</pre>
<pre>

Lessons learned

Setting up the automation environment itself is not always straightforward. I shared my solution with two of my colleagues both using a Windows desktop (i’m using Ubuntu), and they both had issues setting up Ruby and Vagrant itself. It took them several hours and several iterations to get this working. This was the single major throwback for me, and made me sad for a while. Also, I discovered that there are still a few platform-specific quirks. However, in theory this is one-time setup, and can be automated per platform as well.

Starting with the generic public box seemed like a good idea initially, but actually had a significant problem: it took too much time to download and deploy everything, especially if i needed to do it repeatedly. It took almost an hour from zero to full environment. This would be ok for a user of my solution, but not for a developer who makes small changes and wants to see the result immediately.  I had to rely on unit tests and make some clever shortcuts to minimize this time. For example i kept local cache of all the packages downloaded from internet, and made this cache permanent, i.e. survive a complete destroy-rebuild of the solution. Moreover, i had to make sure that my cookbook does not overdo what is already done. Next time i’d consider making a more specific private box to begin with: i.e. Java, Maven, MySQL pre-installed.

The generic public Ubuntu box I used also happened to be not quite up to date and contain a minor issue with dpkg. So as the very first thing after starting up the box I had to do this:

# The first line is a workaround for this issue:
# <a href="http://stackoverflow.com/questions/23671727/error-with-sudo-apt-get-dictionnary">http://stackoverflow.com/questions/23671727/error-with-sudo-apt-get-dictionnary</a>
# -commons-since-update-to-ubuntu-14-04
# The second line is a workaround for an issue similar to this:
# http://askubuntu.com/questions/271590/i-cannot-install-java-from-the-software-center

bash 'Fix_dpkg_issue' do
code &lt;&lt;-EOH
sudo /usr/share/debconf/fix_db.pl &amp;&amp; sudo dpkg --configure -a
sudo apt-get update -y
EOH

end

It would again be a good idea to keep a private box image with this issue fixed by default.

To summarize

The Vagrant+Chef ecosystem is still quite new. Specific versions of components are incompatible with specific versions of other components. New versions of components are actively being produced. One day I discovered that my solution that had worked fine the previous day started to consistently fail with a very strange error deep under the hoods of Ruby. It turned out that a component had been updated. Lesson learned: don’t automatically use the latest version of everything but fix a set of components with particular versions that are known to work together. This applies both to the tools and frameworks (Ruby, Rspec, Chef, Vagrant, Vagrant plugin for Berkshelf, etc.) as well as Chef cookbooks (for Java, Tomcat, ActiveMQ, etc.).

Since, again, the ecosystem is new, some cookbooks for some very commonly used components, that you’d expect to be stable, may not actually be stable. For example, the Chef Tomcat cookbook is quite broken at the moment. I initially used it, but had to override too much to get around its bugs and limitations. Finally I removed it altogether and used standard Chef means to deploy Tomcat, set configuration files, make it run as service, etc.