The Taste of the Salt

devops

I have tried four different flavors of orchestration and provisioning tools, and I enjoyed the taste of the Salt most.

Home-brewed scripts

In Microsoft Office, the development environment is brought up by a home-brewed Perl script, called ocheck. To bootstrap a new dev box, first you PXE boot the machine, install the Windows Server from the network, restart machine several times after the Windows Update; then finally you ran ocheck from the net share. It would take 2 – 3 hours for ocheck to:

  • configure Windows, such as disabling power saving to ensure the overnight build won’t fail.
  • punch holes in Windows firewall.
  • install build toolchains: Visual Studio, MSBuild etc.
  • install potential dependencies, SQL Server, Windows SharepPoint Service etc
  • install Office team-specific tools: windbg, ohome etc.

It is also worthy noting that most applications are installed without attendance. Some installations show the setup wizards, but no action is required; Visual Studio needs extra love though. Not only you have to carefully pick the optional components which turn out to be essential later, such as the x64 toolchain; but also ocheck will not complain if your configuration is broken.

Comment: the ocheck is a bit of cumbersome, but overall it gets the job done and appreciated by thousands of engineers. To be fair, I haven’t seen any provisioning tools really shine in Windows platform either.

Puppet

The engineers in Skytap have the luxury to run the whole stack inside the Skytap cloud: just clone the golden images in the corporation account, then click Run all VMs, done. Unfortunately, the engineers’ VMs must be suspended to yield more resource to the customers in the peak time if the computing resources run short. So the engineers also deployed a private stack on HP ProLiant server. After setting up the ESXi hosting node, an Ubuntu server VM is installed from the scratch as the golden image, then cloned to multiple instances for various roles. An Solaris instance is installed in the PXE boot manner as the storage server. One VM is configured as the Puppet master, then the puppet is invoked in all other VMs for provisioning.

Comment: I think Skytap choosing puppet mainly because of the timing: when DevOps shopped around for a provisioning tool, Puppet was the only viable solution in a heterogeneous environment.

Chef

The development environment in SurveyMonkey is a little bit different: it’s not self-contained. The whole stack should include the .NET stack, the python stack and external services such as splunk. Only the python stack is provisioned, the .NET stack and the MS SQL server are shared across the company. The log is saved locally, and the metrics need to be parsed manually from the log. The provisioning leverages the power of vagrant and chef solo.

vagrant up
vagrant provision
vagrant ssh

Comment: I think Chef was chosen because the DevOps were strongly inclined to using ruby.

SaltStack

It sounds a little funny to automate provisioning a personal website, just mere a single host, right? Well, it starts to make sense when multiple sites are hosted in the same basket, and you hop between ISPs every year. I used the Salt; and the provisioning is just one command away:

sudo salt-call --local state.highstate

It will setup the firewall, harden the ssh service, install the LEMP stack with monitoring. It also did all the plumbing work for three wordpress instances. The succeeding restore procedure will bring the system up to date.

Closing Rants

Declarative vs Imperial

Though ruby is a very expressive language with active communities, I still think Chef takes the wrong direction. The provisioning is all about state transition, i.e bringing the system from one state to another state. The recipes/cookbook should be declarative to define where the destination is instead of describing how we get there.

Verbose vs Terse

If you have to use git submodule to add a service, it is a failure. The DSL should be pretty terse to do simple things and elastic to do complex things. I really appreciate the flexible layout of the Salt, you can use the canonical foo/init.sls or just foo.sls for foo module; and intuitively, foo/bar.sls will be referenced as foo.bar.