The Taste of the Salt
devopsI have tried four different flavors of orchestration and provisioning tools, and I enjoyed the taste of the Salt most.
Home-brewed scripts
In Microsoft Office, the development environment is brought up by a home-brewed Perl script, called ocheck. To bootstrap a new dev box, first you PXE boot the machine, install the Windows Server from the network, restart machine several times after the Windows Update; then finally you ran ocheck from the net share. It would take 2 – 3 hours for ocheck to:
- configure Windows, such as disabling power saving to ensure the overnight build won’t fail.
- punch holes in Windows firewall.
- install build toolchains: Visual Studio, MSBuild etc.
- install potential dependencies, SQL Server, Windows SharepPoint Service etc
- install Office team-specific tools: windbg, ohome etc.
It is also worthy noting that most applications are installed without attendance. Some installations show the setup wizards, but no action is required; Visual Studio needs extra love though. Not only you have to carefully pick the optional components which turn out to be essential later, such as the x64 toolchain; but also ocheck will not complain if your configuration is broken.
Comment: the ocheck is a bit of cumbersome, but overall it gets the job done and appreciated by thousands of engineers. To be fair, I haven’t seen any provisioning tools really shine in Windows platform either.
Puppet
The engineers in Skytap have the luxury to run the whole stack inside the
Skytap cloud: just clone the golden images in the corporation account, then
click Run all VMs, done. Unfortunately, the engineers’ VMs must be suspended
to yield more resource to the customers in the peak time if the computing
resources run short. So the engineers also deployed a private stack on HP
ProLiant server. After setting up the ESXi hosting node, an Ubuntu server VM is
installed from the scratch as the golden image, then cloned to multiple
instances for various roles. An Solaris instance is installed in the PXE boot
manner as the storage server. One VM is configured as the Puppet master, then
the puppet
is invoked in all other VMs for provisioning.
Comment: I think Skytap choosing puppet mainly because of the timing: when DevOps shopped around for a provisioning tool, Puppet was the only viable solution in a heterogeneous environment.
Chef
The development environment in SurveyMonkey is a little bit different: it’s not self-contained. The whole stack should include the .NET stack, the python stack and external services such as splunk. Only the python stack is provisioned, the .NET stack and the MS SQL server are shared across the company. The log is saved locally, and the metrics need to be parsed manually from the log. The provisioning leverages the power of vagrant and chef solo.
vagrant up
vagrant provision
vagrant ssh
Comment: I think Chef was chosen because the DevOps were strongly inclined to using ruby.
SaltStack
It sounds a little funny to automate provisioning a personal website, just mere a single host, right? Well, it starts to make sense when multiple sites are hosted in the same basket, and you hop between ISPs every year. I used the Salt; and the provisioning is just one command away:
sudo salt-call --local state.highstate
It will setup the firewall, harden the ssh service, install the LEMP stack with monitoring. It also did all the plumbing work for three wordpress instances. The succeeding restore procedure will bring the system up to date.
Closing Rants
Declarative vs Imperial
Though ruby is a very expressive language with active communities, I still think Chef takes the wrong direction. The provisioning is all about state transition, i.e bringing the system from one state to another state. The recipes/cookbook should be declarative to define where the destination is instead of describing how we get there.
Verbose vs Terse
If you have to use git submodule
to add a service, it is a failure. The DSL
should be pretty terse to do simple things and elastic to do complex things. I
really appreciate the flexible layout of the Salt, you can use the canonical
foo/init.sls
or just foo.sls
for foo
module; and intuitively,
foo/bar.sls
will be referenced as foo.bar
.