Getting started with Chef tutorial

Chef is a tool for turning your servers into code. Chef is awesome. It took me a while to realize it, though.

After setting up a mysql slave server for the fifth time, I decided that I was done manually configuring servers. Chef seemed like an obvious choice. It took me about two days of hacking to figure out exactly how to get it to work.

I ended up having to use a ton of different resources to get started. Googling for what to do with Chef is non-trivial. If all of the resources had been in the same place, it probably only would have taken me about a day.

This post aims to help small companies with 5-10 servers get started with chef. I recommend following the directions in order. Before you know it, you’ll be cookin’ like a master!

For a brief overview of what Chef is for check out Opscode’s blip.tv stuff (Chef creator).

For this tutorial, we’ll be using the Opscode platform, which saves you the trouble of setting up an opscode Chef server. If you want, you can do that later.

Chef has two parts: a client and a server. Chef-client runs on all of your machines, and they reach out to the server for recipes or configurations.

So, once you’ve watched the first video register for the opscode platform here

After that, instead of going to the Chef wiki (it’s overly complex for newcomers), “check out the “getting started guide. in Opscode’s help app

If you prefer a Railscast style video, you can also watch this video which is just a walk through of the getting started guide above.

I recommend following through with the entire getting started guide. I initially wanted to just use chef-solo, which seemed to make more sense, but it turns out it doesn’t because you miss out on all kinds of cool features.

I advise against adding a bunch of cookbooks to your chef-repo early on, since there are weird dependencies and other issues that can make things more confusing than they need to be initially.

The problem with the getting started guide is that it doesn’t tell you how to actually configure a node outside of your own machine. Doing this is actually pretty easy, but somewhat undocumented.

First, get a blank server somewhere. Note that if you’re using your local machine, it didn’t work for me to just use VMWare Fusion virtual machines. The Opscode platform wouldn’t recognize them.

Instead, I recommend just creating a small server on Linode/Rackspace/Slicehost. I used Ubuntu Lucid, so I’d recommend that as well if you want to follow along. However, any Chef supported distro should work. Once the new server is imaged and you get your root password, do the following:



gem install net-ssh-multi
knife bootstrap 172.x.x.x

The net-ssh-multi gem will save you from a confusing error.

Knife, as you learned in the getting started guide, is the Chef command line tool. Shef is to Chef as script/console is to Rails while Knife is a bit more like Capistrano, sorta.

The knife bootstrap command will try to login to the given IP as root (you’ll need the password, unless you’re using ec2 or something where keys are set). It will then setup the node (the server) to run chef-client. If you followed the getting started guide correctly, it will also upload the ssl stuff.

Once knife bootstrap is finished, login to the server using ssh.

Then run

123	chef-client

This will connect to your opscode Chef server and run any recipes that you’ve specified. At this point, you should not have specified any recipes, so it should run cleanly.

Next, it’s time to make a role. You can do this on the opscode web ui, but I personally prefer just editing raw JSON than clicking around.

So, in the roles roles folder add a new file called base.json.

Put the following json into the file:


{
    "name": "base",
    "default_attributes": {
      "chef": {
        "server_url": "https://api.opscode.com/organizations/#{YOUR COMPANY!}",
        "cache_path": "/var/chef/cache",
        "backup_path": "/var/chef/backup",
        "validation_client_name": "#{YOUR COMPANY!}-validator",
        "run_path": "/var/chef"
      }
    },
    "json_class": "Chef::Role",
    "run_list": [
      "recipe[ubuntu::default]",
      "recipe[build-essential::default]",
      "recipe[openssl::default]",
      "recipe[chef::client]",
      "recipe[chef::delete_validation]",
      "recipe[git::default]"
    ],
    "description": "Basic Server",
    "chef_type": "role",
    "override_attributes": {
    }
  }

Be sure to replace the stuff in #{} with your info. Then run:

123	knife role from file roles/db-slave.json

This will upload the role to the chef server.

Now we need to download those cookbooks. For each of the items in the run_list array above, do

123	knife cookbook site vendor #{name}

For example

123	knife cookbook site vendor ubuntu

This will download the cookbooks to your chef-repo and create a git commit.

Next, it’s time to upload your cookbooks:


knife cookbooks upload -a    # uploads chef cookbooks with knife

# OR

rake upload_cookbooks

Both of the above commands will upload your cookbooks.

At this point, it’s worthwhile to start getting really familiar with the output of knife when you don’t enter any arguments. It lists all the commands, which you will have to learn just like you would rails commands. It takes a while, but they start to feel natural.

Next, let’s try to run chef-client on the node. As root, on your new node, run:

123	chef-client

It should run cleanly. If it doesn’t feel free to e-mail me at f.mischa@gmail.com and I will update this post.

If it does run cleanly: Congratulations! You’re cooking with Chef. Ohai!

From here on, the workflow for me is basically just:

[workstation] knife cookbook site vendor #{cookbook} -d # The -d flag will install dependencies [workstation] knife cookbook upload #{cookbook} [workstation] add the recipie from the cookbook that you want to use to the base role [workstation] update the base role with the command above (from file) [node] run chef-client [node] check out what errors you get [workstation] try to fix them

For many recipies, especially the 37signals ones here there is basically no documentation. This means that you will often have to read through the entire cookbook to figure out how you need to configure it. It’s very much trial and error. The best strategy is to read through the cookbook, creating a json hash for w/e data you’ll need.

The 37signals users cookbook is really great, so I will quickly run through how to set it up, as an example of how to figure out undocumented recipes.

First, grab this gist. and put in your chef repo in chef-repo/tasks and add the rake file stuff to your chef-repo’s rakefile.

That gist will help you deal with databags sanely.

For the 37signals users recipe, clone the 37s cookbooks repo and then copy the users cookbook to your cookbooks directory in your chef-repo.

Next, in chef-repo/databags, you’ll need two files.

databags/groups.json:


[
  {
    "id": "admin"
  },
  {
    "id": "other"
  },
  {
    "id":"whatever"
  }

]

You can start with just admin for now.

Also, you’ll need databags/users.json:


[
  {
    "id": "mischa",
    "groups": [
      "admin"
    ],
    "uid": 7777,
    "local_files": true,
    "full_name": "Mischa",
    "shell": "/bin/bash",
    "ssh_key": "my key",
    "password": "my pw"
  }
]

Change this to your info, obviously. For password, you can get the value to put in by doing:

123	openssl passwd -1

Note that that’s a one, not an l. That will give you a value to put there. For the ssh key, use your ~/.ssh/id_rsa.pub If you dont have one of those, not sure why you’re reading this, but go over to github and they will show you how to make one.

Now it’s time to use our handy rake task from the Gist. Note that it expects a ~/.chef directory, so you may want to link or copy your .chef stuff from your chef repo to your home directory.

123	rake load_data_bags

This will upload our users and groups data bags to the chef server. Now you just have to:

123	knife cookbook upload users

Finally, update your roles/base.json to:


{
    "name": "base",
    "default_attributes": {
      "active_groups": [
        "admin"
      ]
      },
      "chef": {
        "server_url": "https://api.opscode.com/organizations/foo",
        "cache_path": "/var/chef/cache",
        "backup_path": "/var/chef/backup",
        "validation_client_name": "foo-validator",
        "run_path": "/var/chef"
      }
    },
    "json_class": "Chef::Role",
    "run_list": [
      "recipe[ubuntu::default]",
      "recipe[build-essential::default]",
      "recipe[openssl::default]",
      "recipe[chef::client]",
      "recipe[chef::delete_validation]",
      "recipe[git::default]",
      "recipe[users::default]"
    ],
    "description": "Basic Server",
    "chef_type": "role",
    "override_attributes": {
    }
  }

And then upload role the using knife role from file as above.

And then run chef-client on the node. This should install users and setup your ssh stuff. Sometimes chef-client fails on the first run because it can’t fine some package. Running it again somehow fixes this, and it can find the package.

Next steps

From here, the world is yours. You should look through all the opscode cookbooks and the 37s ones and see which you like. I recommend starting small, since it can get pretty complicated fast.

My entire base configuration is:


{
    "name": "base",
    "default_attributes": {
      "active_groups": [
        "admin"
      ],
      "active_sudo_groups": {
        "admin": {
          "enabled": true
        }
      },
      "chef": {
        "server_url": "https://api.opscode.com/organizations/foo",
        "cache_path": "/var/chef/cache",
        "backup_path": "/var/chef/backup",
        "validation_client_name": "foo-validator",
        "run_path": "/var/chef"
      }
    },
    "json_class": "Chef::Role",
    "run_list": [
      "recipe[ubuntu::default]",
      "recipe[build-essential::default]",
      "recipe[openssl::default]",
      "recipe[chef::client]",
      "recipe[chef::delete_validation]",
      "recipe[git::default]",
      "recipe[sysadmin::default]",
      "recipe[munin::client]",
      "recipe[screen::default]",
      "recipe[ssh::server]",
      "recipe[iptables::default]",
      "recipe[users::default]",
      "recipe[sudo::default]"
    ],
    "description": "Basic Server",
    "chef_type": "role",
    "override_attributes": {
    }
  }

To get all of these to run, there are a bunch of dependencies:



chef-repo$ ls cookbooks/
        build-essential/ hosts/           mysql/           runit/           sysadmin/
37s_mysql/       chef/            iptables/        openssl/         screen/          ubuntu/
README           couchdb/         java/            perl/            ssh/             users/
apache2/         erlang/          maatkit/         rabbitmq_chef/   ssh_known_hosts/ xml/
apt/             git/             munin/           ruby-shadow/     sudo/            zlib/

As you can see, I’m using some opscode coobooks and some 37signals ones.

Many of the recipes above required pretty heavy customization to get to work. Note that the iptables chef recipe will lock you out of your server unless you configure it properly before running it. Furthermore, the ssh recipe also required a good amount of customization. If you need help figuring out how to configure the iptables recipe, e-mail me. Unless I’m getting slammed with e-mails, I’ll be happy to help.

However, I am still pretty new at all this, so I really recommend the #chef irc channel on freenode. One of the guys there seemed pretty helpful.

For reading cookbooks, I recommend starting with the Ruby files in the recipes folder, and then looking at the templates and attributes to figure out what data you’ll need to set in the role or node.

For the 37s cookbooks, I recommend treading carefully. Some of them are for HARD CORE servers. The 37s mysql one, for example, does way more than most companies will ever need. Furthermore, they often assume things about 37s systems that will not be true for your systems most likely. For example, I think that they assume a 37s image that has some packages and stuff pre-installed or waiting in various locations.

Overall, I recommend customizing the cookbooks to suit your size and organization. I hope that this was useful, and that it saves others the pain of having to aggregate all of the getting started information. For further help, I recommend the opscode wiki and the #chef channel on freenode.

Please e-mail me if there are any problems with this tutorial. You should also subscribe and follow me on github!

Also please come work with me if you are smart and in Chicago!

July 31, 2010

Getting started with Chef tutorial

Next steps

The Momoro Hoax

Recently,

github