Basic Ansible Playbooks

Ansible is a tool for managing servers en masse, much like Puppet or CFEngine. Ansible has a shallower learning curve than either of those systems, however, and it’s idempotent. How do I know it has a shallower learning curve? Because I learned enough of it to do actual useful work in only a couple of hours.

And before you reach for a dictionary, “idempotent” means that you can run the same script against your servers and have the end result be the same. You can run an Ansible script (or playbook) against a group of servers, take note of those that fail, modify the script, and run it again against the same group of servers, and Ansible will verify that the servers need the playbook run before running it. Only the servers that need the change will get it.

Why would this ever happen? Maybe a datacenter is cut off by a network issue, or a LDAP server chokes, or gremlins invade a server, or a script fails because an intruder has hacked the server and this is your early warning. Your management tools need to deal with all of these.

For example, I have an Ansible playbook that uploads a new PF configuration file and reloads the PF rules. Ansible compares the existing PF configuration to one being distributed, and if the file hasn’t changed, doesn’t reload the rules. This isn’t a huge deal for PF, but for some applications it’s vital.

Another nice feature about Ansible is that it uses only SSH and Python. Most Unixes acquire Python as an application dependency somewhere along the way, and it’s small enough that I have no real objection to installing on servers without it. And both Puppet and CFEngine have dedicated agent software, so some kind of agent is going to wind up on the managed machine anyway.

The biggest problem I had with Ansible was with playbooks. There’s a whole bunch of playbook documentation, and Ansible ships with sample playbooks, but they’re written somewhat like man pages, for people who already have some clue about the topic. So here are a couple really rudimentary Ansible playbooks, with explanations.

---
- hosts: pf
  user: ansible
  sudo: yes
  tasks:
  - name: copy pf.mgmt.conf to servers
    action: copy src=/home/ansible/freebsd/etc/pf.mgmt.conf 
      dest=/etc/pf.mgmt.conf owner=root group=wheel mode=0644
    notify:
      - reload pf

  handlers:
    - name: reload pf
      action: shell /sbin/pfctl -f /etc/pf.conf

Ansible playbooks are written in YAML, Yet Another Markup Language. I concede that my first thought on hearing the words “yes another markup language” is “that statement needs some obscenities between the second and third word.” But XML would be way overkill for Ansible. (And the YAML folks have changed their name to a different acronym, Yaml Ain’t Markup Language, trying to escape the stigma of being yet another bleeping bleepety-bleep markup language.)

All YAML files start with a triple dash. They are space-sensitive — don’t use tabs, only spaces.

At the top level (no indents), we have the triple dash and a dash

---
- hosts: pf

The leading hyphen basically means “new thing here,” as far as I can tell.

At the second level of configuration, indented two spaces, we have five sections: hosts, user, sudo, tasks, and handlers.

The hosts statement gives the name of a group of hosts. Ansible has an easily understood hosts file. This playbook applies to a group of hosts called “pf.”

The user definition tells ansible which user to use. Ansible should SSH into the target servers as the user “ansible.”

The sudo statement tells ansible to use sudo to perform this command. My ansible user on each host has sudo privileges, but needs a password. We’ll get an opportunity to enter the password when we run the playbook. (Ansible could log into the target server as root and avoid the need for sudo, or let the ansible user have sudo without a password, but the thought of doing either makes my spleen threaten to leap up my gullet and block my windpipe, so I don’t.)

The tasks section is where things get interesting. We actually do stuff here. I define a third level of indentation (four spaces) after the tasks statement, and start them with a dash.

Our first task has a name, “copy pf.mgmt.conf to servers.”

The action that follows uses the Ansible copy module. I define a source file, a destination file, and set the owner and permissions.

The notify statement tells the task to activate the handler named “reload pf” If the action changes a target system, the action triggers the handler. If the action doesn’t change anything, the handler is not triggered.

We then have the handler section. It’s at the same indent level as tasks, sudo, user, and hosts, so it’s a major section. There’s one handler, “reload pf.” It performs one action, fires up a shell and runs a command.

Taken as a whole, this playbook copies a file to all the servers in the pf group and reloads the file. The file pf.mgmt.conf contains the IP addresses of my management hosts, as I discussed elsewhere.

Now let’s look at a slightly more complex playbook that does the same thing.

---
- hosts: linux-internal
  user: ansible
  sudo: yes
  tasks:
  - name: copy iptables.mgmt.conf to servers
    action: copy src=/home/ansible/linux/etc/iptables.mgmt.conf 
      dest=/etc/iptables.mgmt.conf owner=root group=root mode=0644
    notify:
      - reload ipset mgmt
  - name: copy iptables.rules to servers
    action: copy src=/home/ansible/linux/etc/solus.iptables.rules 
      dest=/etc/iptables.rules owner=root group=root mode=0644
    notify:
      - reload iptables

  handlers:
    - name: reload ipset mgmt
      action: shell /usr/sbin/ipset restore -! < /etc/iptables.mgmt.conf
    - name: reload iptables
      action: shell /sbin/iptables-restore -! < /etc/iptables.rules

This playbook updates the firewall rules on my Linux hosts. These CentOS hosts are a little simpler in that they all share a common function (virtualization). They can have a common iptables ruleset as well as a common list of management addresses. I talk about how I use ipsets, and why the rules are set up this way, elsewhere. But the important thing is:

  • This is a single procedure, so it's one playbook.
  • It updates two separate files.
  • Changing each file runs a separate command.

    So, if the iptables.rules file changes, Ansible runs iptables-restore. If iptables.mgmt.conf changes, Ansible runs ipset.

    To use these playbooks, I log in as the ansible user on the ansible server and run:

    $ ansible-playbook -K playbook-file-name.yml

    The -K tells ansible to ask for the sudo password. If your ansible user doen't need a sudo password, skip it. (But beware your spleen.)

    Ansible will log onto every host in the group, check the files, update them if needed, and run the handler commands if it updates the files.

    Ansible has many more modules than just copying files and running commands. It can assemble files from variables, install packages, and more. But a few small playbooks will get you started, and even the basic steps of managing servers firewall rules en masse will save you enough time to figure out the new modules.

    I have no doubt that Puppet and CFEngine have serious use cases and environments where they're the best choice. What my network is most short on is sysadmin brainpower, however, and Ansible is a good fit for my feeble brain.

  • iptables and ipsets

    I’m dragging my work environment from “artisan system administration” to mass-managed servers. Part of this is rationalizing, updating, and centralizing management of packet filter rules on individual hosts. Like many environments, I have a list of “management IP addresses” with unlimited access to every host. Managing this is trivial on a BSD machine, thanks to pf.conf’s ability to include an outside file — you upload the new file of management addresses and run pfctl to read it. A PF rules file looks something like this:

    ext_if="em0"
    include "/etc/pf.mgmt.conf"
    ...
    pass in on $ext_if proto icmp from any to any
    #mgmt networks can talk to this host on any service
    pass in on $ext_if from to any
    ...

    The file pf.mgmt.conf looks like this:

    table const { 192.0.2.0/24, 198.51.100.128/25 }

    When I add new management addresses I copy pf.mgmt.conf to each machine, run pfctl -f /etc/pf.conf, and the new addresses can connect.

    But surely there’s some similar function on a Linux box?

    To complicate matters further, our environment includes both Ubuntu and CentOS machines. (Why? Because we don’t run operating systems, we run applications, and applications get picky about what they run on.) Each version has its own way of saving and restoring iptables rules. I want to use the same method for both operating systems. What we’ve used is a single rules file, /etc/iptables.rules, read by iptables-restore at boot. We specifically don’t want to trust a copy of the packet filter rules saved by the local machine, as problems can persist across reboots. The current iptables.rules looks something like this:

    *filter
    #mgmt addrs
    -A INPUT -s 192.0.2.0/24 -i eth0 -j ACCEPT
    -A INPUT -s 198.51.100.128/25 -i eth0 -j ACCEPT
    #keep state
    -A INPUT -p tcp -m state --state ESTABLISHED -j ACCEPT
    -A OUTPUT -p tcp -m state --state NEW,ESTABLISHED -j ACCEPT
    -A INPUT -p udp -m state --state ESTABLISHED -j ACCEPT
    -A OUTPUT -p udp -m state --state NEW,ESTABLISHED -j ACCEPT
    #local stuff here
    ...
    #permit ICMP
    -A INPUT -p icmp -j ACCEPT
    -A OUTPUT -p icmp -j ACCEPT
    -A INPUT -i eth0 -j DROP
    COMMIT

    I don’t want to change /etc/iptables.rules for each machine at this point. They all vary slightly. (One day the machines will be classified by roles, but we’re in an intermediate stage right now.) Instead, I want to have the list of management addresses in a separate file. I want to copy the new file to the server, run a command, and have the new list of management addresses be live.

    ipsets seems to be the way to do this. Let’s find out.

    On my crashbox, I’ll create an ipset. I’m using an ipset of type nethash, because it takes CIDR blocks rather than individual IP addresses. The ipset is called mgmt, just like the management addresses on my BSD machines.

    # ipset create mgmt nethash

    It returns silently. Did it create the ipset?

    # ipset list
    Name: mgmt
    Type: hash:net
    Header: family inet hashsize 1024 maxelem 65536
    Size in memory: 16760
    References: 0
    Members:

    OK, it’s in memory. Now add some addresses.

    # ipset add mgmt 192.0.2.0/24
    # ipset add mgmt 198.51.100.128/25

    Are those addresses really in the set? Let’s ask again.

    # ipset list mgmt
    Name: mgmt
    ...
    Members:
    192.0.2.0/24
    198.51.100.128/25

    Now, export this to a file.

    # ipset save mgmt > iptables.mgmt.conf

    I use the file iptables.mgmt.conf to mirror pf.mgmt.conf. That file should contain something like this:

    create mgmt hash:net family inet hashsize 1024 maxelem 65536
    add mgmt 192.0.2.0/24
    add mgmt 198.22.63.128/25
    add mgmt 198.51.100.128/25

    Can I restore the ipset from the file? Destroy the set.

    # ipset destroy mgmt
    # ipset list

    It’s gone. Now to restore it from memory.

    # ipset restore < iptables.mgmt.conf # ipset list
    ...

    All my rules are there.

    Now, let’s teach iptables how to use an ipset. Rather than defining addresses, we use the -m set option.

    # iptables -A INPUT -i eth0 -m set --match-set mgmt src -j ACCEPT

    In the iptables.rule file, it would look like this.

    *filter
    #allow mgmt IPs
    -A INPUT -i eth0 -m set --match-set mgmt src -j ACCEPT
    ...

    When you have several management networks, this is certainly much shorter and easier to read.

    When you update the iptables.mgmt.conf file, read it in with ipset restore. You must use the -! flag. This tells ipset to ignore that the ipset already exists, and restore the contents of the ipset from the file.

    # ipset restore -! < iptables.mgmt.conf

    I can now copy this file to my hosts, run a command, and the packet filter rules are updated, without touching my main rules file.

    I don’t recall anyone using a symbol as a command-line flag like this before, but I actually kind of like this one. “I said DO IT, damn you!”

    Penguicon Schedule

    Someone pointed out that it would be nice to know when I’ll be doing what at Penguicon. So, here’s my schedule.

    Friday, 26 April:
    6PM-7PM opening ceremonies
    7PM-8PM Guest of Honor Social Hour
    9PM-10PM OpenBSD (kind of obligatory)

    Saturday, 27 April:
    10AM-noon: liquid nitrogen ice cream, my own flavor
    5PM-6PM: Technology Publishing in 2013

    Sunday 28 April:
    3PM-4PM: closing ceremonies

    Other than that, I’m basically free. You can find me elsewhere, as I’ll be playing free range author, but you can definitely intercept me at these times.

    At Penguicon this weekend

    I’m a guest of honor at Penguicon this weekend. I’ll be doing a one-hour talk on technology publishing in 2013. I’ll also be inventing an ice cream flavor.

    Other guests include author Jim C Hines, chef and author Jeff Potter, Arduino and Raspberry pi guru & author Maik Schmidt, hackerspace advocate and censorship activist Nick Farr, plus author and filmmaker Jason Denzel.

    Whenever I go to this type of con, I usually take in a few panels, see some cool demos, and spend a good chunk of time hanging out in the bar or lobby, wondering who all these people are. If you’re in the Detroit area, come by and say hello.

    I will also have hard copies of Absolute OpenBSD for sale at $50 each. Amazon doesn’t have them yet. No Starch doesn’t have them yet. But I have them. Cash and PayPal accepted — I had hoped to take credit cards, but the PayPal credit card app doesn’t like my knockoff Android tablet. But by showing up, you can get the book you’ve been demanding for four years now.

    And now that I’m a guest, maybe the cool kids will let me hang with them. But probably not.

    “Absolute OpenBSD” physically exists

    I hold in my hands the first copy of Absolute OpenBSD off the press.

    IMG00371The fine folks at No Starch Press did a beautiful job designing and creating the print book. I know ebooks are the coming thing, but a well-designed physical book is a delight.

    This specific copy is for Bill Allaire. I’ll have my own copy. But I’m sure Bill won’t mind if I spend a few minutes fondling his book.

    The loop is closed, the project is over. Now to lie back and let the fame and wealth just roll into — Sorry, I can’t even type that with a straight face.

    And if you ask me when the third edition is coming out, I’ll be sure to thank you. With an axe.

    “DNSSEC Mastery” now complete, ebook version available!

    You can now get the complete DNSSEC Mastery: Securing the Domain Name System with BIND at Amazon, Barnes & Noble, Smashwords, and my personal ebookstore. It should (hopefully) trickle through to iTunes & such before long.

    This book was a real education to write. Hopefully it will help improve the state of DNS security across the industry. Various DNS experts have expressed approval of the book, and here’s hoping that the wider world will as well.

    The book has now gone on to physical production. Hopefully I will have a proof by BSDCan. We might even auction it off at the end of the con, as the OpenBSD auction did so well.

    Review copies are available for folks who regularly review books.

    If you should find an error in the ebook, please let me know. Converting an OpenOffice document to umpteen different formats for an incredibly wide variety of devices has its risks. I no longer have a PalmPilot to test that format, for example, and I have a specific model of Kindle that’s probably not the same as yours.

    Thanks to everyone for their support.

    “Absolute OpenBSD” auction winner

    The final total on the Absolute OpenBSD first copy auction was: $1145. The lucky (for various interpretations of lucky) winner is Bill Allaire, long-time OpenBSD supporter.

    Bill has already sent Austin the money, which is winging its way to the OpenBSD Foundation as I write this. So he’s lived up to his end of the bargain.

    I’m not going to ship him the book when it’s printed, however. Us writers are flaky and untrustworthy, and it’s time people realize and accept this. For another reason, though, there’s a small OpenBSD hackathon in Toronto at the end of May. I’ve been invited to come hang out on beer night.

    I’m taking Bill’s book with me, so various OpenBSD developers can sign it and point out any errors. I’m sure they’ll also offer corrections and commentary because, well, you give a BSD developer a beer and he’ll tell you what he really thinks.

    So, Bill not only gets the first copy off the press. He gets the most correct version. A book more awesome than anything anyone else will have. Don’t you wish you had outbid him?

    “Absolute OpenBSD” auction clears $1000

    Apparently double-dog daring you people works.

    The Absolute OpenBSD Foundation auction is currently at $1035.

    You have another day and a half to bid. Do so.

    I’m delighted. And, as promised, I’ll have an ebook sale as a result. Probably during BSDCan.

    This week has been maniacal for reasons completely outside of the new Absolute OpenBSD, but I should start to catch up on all the queued email, tweets, and whatnot next week.

    “Absolute OpenBSD, 2nd edition” ebook download available

    I’m told that No Starch Press now has the ebook for the new Absolute OpenBSD available for download. If you preordered the book, go get the electronic version while you’re waiting for the print to arrive.

    If you haven’t preordered, go get it now. If you use coupon code ILUVMICHAEL, you save 30% and I make a couple extra bucks on it.

    The auction for the first print copy off the press is up to $910. If it breaks $1000, I’ll post a coupon code for discounts on the electronic versions of my Mastery books from my Web site.

    First barrier breached

    The Absolute OpenBSD auction has been going about 10 hours now. In less than those 10 hours, the price exceeded the amount raised for the FreeBSD Foundation.

    Well done.

    But I bet you slackers can’t possibly double it. No, I DOUBLE DOG-DARE YOU to double it.

    How I love picking a fight in a good cause.