Category: Tech

Ansible is a tool for managing servers en masse, much like Puppet or CFEngine. Ansible has a shallower learning curve than either of those systems, however, and it’s idempotent. How do I know it has a shallower learning curve? Because I learned enough of it to do actual useful work in only a couple of hours.

And before you reach for a dictionary, “idempotent” means that you can run the same script against your servers and have the end result be the same. You can run an Ansible script (or playbook) against a group of servers, take note of those that fail, modify the script, and run it again against the same group of servers, and Ansible will verify that the servers need the playbook run before running it. Only the servers that need the change will get it.

Why would this ever happen? Maybe a datacenter is cut off by a network issue, or a LDAP server chokes, or gremlins invade a server, or a script fails because an intruder has hacked the server and this is your early warning. Your management tools need to deal with all of these.

For example, I have an Ansible playbook that uploads a new PF configuration file and reloads the PF rules. Ansible compares the existing PF configuration to one being distributed, and if the file hasn’t changed, doesn’t reload the rules. This isn’t a huge deal for PF, but for some applications it’s vital.

Another nice feature about Ansible is that it uses only SSH and Python. Most Unixes acquire Python as an application dependency somewhere along the way, and it’s small enough that I have no real objection to installing on servers without it. And both Puppet and CFEngine have dedicated agent software, so some kind of agent is going to wind up on the managed machine anyway.

The biggest problem I had with Ansible was with playbooks. There’s a whole bunch of playbook documentation, and Ansible ships with sample playbooks, but they’re written somewhat like man pages, for people who already have some clue about the topic. So here are a couple really rudimentary Ansible playbooks, with explanations.

---
- hosts: pf
  user: ansible
  sudo: yes
  tasks:
  - name: copy pf.mgmt.conf to servers
    action: copy src=/home/ansible/freebsd/etc/pf.mgmt.conf 
      dest=/etc/pf.mgmt.conf owner=root group=wheel mode=0644
    notify:
      - reload pf

  handlers:
    - name: reload pf
      action: shell /sbin/pfctl -f /etc/pf.conf

Ansible playbooks are written in YAML, Yet Another Markup Language. I concede that my first thought on hearing the words “yes another markup language” is “that statement needs some obscenities between the second and third word.” But XML would be way overkill for Ansible. (And the YAML folks have changed their name to a different acronym, Yaml Ain’t Markup Language, trying to escape the stigma of being yet another bleeping bleepety-bleep markup language.)

All YAML files start with a triple dash. They are space-sensitive — don’t use tabs, only spaces.

At the top level (no indents), we have the triple dash and a dash

---
- hosts: pf

The leading hyphen basically means “new thing here,” as far as I can tell.

At the second level of configuration, indented two spaces, we have five sections: hosts, user, sudo, tasks, and handlers.

The hosts statement gives the name of a group of hosts. Ansible has an easily understood hosts file. This playbook applies to a group of hosts called “pf.”

The user definition tells ansible which user to use. Ansible should SSH into the target servers as the user “ansible.”

The sudo statement tells ansible to use sudo to perform this command. My ansible user on each host has sudo privileges, but needs a password. We’ll get an opportunity to enter the password when we run the playbook. (Ansible could log into the target server as root and avoid the need for sudo, or let the ansible user have sudo without a password, but the thought of doing either makes my spleen threaten to leap up my gullet and block my windpipe, so I don’t.)

The tasks section is where things get interesting. We actually do stuff here. I define a third level of indentation (four spaces) after the tasks statement, and start them with a dash.

Our first task has a name, “copy pf.mgmt.conf to servers.”

The action that follows uses the Ansible copy module. I define a source file, a destination file, and set the owner and permissions.

The notify statement tells the task to activate the handler named “reload pf” If the action changes a target system, the action triggers the handler. If the action doesn’t change anything, the handler is not triggered.

We then have the handler section. It’s at the same indent level as tasks, sudo, user, and hosts, so it’s a major section. There’s one handler, “reload pf.” It performs one action, fires up a shell and runs a command.

Taken as a whole, this playbook copies a file to all the servers in the pf group and reloads the file. The file pf.mgmt.conf contains the IP addresses of my management hosts, as I discussed elsewhere.

Now let’s look at a slightly more complex playbook that does the same thing.

---
- hosts: linux-internal
  user: ansible
  sudo: yes
  tasks:
  - name: copy iptables.mgmt.conf to servers
    action: copy src=/home/ansible/linux/etc/iptables.mgmt.conf 
      dest=/etc/iptables.mgmt.conf owner=root group=root mode=0644
    notify:
      - reload ipset mgmt
  - name: copy iptables.rules to servers
    action: copy src=/home/ansible/linux/etc/solus.iptables.rules 
      dest=/etc/iptables.rules owner=root group=root mode=0644
    notify:
      - reload iptables

  handlers:
    - name: reload ipset mgmt
      action: shell /usr/sbin/ipset restore -! < /etc/iptables.mgmt.conf
    - name: reload iptables
      action: shell /sbin/iptables-restore -! < /etc/iptables.rules

This playbook updates the firewall rules on my Linux hosts. These CentOS hosts are a little simpler in that they all share a common function (virtualization). They can have a common iptables ruleset as well as a common list of management addresses. I talk about how I use ipsets, and why the rules are set up this way, elsewhere. But the important thing is:

One of the most tedious tasks any network admin faces is replicating changes across multiple devices. I recently stood up new RADIUS servers, and needed to tell all of my routers and switches about it. Rather than logging into each router by hand and pasting in the new configuration, I decided to try RANCID‘s ability to run arbitrary commands on your routers.

Using this method requires that the commands you run don’t generate interactive output. A reload command won’t work, because it prompts you for confirmation. But adding configurations to a Cisco router doesn’t.

I assume you have a working RANCID install.

Start by creating a text file containing your commands. RANCID expects to log on and log off of the router. All you need to provide is what happens between those two points.

conf t
radius-server host 192.0.2.2 auth-port 1812 acct-port 1813 key BuyBooksFromLucas
radius-server host 192.0.2.14 auth-port 1812 acct-port 1813 key BuyBooksFromLucas
exit
wr

Now use the device-specific login command, specify the file containing your commands with -x, and list every router you want to run the commands on.

# clogin -x newradius.conf router-1 router-2 router-3

RANCID logs into each device and add the new configuration. You can watch the process in action, and catch any problems.

I found that the Mikrotik login script had problems when the script changed the prompt. I’ve reported this to the RANCID mailing list, and expect it will be patched shortly. But fortunately, that’s pretty easy to work around in a Mikrotik, by giving the entire command in one line, as shown below.

/radius add accounting-backup=no accounting-port=1813 address=192.0.2.2 authentication-port=1812 called-id="" disabled=no domain="" realm="" secret=BuyBooksFromLucas service=login timeout=300ms
/radius add accounting-backup=no accounting-port=1813 address=192.0.2.14 authentication-port=1812 called-id="" disabled=no domain="" realm="" secret=BuyBooksFromLucas service=login timeout=300ms

Having RANCID run commands for you is much more accurate and less tedious than doing it yourself. And this way, if you make a mistake in your commands, at least it’ll be consistent across all your devices.

One of the problems with the Internet is that old stuff hangs around forever. Configuring DNSSec validation on BIND 9.8 and newer is a lot easier than many of the popular tutorials would lead you to suspect. It’s so simple that I wonder why it isn’t the default.

options {
...
dnssec-enable yes;
dnssec-validation auto;
dnssec-lookaside auto;
};

This automatically loads the root zone and dlv.isc.org trust anchors distributed with the BIND source code, verifies them, and uses them to validate all signed responses.

One trick is that named needs to write journal files for these keys. If you used the directory option to set a directory writable by named, you’re all set. If, like me, you have a whole bunch of directories that you don’t want named to write to, and you don’t want named to write to /etc/namedb, you can set a directory for these named-managed keys.

/var/log/messages will have errors like this:

Jan 7 11:59:02 ns11 named[78246]: the working directory is not writable

These errors are harmless. named cannot write /etc/namedb, but it will log an error when it tries to write a file. The errors you need to worry about look like this:

Jan 5 13:07:05 ns11 named[68130]: dumping master file: tmp-XI0K5awL6p: open: permission denied

These indicate that named is trying to record important data, and can’t. Giving named a directory where it can write these managed key files will solve this problem.

options {
...
managed-keys-directory "managed-keys";
}

Make that directory owned by the user running named. Restart (not reload) named, and you’re all set.

One of the rules of public key encryption is that your private key must be private. Most of us have had private keys stolen at some time. (If you haven’t, either you’re not using public key crypto, or you didn’t notice.) What happens if someone steals the private key for a DNSSec trust anchor?

There’s a protocol for automatically updating the trust anchor private keys, documented in RFC 5011. It’s not perfect — you’ll still want to keep half an eye out for announcements of a trust anchor compromise.

On a related note, I’d be interested in hearing from anyone who knows how the root zone private key is protected, just so I can put a sentence or two in the DNSSec book.