Author: Michael Lucas

I use SolusVM as a virtualization solution, mainly because it’s pretty cheap and mostly effective. The new web-managed migration feature requires that the master node have SSH access into the slave nodes. As root. (Insert lots of swearing here.)

This isn’t a problem, except that I centrally manage my OpenSSH configuration with Ansible. I don’t want all of my hosts to permit the master SolusVM node to log in as root; I only want the Solus hosts to get that setting, and even then only from the master node’s IP address.

The good news is, I have an Ansible group defined for SolusVM hosts. I must modify my template so that it checks if the host is in this group. Ansible provides two variables for the host name, ansible_fqdn (fully qualified name) and ansible_hostname (short hostname). Use the one that reflects your inventory file.

{% if ansible_fqdn in groups['solus-hw'] %}
#solus needs root login from master node
Match Address 192.0.2.12
PermitRootLogin without-password
{% endif %}

As this is a Match statement, it goes at the end of your configuration. My complete sshd_config.j2 looks like this:

#{{ ansible_managed }}
Protocol 2
ListenAddress {{ ansible_ssh_host }}
X11Forwarding yes
PubkeyAuthentication yes
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
{% if ansible_system == 'OpenBSD' %}
#no PAM on OpenBSD
{% else %}
UsePAM yes
{% endif %}
Subsystem sftp {{ program_sftp_server }}
{% if ansible_fqdn in groups['solus'] %}
#solus needs root login from master node
Match Address 192.0.2.12
PermitRootLogin without-password
{% endif %}

Do a push, and done. On to the next problem!

I’ve previously written about managing the OpenSSH server with Ansible. That example focused on my BSD servers. I also manage Ubuntu and CentOS machines as well as my FreeBSD and OpenBSD. While the BSD machines are very similar, Ubuntu and CentOS might are two different operating systems. Can I manage all of them by hand? Sure. But some of these call their SSH service ssh, while others call it sshd. They store their SFTP servers in different directories.

I want to manage all of these simultaneously. With one script. When I need to change my sshd configuration, I want the change to propagate across the network, to all of my machines, simultaneously.

Ansible makes it pretty easy to write a single sshd configuration that works on all systems, through templates. A template is a Jinja2 file that fills in variables. Jinja2 is closely related to Python, but it isn’t Python.

The first thing you must do is decide what how you want your SSH server configured. Most operating systems ship a very inclusive example sshd_config, with most options commented out out. But to see how your SSH server is actually configured, run sshd -T.

# sshd -T
port 22
protocol 2
addressfamily any
listenaddress [::]:22
listenaddress 0.0.0.0:22
usepam 1
serverkeybits 1024
logingracetime 120
keyregenerationinterval 3600
x11displayoffset 10
…

This prints out the running sshd configuration, with all the options as the running instance uses them.

Generate this configuration across your operating systems. Compare them. Which settings do you really want? What differences don’t matter in your environment? Which differences are set in your server by default? (If you aren’t familiar with the innards of OpenSSH configuration, permit me to suggest a book on the subject.)

Some operating systems set their default sshd_config options differently. For example, FreeBSD’s default is UsePam yes. CentOS and Ubuntu default to UsePam no. I’m safest if I specify UsePAM in my configuration file, to avoid ambiguity.

But OpenBSD doesn’t use PAM. If I use that configuration item in an sshd_config for that machine, then sshd won’t run.

Similarly, there’s the location of the sftp-server program. Different operating systems store these helper programs in different locations.

The way to handle all of these is to use a group variables, and assign the things that change a variable. Here’s my new cross-platform template for sshd_config.

#{{ ansible_managed }}
Protocol 2
ListenAddress {{ ansible_ssh_host }}
X11Forwarding yes

PubkeyAuthentication yes
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
{% if ansible_system == 'OpenBSD' %}
#no PAM on OpenBSD
{% else %}
UsePAM yes
{% endif %}

Subsystem sftp {{ program_sftp_server }}

Each per-operating-system group file defines the variable program_sftp_server. Ansible automatically populates the variable ansible_system with the operating system name. And ansible_ssh_host comes from a per-host variable.

I would rather use something like “unless ansible_system=’OpenBSD'” to print UsePAM in the configuration file for the operating systems that need it, but that doesn’t seem to be an option in Jinja2.

The playbook changes a little as well:

---
- hosts: managed-sshd
user: ansible
sudo: yes

tasks:
- name: create key directory
action: file path=/etc/ssh/authorized_keys state=directory
owner=0 group=0 mode=0755

- name: upload user key
action: copy src=/home/ansible/crossplatform/etc/ssh/authorized_keys/{{ item }}
dest=/etc/ssh/authorized_keys/
owner=0 group=0 mode=644
with_items: sshusers

- name: sshd configuration file update
template: src=/home/ansible/crossplatform/etc/ssh/common-sshd_config.j2
dest=/etc/ssh/sshd_config
backup=yes
owner=0 group=0 mode=0644
validate='/usr/sbin/sshd -T -f %s'
notify:
- restart sshd

handlers:
- name: restart sshd
service: name={{ service_sshd }} state=restarted

The key difference here is that I’ve defined a per-OS variable, service_sshd. The operating systems can call their sshd servers sshd, ssh, or derangedweasel for all I care. I figure it out once, set it in the variable file, and get on with my life.

My environment has two common tasks when managing OpenSSH servers: copying user’s authorized_keys files to the server, and changing the sshd configuration file /etc/ssh/sshd_config. I use Ansible for both, using a single playbook. Running the playbook updates all the authorized_keys files on every host and verifies that sshd is properly configured. (Not that any of my minions would reconfigure sshd without going through change control, or anything like that.)

I’ll start with authorized_keys management.

My servers are grouped by functions. Not all users have access to all servers, and I only want to copy a user’s authorized_keys file to a server if that user can access that server. Start by creating a group_vars directory at the top level of your Ansible install. Create a file for each group that you manage authorized_keys on. Within that file, create a YAML list of all users who can access your server. For example, my DNS servers are all in the Ansible group “dns.” I’ve created the file /etc/ansible/group_vars/dns, and it contains:

---

#users who get SSH access to these machines
sshusers:
  - mwlucas
  - ansible
  - john
  - harry
  - mandy

I have a similar file for my LDAP servers, in the group “ldap.” The file is /etc/ansible/group_vars/ldap.

---

#users who get SSH access to these machines
sshusers:
  - mwlucas
  - ansible
  - jake
  - harry

I think you see the trend here. Remember, YAML lists are space-sensitive, and tabs are forbidden.

You cannot define these lists in /etc/ansible/hosts: you must use a variables file.

Then there’s the SSH configuration part. I run an artisan network. (“Artisan network” means “each server is set up uniquely, managed uniquely, and has little in common with anything else, really frustrating any hopes of easy or consistent systems administration.” While I’m slowly dragging everything towards mass manageability, it’s not there yet.) Each host has multiple IP addresses. I want sshd to only listen on a single IP address. I can’t consistently pull the IP from the host itself, so I need to hard-code it in Ansible.

Fortunately, Ansible should already have this information. A host’s ssh daemon probably isn’t listening on the IP address that DNS gives for the host — that is, the IP referred to by the hostname ldap.michaelwlucas.com probably doesn’t have an SSH server on it. It might be ldap-ssh or ldap-mgmt or ldap-pleasekillmenow. So I’ve had to tell Ansible the address to connect to.

The directory /etc/ansible/host_vars contains a file for each host. The file /etc/ansible/host_vars/ldap.michaelwlucas.com contains something like this:

---

ansible_ssh_host: 192.0.2.88

The ansible_ssh_host is a dedicated Ansible variable, giving the hostname or IP address that Ansible should use to SSH into this host. I’m going to piggyback this for my SSH configuration file. (If you use hostnames in your ansible_ssh_host, you’ll need to either create a separate variable with the IP or to convert the hostname to an IP address somewhere in this process. If you figure out an easy way to do this, do let me know.)

So, how do I want sshd configured?

As I’m mass-managing these machines’ sshd service, I neither need nor want all the various default options in sshd_config. Having those options in the configuration file is great when you’re manually configuring sshd, but I don’t want anyone on any of these servers to change the sshd configuration without going through change control.

So, let’s go to a machine that has a properly configured sshd and get the non-default options.

$ grep -v ^# /etc/ssh/sshd_config > sshd_config.j2

The .j2 indicates this is a Jinja2 template, which is what Ansible uses for creating files. Look at what’s left after I remove the blank lines.

ListenAddress 192.0.2.88
PermitRootLogin without-password
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
Subsystem sftp /usr/libexec/sftp-server

The only problem here is the ListenAddress setting, which needs to be different on each host. I make that a variable. I also add a revision tag and an Ansible management statement.

#$Id$
#{{ ansible_managed }}
ListenAddress {{ ansible_ssh_host }}
PermitRootLogin without-password
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
Subsystem sftp /usr/libexec/sftp-server

The {{ ansible_ssh_host }} is a variable statement, pulling in that variable from Ansible itself.

So, how to use all this in a playbook?

---
- hosts: freebsd
  user: ansible
  sudo: yes

  tasks:
  - name: create key directory
    action: file path=/etc/ssh/authorized_keys state=directory
      owner=0 group=0 mode=0755

  - name: upload user key
    action: copy src=/home/ansible/crossplatform/etc/ssh/authorized_keys/{{ item }}
      dest=/etc/ssh/authorized_keys/
      owner=0 group=0 mode=644
    with_items: sshusers

  - name: sshd configuration file update
    template: src=/etc/ansible/configs/etc/ssh/sshd_config.j2
      dest=/etc/ssh/sshd_config
      backup=yes
      owner=0 group=0 mode=0644
      validate='/usr/sbin/sshd -T -f %s'
    notify:
    - restart sshd

  handlers:
    - name: restart sshd
      service: name=sshd state=restarted

The first task creates the directory for key storage. I do not allow users to upload authorized_keys files for their own account. We don’t want an intruder to add their own key to a user account. Instead, each user’s authorized keys are in a file named after the username, in the directory /etc/ssh/authorized_keys, and our sshd_config tells sshd to look in that directory.

The second task copies the user keys listed in the sshusers variable defined in the group_vars file. While Ansible has an authorized_keys module specifically for handling these files, it has problems with quotes in restricted keys. Until that’s fixed, I’ll fall back to the perfectly adequate “copy” module.

The third task reads the jinja2 template for sshd_config, adds the necessary information, and copies the file to the server. It also validates that the configuration is legitimate — not that it will do what you want, mind you, but it will verify that sshd understands this sshd_config file.

Last, we restart sshd.

My next steps with this will be to update the template so it works on non-FreeBSD hosts. But that’ll be a topic for another day.

Want more on configuring SSH? Check out my book SSH Mastery.