Category: Tech

It seems that ntpd has turned into the latest DDOS amplifier. I run a lot of servers, and most of them use the standard ntp client. I need to verify that none of my servers can be used for DDOS amplification. To do this, I need to give all the clients a standard NTP configuration, pointing at my personal NTP servers.

While my internal addresses need access to the port 123 on my servers, the public doesn’t. And I occasionally add internal addresses. Automating PF and NTP configuration via Ansible will simplify my life in the future.

I’ve used Ansible templates to configure services before, but packet filters are a little different. Packet filtering rules involve lots of information about the local host, such as interface names and the various system roles. It’s entirely possible to write an Ansible template that expresses your PF ruleset, it just took a little work.

First, I define an Ansible group for the NTP servers (as well as other server duties). The time servers run FreeBSD 9.2.

Here’s the playbook, with added NTP.

---
- hosts: ntp-servers
  user: ansible
  sudo: yes

  tasks:

  - name: enable ntpdate
    action: command sysrc ntpdate_enable=yes
  - name: enable ntpdate server
    action: command sysrc ntpdate_hosts=pool.ntp.org
  - name: enable ntpd
    action: command sysrc ntpd_enable=yes

  - name: copy ntp.server.conf to servers
    action: copy src=/home/ansible/freebsd/etc/ntp.server.conf
      dest=/etc/ntp.conf owner=root group=wheel mode=0644
    notify:
      - restart ntpd

  - include: tasks/pf-compile.yml

  handlers:

    - include: handlers/restarts.yml

Simple enough, no?

Except there’s nothing in here about the packet filter. Or restarting ntp.

These functions are pretty common, so I’ve moved them to separate files. I might need to rebuild the packet filter rules for any number of playbooks, after all.

The file tasks/pf-compile.yml looks like this.

---
#build pf.conf from template

  - name: configure firewall
    template: src=/home/ansible/freebsd/etc/pf.conf.j2
      dest=/etc/pf.conf owner=0 group=0 mode=0444
      validate='/sbin/pfctl -nf %s'
    notify:
      - reload pf

This task uses a jinja2 template to build a pf.conf specifically for this host, copies it to the host, validates its syntax, puts it in place, and triggers a PF reload. Always validate your files before deploying them. Ansible doesn’t prevent mistakes, but rather allows you to deploy mistakes faster than ever.

Similarly, I’ve split the “restart services” handlers off into the file handlers/restarts.yml. Here are the relevant bits.

---
#restart assorted services

  - name: restart ntpd
    service: name=ntpd state=restarted

  - name: reload pf
    action: shell /sbin/pfctl -f /etc/pf.conf

So, where is this firewall template? That’s probably what dragged you here.

#{{ ansible_managed }}
#$Id: pf.conf.j2,v 1.2 2014/01/16 16:10:54 mwlucas Exp $

ext_if="{{ ansible_default_ipv4.device }}"

include "/etc/pf.mgmt.conf"
include "/etc/pf.ournets.conf"

set block-policy return
set loginterface $ext_if
set skip on lo0

scrub in all
block in all

pass in on $ext_if proto icmp all
pass in on $ext_if proto icmp6 all

#this host may initiate anything
pass out on $ext_if from any to any

#mgmt networks can talk to this host on any service
pass in on $ext_if from  to any

#Allowed services, in port order

{% if inventory_hostname in groups['dns'] %}
#DNS access
pass in on $ext_if proto {tcp, udp} from any to any port 53
{% endif %}

{% if inventory_hostname in groups['tftpd'] %}
#allow tftp from the world
pass in on $ext_if proto udp from any to any port 69
{% endif %}

{% if inventory_hostname in groups['ntp-servers'] %}
#allow time from our networks
pass in on $ext_if proto udp from  to any port 123
{% endif %}

#end of services

The first bit of new (to me) trickery in this is getting the interface name. I use an Ansible-provided variable for this. Get the complete list of Ansible-provided variables for a host by running ansible -m setup hostname. The variable ansible_default_ipv4.device contains the network interface name. (If your host has multiple network-facing interfaces, you’ll need to modify this.)

This PF ruleset pulls in two external files, one containing a list of management addresses and one containing the complete list of my internal networks.

I allow access from my management networks, allow ICMP, default block, all the routine packet filter stuff. The next interesting bit is the allowed services. I check for the host’s presence in a group, and if it’s there’ I add a rule to permit the access that protocol needs.

One detail that gave me trouble made me use inventory_hostname rather than ansible_fqdn or ansible_hostname to check group membership. I manage systems in several domains, and many of them have one name in our management systems and another in DNS. I put machines in Ansible by their fully qualified domain name. Using ansible_fqdn to get a hostname returns the hostname given in reverse DNS. inventory_hostname returns the hostname as it appears in the hosts file. If ansible_fqdn doesn’t match the hostname in the hosts file, the group comparison fails. Using inventory_hostname gave me one consistent set of hostnames for comparisons.

So now I can easily deploy a secure NTP configuration to my servers. When I have to deploy some other service that requires updating the packet filter, I can include the same task file. And the handlers are now similarly reusable.

Configuring the clients across several different operating systems will probably require Ansible roles, however. I’d best get on that next…

After years of only needing central auth for Unix-like systems, I need to integrate Windows clients into my auth mix. Rather than munging my current OpenLDAP directory to contain Windows information, I elected to migrate to Samba 4. Samba 4 can act as a Windows domain controller and also exposes an LDAP interface for Unix clients.

I assume that you’ve read the FreeBSD and Samba documentation on LDAP auth. This article is meant as a bridge between the two information sets.

Most of the tutorials out there cover using LDAP and Kerberos with Samba. Kerberos requires that all hosts be in a single domain. My employer manages hosts within a variety of domains and business units, so Kerberos is a no-go. I need to use pure LDAP authentication.

I configured my samba4 domain and ensured that Windows clients could join the domain and that the general Microsoft-esque features worked, such as failover to a backup domain controller. Once it appeared that everything worked, I set up a couple of OpenLDAP proxies exactly by the Samba documentation. (My domain controllers are in private address space, and I’m not willing to expose them to the larger network.)

Then I created an account for services to bind with to perform basic queries. My binding account is called unixstuff. It’s not a member of any AD groups.

Then I can configure OpenLDAP on the client. Install the various LDAP utilities as in the FreeBSD documentation.

Configure the OpenLDAP tools to query your directory first. I use a private CA, so I set assorted TLS options.

BASE dc=internal,dc=mwlucas,dc=org
URI ldap://snarky.mwlucas.org
ssl start tls
tls_cacert /usr/local/etc/ssl/mwlucas.crt

At this point I should be able to use ldapsearch on the Samba directory.

# ldapsearch -WxD "cn=unixstuff,cn=users,dc=internal,dc=mwlucas,dc=org"
Enter LDAP Password:
...

If you enter the correct password, the directory should spill its guts.

Once that works, we can configure LDAP authentication in /usr/local/etc/ldap.conf.

host snarky.mwlucas.org
base dc=internal,dc=mwlucas,dc=org
ldap_version 3
binddn cn=unixstuff,cn=users,dc=internal,dc=mwlucas,dc=org
bindpw WhyHardCodePasswords

pam_filter objectclass=posixAccount
pam_login_attribute uid
pam_member_attribute member
pam_lookup_policy yes
nss_base_passwd cn=users,dc=internal,dc=mwlucas,dc=org
nss_base_group cn=users,dc=internal,dc=mwlucas,dc=org
pam_groupdn cn=sysadmins,cn=users,dc=internal,dc=mwlucas,dc=org

# Services for UNIX 3.5 mappings
nss_map_attribute homeDirectory unixHomeDirectory
nss_map_objectclass posixAccount User
nss_map_objectclass shadowAccount User
nss_map_attribute uid msSFU30Name
nss_map_attribute uniqueMember msSFU30PosixMember
nss_map_attribute userPassword msSFU30Password
nss_map_objectclass posixGroup Group
pam_filter objectclass=User
pam_password ad

With this set, and LDAP activated in /etc/nsswitch.conf, I can now “getent passwd” and “getent group” and get responses from the directory.

That still leaves PAM. I’m no PAM expert, but I’ve hacked together something that permits LDAP access, and falls back to the local password file when the LDAP servers are unavailable. Here’s my /etc/pam.d/system:


auth sufficient /usr/local/lib/pam_ldap.so
auth required pam_unix.so no_warn try_first_pass nullok

account required pam_login_access.so
account sufficient /usr/local/lib/pam_ldap.so
account required pam_unix.so

session required /usr/local/lib/pam_ldap.so
session required pam_lastlog.so no_fail

password required pam_unix.so no_warn try_first_pass

This gets you basic access. Realistically, though, users want a home directory. Install pam_mkhomedir and enable it on a per-service basis.

auth sufficient /usr/local/lib/pam_ldap.so no_warn
auth required pam_unix.so no_warn try_first_pass

account required pam_nologin.so
account required pam_login_access.so
account required pam_unix.so
account required /usr/local/lib/pam_ldap.so no_warn ignore_authinfo_unavail ignore_unknown_user

session required pam_permit.so
session required /usr/local/lib/pam_mkhomedir.so

password required pam_unix.so no_warn try_first_pass

With this set, users in the “sysadmins” group in AD have shell access to the servers. So far, this works well. But if you’re a Samba, LDAP, or PAM expert and see a problem, please let me know in the comments.

I’m setting up a new FreeBSD web server. As 10.0 is just around the corner, I installed 10.0-BETA2. BETA4 is out, so it’s time to upgrade.

# freebsd-update -r 10.0-BETA4 upgrade
Looking up update.FreeBSD.org mirrors... 5 mirrors found.
...

That all looks good. Then I installed the update

# freebsd-update install
...

And the install hung. Overnight. Why?

Turns out that there are multiple errata on freebsd-update. While you can upgrade to BETA2-p2, you can’t jump directly to the next version.

So you need to do:

# freebsd-update fetch
# freebsd-update install
# reboot
# freebsd-update -r 10.0-BETA4 upgrade
# freebsd-update install

When freebsd-update appears to hang, use ^T to print the current status.

# freebsd-update install
Installing updates...
^T
load: 0.17 cmd: install 2796 [nanslp] 156.82r 0.01u 0.02s 0% 6220k

Here we see that the install process, PID 2796, is running. Wait a few minutes and try it again.

^T
load: 0.16 cmd: install 2803 [nanslp] 26.09r 0.01u 0.01s 0% 6160k

There’s been no visible output, and install is still running, but it’s now PID 2803. It’s a new install process. The update is proceeding, if not as quickly as you’d like.

So, be patient and let the machine do its work.

I have an old mail server running Postfix and courier-imap. We want to split our customers off onto their old server, preferably something with a pretty pointy-clicky interface so that they can manage their own accounts. (Yes, people do still buy email service these days.)

The old server runs FreeBSD, postfix, and courier-imap. The new server runs FreeBSD 9.2 with DirectAdmin. DirectAdmin is a canned package that puts a customer-friendly front end on a whole bunch of standard Unix software. (Why FreeBSD? ZFS. When my customers fill up the hard disk, I can just “zfs send” the whole machine to bigger disk.)

The good news is, both my old server and the new one use Maildir. The bad news is, they’re arranged slightly differently and have somewhat different file formats. Tools exist to do 80% of the migration for you, but my setup has a few edges.

I suspect that nobody else has my exact legacy setup, and my readers don’t care about this level of detail. I also suspect that the migration project will last weeks and I will completely forget how I did it in between individual customer migrations, so I’m writing this anyway. But maybe someone else can learn something from it, even if it’s just “man, I would never hire this Lucas character.”

Create your email domain and all the user accounts in DirectAdmin. I don’t know any way to migrate encrypted virtual user passwords from Postfix into DirectAdmin, so I’ll need to create new passwords for the users. That’s not uncommon for a packaged mail server migration, so my users will live with it. This is a tedious point-and-click operation, but point-and-click is the point of DirectAdmin, so give the job to a minion and get it done.

But how to transparently migrate the mailboxes from the Courier server to Dovecot?

On the old server, the mailboxes are in /disk2/mail/vhosts/domainname/account/. On DirectAdmin, the mailboxes are in /home/userid/imap/domain/account/Maildir/. I can tar up the old accounts and drag the whole directory over to the new server, but I need to massage the directories so that they’re arranged properly.

Courier and Dovecot both use files to store mailbox state information. IMAP state is important. Without it, users will re-download every single message. I don’t care about the bandwidth or the disk I/O this will cause, but the users will complain. Complaints lead to meetings, and meetings lead to the Dark Side. There’s a script to convert from one to the other. Dovecot also has good migration documentation, I recommend you read that before doing your own migration.

The file ownership needs correcting. DirectAdmin creates user accounts for each customer (more or less), and that account needs to own the email files for the various domains.

So, what am I willing to do manually?

I will manually tar up the domain’s mail directory, copy it to the destination server, and untar it in /home/userid/imap/domain/. And I’m willing to provide the username that should own the files. I’m sure both of these could be automated, but I don’t have enough accounts to migrate to do this correctly.

I’m not willing to move files around or run migration scripts. Because these are things that human beings mess up.

There’s ways to do this in shell scripts, but I’m using Perl. Because while my Perl makes small children cry, my shell scripts feature in rituals praising Nyarlathotep.

So, the script:

#!/usr/bin/perl
#migrate from old mail server to new
#run from domain directory
#Takes one argument, the username expected to own the files when done.

unless ($ARGV[0]) {
die "\nNeed a username to own files!\n\n";
}

opendir (USERS, ".") || die "Cannot open current directory";

foreach $user (readdir (USERS)) {

next if ($user =~ /^\.{1,2}$/);
print "Converting $user...\n";
mkdir ("$user/Maildir") || die "Cannot make Maildir";
opendir (USERDIR, "$user") || die "Cannot open user directory";
foreach $file (readdir (USERDIR)) {
next if ($file =~ /^\.{1,2}$|Maildir/);
print "Moving $user/$file\n";
rename ("$user/$file","$user/Maildir/$file")
|| die "Cannot move $file";
}
#next user
}
print "Now performing courier-dovecot migration...\n";
system ("courier-dovecot-migrate.pl --to-dovecot --recursive --convert");
print "Fixing ownership...\n";
system ("chown -R $ARGV[0]:mail .");

Is this pretty trivial? Yep. But the most error-prone part of any process is the part I do. The more of the migration the machine does, the fewer screw-ups in the migration.

Last night’s talk on OpenBSD is now live in the mug.org channel.

Part 1
Part 2

UPDATE: All in one.

Among other things, I compare OpenBSD to Richard Stallman and physically assault an audience member. (Brian was a very good spots, and learned an important lesson about volunteering, e.g.: don’t.) We also talk long long time, memory randomization, PF, BSD license versus GPL, Microsoft, and other OpenBSD stuff.

And, of course, the importance of the VAX.

I was researching next week’s OpenBSD talk and thought “You know, I ought to tell the story about VRRP, CARP, and Cisco. That’s a good one, and it illustrates how the OpenBSD community works and thinks.” It’s been ten years, so I decided to do some research to make sure I had my facts straight.

And I came across the Cisco Nexus 1000V manual. This big and mighty Cisco switch… supports CARP.

This is absolutely hilarious. I laughed so much my sides hurt.

Some of you younger folks are probably wondering what the big deal is. Well…

Back in the late 1990s, Cisco came up with the Virtual Router Redundancy Protocol (VRRP), using some of the lessons of their Hot Standby Router Protocol (HSRP). This was a quick-acting router failover protocol. If one router died, a second would notice and automatically take over for it. VRRP isn’t rocket surgery, it’s just that Cisco’s hardware could now support it and the market demanded it. Fair enough.

But then Cisco patented VRRP.

Cisco announced that anyone could implement VRRP, so long as they didn’t sue Cisco over it. Cisco wanted to offer something to the world, and didn’t want it to come back and bite them. Again, fair enough. Perfectly sensible from Cisco’s perspective.

The OpenBSD folks wanted router redundancy, too. And they wanted it in the base system. But Cisco’s licensing terms were a problem.

The modern BSD license boils down to:

1) Keep our copyright notice on this code
2) Don’t sue us if it breaks

There’s nothing in there about “And don’t sue Cisco if something breaks.” Specifically, the code can be used for any purpose, including suing Cisco. Mind you, you’d have a pretty hard time using OpenBSD code to sue Cisco, but the license doesn’t prohibit it.

So, while the VRRP patent terms were fine for Cisco, they weren’t acceptable under the BSD license.

And the OpenBSD devs wanted redundancy.

What to do? Go off and write your own protocol, the Common Address Redundancy Protocol (CARP). Make it different from VRRP. Field-test the protocol, using your legions of willing lackeys — er, devoted userbase. Make CARP not only a usable replacement for VRRP, but inherently better and stronger. Put the protocol under the BSD license, and give the protocol and code away.

This caused something of a kerfuffle at the time. Ugly accusations flew around. “It’s a VRRP knock-off!” “No, it’s a different protocol!” Great big reams of email were written about the whole thing.

The OpenBSD folks applied to IANA for a protocol number. IANA rejected the application, telling them to use VRRP instead. VRRP was assigned protocol 112. So OpenBSD used protocol 112 for CARP. And putting CARP hosts on a network with Cisco VRRP hosts made Cisco routers crash. The Cisco stack wasn’t robust enough to handle strange packets on the network. Cisco updated their hardware to survive seeing a lone CARP packet.

This escalated the kerfuffle into industry news. You’d see articles in all kinds of industry magazines about OpenBSD versus Cisco.

The OpenBSD folks responded by doing a CARP/VRRP-themed 3.5 release, complete with a Monty Python parody (lyrics, MP3).

And in the end of it all… everyone shut up. Other people started implementing CARP. Because it’s a solid, respectable redundancy protocol. You can get CARP from FreeBSD, Linux, Solaris, and a whole bunch of other vendors…

…including Ciso.

I had plans for today, but I’m too busy laughing. And then I need to go watch some Monty Python.

The fine folks at BSDNow.tv have requested my presence this Wednesday for an interview. We’ll talk sudo, FreeBSD, OpenBSD, and whatever else comes to mind.

You can watch it live, this Wednesday at 2PM EST.

PS: Hey, boss, I’ll be out Wednesday afternoon. Personal business.