Tech - Michael W Lucas

December 17, 2014January 30, 2015

Sudo talk now on YouTube

My talk Sudo: You’re Doing It Wrong is now live on YouTube. (Thanks to TJ for letting me know.) The talk is based on my book Sudo Mastery.

This talk went better than my NYCBSDCon talk. Probably because I hadn’t confused “buzzing with caffeine, adrenaline, and sleeplessness” with “raging tonsilitis.” The Q&A at the end took us wildly astray, and ended with the general conclusion that “Lucas needs to present to mug.org about how to use SSH correctly.”

I gave away a couple books, one Sudo Mastery and one SSH Mastery. The SSH book went to the first person to raise their hand and admit that they used passwords with SSH.

But I’m sure none of you use password-only authentication with SSH. You’re all good, decent, moral people who wouldn’t do anything that vile.

November 13, 2014

Sudo talk at mug.org, 9 Dec 2014

I’ll be talking at mug.org in Farmington Hills, MI, on 9 December 2014.

The topic is Sudo: You’re Doing It Wrong. If you use sudo, you need to show up for this. Because you’re doing it wrong. It’s based on Sudo Mastery, as you might guess.

Come to mug.org. They have cookies.

They usually record and show their talks, so if you can’t be bothered to go to Farmington Hills in December you can probably catch it on YouTube later. But it won’t be nearly as awesome.

Also, I’m planning to go to the IT in the D casual social event on 20 November 2014. I’m not speaking, just hanging out. Why?

Now that I’m a full time writer, this is my staff.

They’re perfectly sociable, and definitely cuter than most of my previous co-workers, but they’re a little short in the techie conversation department.

So, yeah. Two chances to see me in the near future. No public appearances planned afterwards. I’ll be busy trying to teach my staff how to copyedit.

November 5, 2014

FreeBSD “Working copy ‘/usr/src’ locked.”

Poul-Henning Kamp is working with me on some GBDE fixes. Which means he sends me patches and says “Here, try this,” along with very valuable exposition on how GBDE works and the threat model it applies to. This means I’m updating frequently.

My usual update process is:

# cd /usr/src # make update && make -j8 buildworld && make -j8 kernel && reboot

Half a second after typing this, I realized I’d forgotten to apply PHK’s latest patch. I hit CTRL-C during the make update before building an unsuitable userland. I need to do the source update, apply the patch, and then build everything.

# make update -------------------------------------------------------------- >>> Updating /usr/src using Subversion -------------------------------------------------------------- svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for det ails) svn: E155004: Working copy '/usr/src' locked. svn: E155004: '/usr/src' is already locked. *** Error code 1

Stop. make[1]: stopped in /usr/src *** Error code 1

Fine. svn cleanup it is. But wait–I don’t have svn installed!

As a matter of principle, I don’t want to install svn on this test box. svnlite is what FreeBSD offers to users, so it should be able to handle everything.

The good news is, svnlite also have a cleanup feature.

# svnlite cleanup # make update

And the update proceeds as I would hope.

Is this worth a PR to get the error message changed? Dunno. What do you think?

Now all I must do is master all the GBDE wisdom PHK dumped in my brain…

November 3, 2014

Next Project: “Networking for Sysadmins”

FreeBSD Mastery: Storage Essentials is out for tech review. (If you’re reading the pre-pub book, you’ve got a few more days to get comments back to me.) I’ll then make the corrections and send it to copyediting.

So I’m writing another book.

The current title is Networking for System Administrators. (I’d like to work the word “Mastery” in there, but it sounds artificially kludged together, because it would be.) It’s a small book, readable in a couple hours.

I’ve worked in a whole bunch of IT organizations as both a system administrator and a network administrator. In most of them I get sucked into a bridge role because I can speak to both teams in their own language.

It’s hard to teach a network administrator to be a sysadmin. An enterprise often runs a dozen or more different operating system, and who knows how many variants of each. Plus, each team might configure their differently. “You need a password to sudo here, you need a Yubikey to log on here, you need a hole in the head to log on here…” oy vey! Asking a network administrator to learn all this is like asking a sysadmin to configure Cisco, RouterOS, and OpenBSD routers. It just isn’t going to happen.

But the basic principles of networking isn’t hard, and understanding basic networking can save the sysadmin so much time. A sysadmin who wants to learn networking is often referred to books like The TCP/IP Guide or TCP/IP Illustrated. These are awesome books, and some systems administrators (and all non-web app developers) need to read them. For the majority of sysadmins, they’re overkill. An enterprise database administrator who needs to understand TCP/IP window scaling to do his job should call his network administrator.

Instead, most sysadmins learn networking via occasional blog posts, Google searches, and oral tradition. This is a ghastly way to learn any technical topic.

The result? Calls the sysadmin doesn’t want to make and the network administrator doesn’t want to get.

“Did that firewall port ever get opened?”

“Is my server plugged into the right network?”

“What do you mean that service is broken, I can ping it?”

“That service isn’t working, I can’t ping it.”

“That UDP port isn’t open, I can’t telnet to it!”

A knowledgeable sysadmin can quickly answer all of these questions for themselves without picking up the phone. And we wouldn’t be in IT if we wanted to talk on the phone.

The table of contents so far is:

Introduction

Network layers — the bottom 4 layers, and troubleshooting pointers to later chapters

Ethernet

IPv4

IPv6

TCP/IP (protocols, ports, etc)

Active traffic (netstat)

DNS

Checking the Network (sending vs receiving)

tcpdump (what we receive)

netcat (what we send)

packet filtering for sysadmins

tracing problems (traceroute & mtr)

This book also contains guidance on detecting an uneducated network administrator. “Filtering all ICMP, because ICMP is bad? Bzzzt!” I don’t put it in quite those terms, but… yeah. You at least need to know what you’re dealing with.

Unlike my earlier Mastery books, the incomplete draft of this book will not be available for pre-order. Sales of books that I offer for pre-order are much lower than books I don’t offer pre-order on. Part of this is the topic–DNSSEC has less popular interest than SSH. But the sudo book is doing much less well than I expected, excluding a spike from the Slashdot review. (Reviews on sites like Slashdot help sales more than anything I’ve found.)

From talking to other indie authors, it seems that an initial surge of sales strongly affects online bookstore’s algorithms. I say seems because most online bookstores do not make their algorithms public–they don’t want clever buggers like you telling me how to game their system.

The only way for me to tell is to test it, however. I won’t be doing preorders for this book and the next FreeBSD Mastery title.

I believe that many of my readers don’t need this book. I do hope that you’ll tell certain people you work with to read it, however. You know the ones I mean.

More updates as events warrant. Or you can check Twitter for the hashtag #n4sa. (I’m not the only one with that hashtag, but it seems pretty rarely used, so I’ll claim it.)

October 15, 2014

Revoked and Replaced OpenPGP Key

I uploaded a GPG key to subkeys.pgp.net back in 2005. It’s well past time for me to replace it. I covered creating your revocation certificate back in PGP & GPG, but didn’t actually write about using that revocation certificate. Nine years later… yeah, I better figure this out.

So Io to the machine with my keypair, and create my revocation certificate.

# gpg --output oldgpg.revoke.asc --gen-revoke E68C49BC

sec 1024D/E68C49BC 2005-02-21 Michael Warren Lucas Jr (Author, consultant, sysadmin)

Yep, that’s my old key.

Create a revocation certificate for this key? (y/N) y Please select the reason for the revocation: 0 = No reason specified 1 = Key has been compromised 2 = Key is superseded 3 = Key is no longer used Q = Cancel (Probably you want to select 1 here) Your decision? 2

Why is this key being revoked? Because it’s nine years old. I’ve generated a new key,

Enter an optional description; end it with an empty line: > Reason for revocation: Key is superseded (No description given) Is this okay? (y/N) y

Nobody cares about the details, so I don’t enter any.

You need a passphrase to unlock the secret key for user: "Michael Warren Lucas Jr (Author, consultant, sysadmin) " 1024-bit DSA key, ID E68C49BC, created 2005-02-21

I enter my passphrase.

ASCII armored output forced. Revocation certificate created.

I now have a revocation certificate, oldgpg.revoke.asc. To activate it, I import it into my keyring.

# gpg --import oldgpg.revoke.asc gpg: key E68C49BC: "Michael Warren Lucas Jr (Author, consultant, sysadmin) " revocation certificate imported gpg: Total number processed: 1 gpg: new key revocations: 1 gpg: 3 marginal(s) needed, 1 complete(s) needed, classic trust model gpg: depth: 0 valid: 2 signed: 14 trust: 0-, 0q, 0n, 0m, 0f, 2u gpg: depth: 1 valid: 14 signed: 1 trust: 14-, 0q, 0n, 0m, 0f, 0u gpg: next trustdb check due at 2020-10-13

No passphrase needed–it just happens.

Now: sleep tight, sweet prince.

# gpg --send-keys E68C49BC gpg: sending key E68C49BC to hkp server subkeys.pgp.net

My old key is dead.

For the record, my new key is 1F2E54A8, for mwlucas at michaelwlucas dot com.

Now if I could only kill 4EBA9723…

September 23, 2014

Shuffling Partitions on FreeBSD

I’ve recently moved my personal web sites to https://www.vultr.com/, using virtual machines instead of real hardware. (I’ve caught up to the 2000s, hurrah!) I didn’t track server utilization, so I provisioned the machines based on a vague gut feeling.

The web server started spewing signal 11s, occasionally taking down the site by killing mysql. Investigation showed that this server didn’t have enough memory. How can 1GB RAM not be enough for WordPress and MySQL? Why, back in my day–

<SLAP>

Right. Sorry about that.

Anyway, I needed to increase the amount of memory. This meant moving up to a larger hosting package, which also expanded my hard drive space. After running gpart recover to move the backup GPT table to the end of the new disk, my new disk was partitioned like so:

# gpart show

=>        34  1342177213  vtbd0  GPT  (640G)
          34         128      1  freebsd-boot  (64K)
         162           6         - free -  (3.0K)
         168   666988544      2  freebsd-ufs  (318G)
   666988712     4096000      3  freebsd-swap  (2.0G)
   671084712   671092535         - free -  (320G)

I have 320 GB of space space at the end of the disk.

The easy thing to do would be to create a new partition in that space. I advocate and recommend partitioning servers. The only reason that this system has one large partition is because that’s what the hosting provider gave me.

I’m writing a book on FreeBSD disk partitioning, however, so this struck me as an opportunity to try something that I need for the book. (As I write this, you can still get FreeBSD Mastery: Storage Essentials at a pre-pub discount.) How would I expand the root partition, with the swap space smack dab in the middle of the disk?

Virtualized systems have no awareness of the underlying disk. Advice like “put the swap partition at the beginning of the disk” becomes irrelevant, as you have no idea where that physically is. On a system like this, how would I use the built-in FreeBSD tools to create a swap partition at the end of the disk, and expand the existing partition to fill the remaining space?

This isn’t as easy as you might think. FreeBSD’s gpart command has no feature to add a partition at a specific offset. But it can be done.

Any time you touch disk format or partitions, you might lose filesystems. Back up your vital files. For a WordPress web server, this is my web directory and the SQL database. (My backup includes a half-complete version of this article. If my repartitioning goes badly, I’ll retitle this piece “How to Not Repartition.” But anyway…) Copy these files off the target server.

Now, what exactly do I want to do?

I want 4-5GB of swap space, at the end of the disk. (The server now has 2GB RAM.)

I want to remove the current swap space.

I want to expand the root partition to fill the remaining space.

gpart(8) won’t let me say “create a 4GB partition at the end of the disk.” It will let me create a filler partition that I have no intention of actually using, however. As I’m sure the disk is not precisely 320GB, I’m going to play it safe and give myself 5GB of room for this swap partition. I give this partition a label to remind me of its purpose and role.

# gpart add -t freebsd -s 315GB -l garbage vtbd0 vtbd0s4 added

The partitioning now looks like this.

=>        34  1342177213  vtbd0  GPT  (640G)
          34         128      1  freebsd-boot  (64K)
         162           6         - free -  (3.0K)
         168   666988544      2  freebsd-ufs  (318G)
   666988712     4096000      3  freebsd-swap  (2.0G)
   671084712   660602880      4  freebsd  (315G)
  1331687592    10489655         - free -  (5.0G)

Now I can add a swap partition at the end of the disk.

# gpart add -t freebsd-swap -l swap0 vtbd0 vtbd0p5 added

The resulting partitioning looks like this.

# gpart show

=>        34  1342177213  vtbd0  GPT  (640G)
          34         128      1  freebsd-boot  (64K)
         162           6         - free -  (3.0K)
         168   666988544      2  freebsd-ufs  (318G)
   666988712     4096000      3  freebsd-swap  (2.0G)
   671084712   660602880      4  freebsd  (315G)
  1331687592    10489655      5  freebsd-swap  (5.0G)

Tell /etc/fstab about the new swap partition, and remove the old one.

/dev/gpt/swap0 none swap sw 0 0

(In looking at the old entry, I realized that Vultr uses glabel(8) labels, where I use gpart(8) labels. Either type is fine, but I need to remember that /dev/label/swap0 is the old swap partition, and /dev/gpt/swap0 is the new one.)

Activate the new swap space. I could reboot, but why bother?

# swapon /dev/gpt/swap0

My swap now looks like this.

# swapinfo -h

Device          1K-blocks     Used    Avail Capacity
/dev/label/swap0   2047996       0B     2.0G     0%
/dev/gpt/swap0    5244824       0B     5.0G     0%
Total             7292820       0B     7.0G     0%

Turn off the old swap.

# swapoff /dev/label/swap0

The old swap is unused. I can put it out of my misery. Double-check gpart show to learn which partition is your swap space (3) and your temporary placeholder (4). Double double-check these numbers. We’re going to remove these partitions. If you delete your data partition due to your own stupidity you will be… unhappy.

# gpart delete -i 3 vtbd0 vtbd0p3 deleted # gpart delete -i 4 vtbd0 vtbd0s4 deleted

Triple double-check: do you still have a root filesystem? (Yes, FreeBSD has safeguards to prevent you from deleting mounted partitions. Check anyway.)

# gpart show

=>        34  1342177213  vtbd0  GPT  (640G)
          34         128      1  freebsd-boot  (64K)
         162           6         - free -  (3.0K)
         168   666988544      2  freebsd-ufs  (318G)
   666988712   664698880         - free -  (317G)
  1331687592    10489655      5  freebsd-swap  (5.0G)

Our swap space is at the end of the disk. And we have 317GB of free space right next to our root filesystem. You have not ruined your day. Yet.

Double-check your backups. Do you really have everything you need to recreate this server? If so, expand the root filesystem with gpart resize. Don’t specify a size, and the new partition will fill all available contiguous space.

# gpart resize -i 2 vtbd0 vtbd0p2 resized # gpart show

=>        34  1342177213  vtbd0  GPT  (640G)
          34         128      1  freebsd-boot  (64K)
         162           6         - free -  (3.0K)
         168  1331687424      2  freebsd-ufs  (635G)
  1331687592    10489655      5  freebsd-swap  (5.0G)

Now I have a 318GB filesystem on a 636GB partition. Let’s expand that filesystem to fill the partition. You can’t resize a filesystem label such as /dev/label/root0, you must use a partition identifier like vtbd0p2 or /dev/gpt/rootfs0. In FreeBSD 10, you can use growfs on mounted file systems.

# growfs /dev/vtbd0p2 It's strongly recommended to make a backup before growing the file system. OK to grow filesystem on /dev/vtbd0p2 from 318GB to 635GB? [Yes/No] yes growfs: /dev/vtbd0p2: Operation not permitted

Not permitted? I activated GEOM debugging mode by setting kern.geom.debugflags to 0x10, but was still denied. I’ve grown mounted filesystems before, so what the heck?

This virtual server has disabled soft updates, journaling, and all the fancy FreeBSD disk performance features. I suspect this error is tied to that. Let’s go to single user mode and grow the filesystem unmounted.

I reboot, and get:

Mounting from ufs:/dev/label/rootfs0 failed with error 19.

Even when you know what’s wrong, this message makes that little voice in the back of my skull simultaneously call me an idiot and scream “You destroyed your filesystem! Ha ha!” Plus, I can no longer make notes in my web browser–the article is on the non-running server.

Fortunately, I know which partition is the root partition. I enter

mountroot> ufs:/dev/vtbd0p2
and get the familiar single-user-mode prompt. Now I can do:

# growfs vtbd0p2

I answer yes, and new superblocks scroll across the screen. The filesystem grows to fill all available contiguous space.

My suspicion is that resizing the partition destroyed the label. Many GEOM classes store information in the last sector of the partition. Let’s use a GPT label instead.

# gpart modify -i 2 -l rootfs vtbd0

Mount the root filesystem read-write.

# mount -o rw /dev/vtbd0p2 /

Create a new /etc/fstab entry for the root filesystem, using the GPT label instead of the glabel(8) one.

/dev/gpt/rootfs / ufs rw,noatime 1 1

And then a reboot to see if everything comes back. It does.

My partitions now look like this:

# df -h

Filesystem         Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs    615G    6.1G    560G     1%    /
devfs              1.0K    1.0K      0B   100%    /dev
devfs              1.0K    1.0K      0B   100%    /var/named/dev

All installed disk space is now in use. Mission accomplished!

Having written this, though, I have no chance of forgetting that I need to go back and do a custom install to partition the server properly.

August 21, 2014August 22, 2014

Phabricator on FreeBSD installation notes

BSDs generally break their PHP packages into smaller units than most Linux distribution. This means that you need extra packages when following installation guides. I’m installing Phabricator on FreeBSD because I want ZFS under it.

This is the complete list of PHP modules and related stuff I needed to install to get Phabricator to run on FreeBSD 10.0p7/amd64. Don’t use PHP 5.5, as some modules are only available with PHP 5.4.

php5 mod_php5 php5-curl php5-gd php5-iconv php5-mbstring php5-mysql php5-pcntl pecl-APC php5-filter pear-Services_JSON php5-json php5-hash php5-openssl php5-ctype php5-posix php5-fileinfo

Restart your web server after installing everything.

Phabricator wants a lot of control over its database. I don’t like giving web applications root privileges on a database. this article by David Antaramian was quite helpful there.

Once you have your user set up, initialize the Phabricator database by running

# ./storage upgrade --user root --password MyRootPassword

This gives the script the access needed to actually create and adjust Phabricator databases.

After that, the Phabricator installer seems to do a good job of walking you through fixing the various setup niggles.

July 30, 2014July 31, 2014

a survey of FreeBSD ZFS snapshot automation tools

Why automatically snapshot filesystems? Because snapshots let you magically fall back to older versions of files and even the operating system. Taking a manual snapshot before a system upgrade is laudable, but you need to easily recover files when everything goes bad. So I surveyed my Twitter followers to see what FreeBSD ZFS snapshot automation tools they use.

The tools:

A few people use custom shell scripts of varying reliability and flexibility. I’m not going to write my own shell script. The people who write canned snapshot rotation tools have solved this problem, and I have no desire to re-solve it myself.

One popular choice was sysutils/zfs-snapshot-mgmt. This lets you create snapshots as often as once per minute, and retain them as long as you desire. Once a minute is a bit much for me. You can group snapshot creation and deletion pretty much arbitrarily, letting you keep, say, 867 per-minute snapshots, 22 every-seven-minute snapshots, and 13 monthlies, if that’s what you need. This is the Swiss army knife of ZFS snapshot tools. One possible complication with zfs-snapshot-mgmt is that it is written in Ruby and configured in YAML. If you haven’t seen YAML yet, you will–it’s an increasingly popular configuration syntax. My existing automation is all in shell and Perl, however. I added Python for Ansible. Adding yet another interpreter to all of my ZFS systems doesn’t thrill me. Ruby is not a show-stopper, but it doesn’t thrill me. The FreeBSD port is outdated, however–the web site referenced by the port says that the newest code, with bug fixes, is on github. If you’re looking for a FreeBSD porting project, this would be an easy one.

The zfs-periodic web page is down. NEC Energy Solutions owns the domain, so I’m guessing that the big corporate overlord claimed the blog and the site isn’t not coming back. The code still lives at various mirrors, however. zfs-periodic is tightly integrated with FreeBSD’s periodic system, and can automatically create and delete hourly, daily, monthly, and weekly snapshots. It appears to be the least flexible of the snapshot systems, as it runs with periodic. If you want to take your snapshots at a time that periodic doesn’t run, too bad. I don’t get a very good feeling from zfs-periodic–if the code had an owner, it would have a web site somewhere.

sysutils/zfsnap can do hourly, daily, weekly, and monthly snapshots. It’s designed to run from periodic(8) or cron(8), and is written in /bin/sh.

sysutils/zfstools includes a clone of OpenSolaris’ automatic snapshotting tools. I no longer run OpenSolaris-based systems, except on legacy servers that I’m slowly removing, but I never know what the future holds around the dayjob. (I’m waiting for the mission-critical Xenix deployment, I’m sure it’s not far off.) This looks highly flexible, being configured by a combination of cron scripts and ZFS attributes, and can snapshot every 15 minutes, hour, day, week, and month. It’s written in Ruby (yet another scripting language on my system? Oh, joy. Joy and rapture.) On the plus side, the author of zfstools is also a FreeBSD committer, so I can expect him to keep the port up to date.

In doing this survey I also came across sysutils/freebsd-snapshot, a tool for automatically scheduling and automounting UFS snapshots. While I’m not interested in UFS snapshots right now, this is certainly worth remembering.

My choice?

So, which ones will I try? I want a tool that’s still supported and has some flexibility. I want a FreeBSD-provided package of the current version of the software. I’m biased against adding another scripting language to my systems, but that’s not veto-worthy.

If I want compatibility with OpenSolaris, I’ll use zfstools. I get another scripting language, yay!

If I don’t care about OpenSolaris-derived systems, zfsnap is the apparent winner.

Of course, I won’t know which is better until I try both… which will be the topic of a couple more blogs.

UPDATE, 07-31-2014: I screwed up my research on zfsnap. I have rewritten that part of the article, and my conclusions. My apologies — that’s what happens when you try to do research after four hours sleep. Thanks to Erwin Lansing for pointing it out.

(“Gee, I’m exhausted. Better not touch any systems today. What shall I do? I know, research and a blog post!” Sheesh.)

July 16, 2014

Installing and Using Tarsnap for Fun and Profit

Well, “profit” is a strong word. Maybe “not losing money” would be a better description. Perhaps even “not screwing over readers.”

I back up my personal stuff with a combination of snapshots, tarballs, rsync, and sneakernet. This is fine for my email and my personal web site. Chances are, if all four of my backup sites are simultaneously destroyed, I won’t care.

A couple years ago I opened my own ecommerce site, so I could sell my self-published books directly to readers. For the record, I didn’t expect Tilted Windmill Press direct sales to actually, y’know, go anywhere. I didn’t expect that people would buy books directly from me when they could just go to their favorite ebookstore, hit a button, and have the book miraculously appear on their ereader.

I was wrong. People buy books from me. Once every month or two, someone even throws a few bucks in the tip jar, or flat-out overpays for their books. I am pleasantly surprised.

So: I was wrong about self-publishing, and now I was wrong about author direct sales. Pessimism is grand, because you’re either correct or you get a pleasant surprise. Thank you all.

But now I find myself in a position where I actually have commercially valuable data, and I need to back it up. Like a real business. I need offsite backups. I need them automated. And I need to be able to recover them, so that people who have bought my books can continue to download them, in the off chance that the Detroit area is firebombed off the Earth while I’m at BSDCan.

So it’s time for Tarsnap.

Why Tarsnap?

It works very much like tar, so I don’t have to learn any new command-line arguments. (If you’re not familiar with tar, you need to be.)

The terms of service are readable by human beings and more reasonable than other backup services.

The code is open and auditable.

When Tarsnap’s author screws up, he admits it and handles it correctly.

It’s cheap. Any backup priced in picodollars gets my attention.

I also see the author regularly at regular BSD conferences. I can slap him in person if he does anything truly daft.

Tarsnap has a quick Getting Started page. We’ll do the easy things first. Sign up for a Tarsnap account. Once your account is active, put some money in it–$5 will suffice.

Now let’s check your prerequisites. You need:

GnuPG

BSD systems come with everything else you need.

Linux users must install

a compiler, like gcc or clang

make

OpenSSL (including header files)

zlib (including header files)

System header files

OpenSSL header files

The ext2fs/ext2_fs.h (not the linux/ext2_fs.h header)

The Tarsnap download page lists specific packages for Debian-based and Red Hat-based Linuxes.

Go to the download page and get both the source code and the signed hash file. Tarsnap is only available as source code, so that you can verify the code integrity yourself. So let’s do that.

Start by using GnuPG to verify the integrity of the Tarsnap code. If you’re not familiar with GnuPG and OpenPGP, some daftie wrote a whole book on PGP & GPG. Once you install GnuPG, run the gpg command to get the configuration files.

# gpg gpg: directory `/home/mwlucas/.gnupg' created gpg: new configuration file `/home/mwlucas/.gnupg/gpg.conf' created gpg: WARNING: options in `/home/mwlucas/.gnupg/gpg.conf' are not yet active during this run gpg: keyring `/home/mwlucas/.gnupg/secring.gpg' created gpg: keyring `/home/mwlucas/.gnupg/pubring.gpg' created gpg: Go ahead and type your message ... ^C gpg: signal Interrupt caught ... exiting

Hit ^C. I just wanted the configuration and key files.

Now edit $HOME/.gnupg/gpg.conf. Set the following options.

keyserver hkp://keys.gnupg.net keyserver-options auto-key-retrieve

See if our GPG client can verify the signature file tarsnap-sigs-1.0.35.asc.

# gpg --decrypt tarsnap-sigs-1.0.35.asc SHA256 (tarsnap-autoconf-1.0.35.tgz) = 6c9f6756bc43bc225b842f7e3a0ec7204e0cf606e10559d27704e1cc33098c9a gpg: Signature made Sun Feb 16 23:20:35 2014 EST using RSA key ID E5979DF7 gpg: Good signature from "Tarsnap source code signing key (Colin Percival) " [unknown] gpg: WARNING: This key is not certified with a trusted signature! gpg: There is no indication that the signature belongs to the owner. Primary key fingerprint: 634B 377B 46EB 990B 58FF EB5A C8BF 43BA E597 9DF7

Some interesting things here. The most important line here is the statement ‘Good signature from “Tarsnap source code signing key”.’ Your GPG program grabbed the source code signing key from a public key server and used it to verify that the signature file is not tampered with.

As you’re new to OpenPGP, this is all you can do. You’re not attached to the Web of Trust, so you can’t verify the signature chain. (I do recommend that you get an OpenPGP key and collect a few signatures, so you can verify code signatures if nothing else.)

Now that we know the signature file is good, we can use the cryptographic hash in the file to validate that the tarsnap code we downloaded is what the Tarsnap author intended. Near the top of the signature file you’ll see the line:

SHA256 (tarsnap-autoconf-1.0.35.tgz) = 6c9f6756bc43bc225b842f7e3a0ec7204e0cf606e10559d27704e1cc33098c9a

Use the sha256(1) program (or sha256sum, or shasum -a 256, or whatever your particular Unix calls the SHA-256 checksum generator) to verify the source code’s integrity.

# sha256 tarsnap-autoconf-1.0.35.tgz SHA256 (tarsnap-autoconf-1.0.35.tgz) = 6c9f6756bc43bc225b842f7e3a0ec7204e0cf606e10559d27704e1cc33098c9a

The checksum in the signature file and the checksum you compute match. You have valid source code, and can proceed.

Extract the source code.

# tar -xf tarsnap-autoconf-1.0.35.tgz # cd tarsnap-autoconf-1.0.35 # ./configure ... configure: creating ./config.status config.status: creating Makefile config.status: creating config.h config.status: executing depfiles commands #

If the configure script ends any way other than this, you’re on Linux and didn’t install the necessary development packages. The libraries alone won’t suffice, you must have the development versions.

If configure completed, run

# make all install clean

Tarsnap is now ready to use.

Start by creating a Tarsnap key for this machine and attaching it to your Tarsnap account. Here I create a key for my machine www.

# tarsnap-keygen –keyfile /root/tarsnap.key –user mwlucas@michaelwlucas.com –machine pestilence
Enter tarsnap account password:
#

I now have a tarsnap key file. /root/tarsnap.key looks like this:

# START OF TARSNAP KEY FILE dGFyc25hcAAAAAAAAAAzY6MEAAAAAAEAALG8Ix2yYMu+TN6Pj7td2EhjYlGCGrRRknJQ8AeY uJsctXIEfurQCOQN5eZFLi8HSCCLGHCMRpM40E6Jc6rJExcPLYkVQAJmd6auGKMWTb5j9gOr SeCCEsUj3GzcTaDCLsg/O4dYjl6vb/he9bOkX6NbPomygOpBHqcMOUIBm2eyuOvJ1d9R+oVv ...

This machine is now registered and ready to go.

This key is important. If your machine is destroyed and you need access to your remote backup, you will need this key! Before you proceed, back it up somewhere other than the machine you’re backing up. There’s lots of advice out there on how to back up private keys. Follow it.

Now let’s store some backups in the cloud. I’m going to play with my /etc/ directory, because it’s less than 3MB. Start by backing up a single directory.

# tarsnap -c -f wwwetctest etc/ Directory /usr/local/tarsnap-cache created for "--cachedir /usr/local/tarsnap-cache" Total size Compressed size All archives 1996713 382896 (unique data) 1946025 366495 This archive 1996713 382896 New data 1946025 366495

Nothing seems to happen on the local system. Let’s check and be sure that there’s a backup out in the cloud:

# tarsnap --list-archives wwwetctest

I then went into /etc and did some cleanup, removing files that shouldn’t have ever been there. This stuff grows in /etc on any long-lived system.

# tarsnap -c -f wwwetctest-20140716-1508 etc/ Total size Compressed size All archives 3986206 765446 (unique data) 2120798 403833 This archive 1989493 382550 New data 174773 37338

# tarsnap --list-archives wwwetctest wwwetctest-20140716-1508

Note that the compressed size of this archive is much smaller than the first one. Tarsnap only stored the diffs between the two backups.

If you want more detail about your listed backups, add -v to see the creation date. Add a second -v to see the command used to create the archive.

# tarsnap --list-archives -vv wwwetctest 2014-07-16 15:02:41 tarsnap -c -f wwwetctest etc/ wwwetctest-20140716-1508 2014-07-16 15:09:38 tarsnap -c -f wwwetctest-20140716-1508 etc/

Let’s pretend that I need a copy of my backup. Here I extract the newest backup into /tmp/etc.

# cd /tmp # tarsnap -x -f wwwetctest-20140716-1508

Just for my own amusement, I’ll extract the older backup as well and compare the contents.

# cd /tmp # tarsnap -x -f wwwetctest

The files I removed during my cleanup are now present.

What about rotating backups? I now have two backups. The second one is a differential backup against the first. If I blow away the first backup, what happens to the older backup?

# tarsnap -d -f wwwetctest Total size Compressed size All archives 1989493 382550 (unique data) 1938805 366149 This archive 1996713 382896 Deleted data 181993 37684

It doesn’t look like it deleted very much data. And indeed, a check of archive shows that all my files are there.

And now, the hard part: what do I need to back up? That’s a whole separate class of problem…

May 17, 2014May 17, 2014

LibreSSL at BSDCan

Thanks to various airline problems, we had an open spot on the BSDCan schedule. Bob Beck filled in at the last moment with a talk on the first thirty days of LibreSSL. Here are some rough notes on Bob’s talk (slides now available).

LibreSSL forked from OpenSSL 1.0.1g.

Why did “we” let OpenSSL happen? Nobody looked. Or nobody admitted that they looked. We all did it. The code was too horrible to look at. This isn’t just an OpenSSL thing, or just an open source thing. It’s not unique in software development, it’s just the high profile one of the moment.

Heartbleed was not the final straw that caused the LibreSSL fork. The OpenSSL malloc replacement layer was the final straw. Default OpenSSL never frees memory, so tools can’t spot bugs. It uses LIFO recycling, so you can use after free. The debugging malloc sends all memory information to a log. Lots more in Bob’s slides, but this all combined into an exploit mitigation technique countermeasure. Valgrind, Coverity, and OpenBSD’s randomized memory tools don’t catch this.

Someone discovered all this this four years ago and opened an OpenSSL bug. It’s still sitting there.

LibreSSL started by ripping out features. VMS support, 16-bit Windows support, all gone.

LibreSSL goals:

Preserve API/ABI compatibility – become a drop-in replacement.

Bring more people into working on the codebase, by making the code less horrible

Fix bugs and modern coding processes

Do portability right

As an example, how does OpenSSH (not LibreSSL, but another OpenBSD product) do portable?

Assume a sane target OS, and code to that standard.

Build and maintain code on the above, using modern C

Provide portability shims to correctly do things that other OS’s don’t provide, only for those who need it.
– No ifdef maze
– No compromise on what the intrinsic functions actually do
– Standard intrinsics
– Don’t reimplement libc

How does OpenSSL do portable?

Assume the OS provides nothing, because you mustn’t break support for Visual C 1.52.

Spaghetti mess of #ifdef #ifndef horror nested 17 deep

Written in OpenSSL C – essentially it’s own dialect – to program to the worst common denominator

Implement own layers and force all platforms to use it

The result? “Chthulhu sits in his house in #define OPENSSL_VMS and dreams”

Removed scads of debugging malloc and other nasties.

What upstream packages call and use them? No way to tell. LibreSSL makes some of the very dangerous options no-ops. Turn on memory debugging? Replace malloc wrappers at runtime? These do nothing. The library internally does not use them.

Some necessary changes that were implemented in massive sweeps:

malloc+memset -> calloc

malloc (X*Y) -> reallocarray(X, Y)

realloc and free handle NULL, so stop testing everywhere

OpenSSL used EGD for entropy, and faked random data. OpenSSL gathered entropy from the following sources:

Your RSA private key is pretty random

“string to give random number generator entropy”

getpid()

gettimeofday()

In LibreSSL, entropy is the responsibility of the OS. If your OS cannot provide you with entropy, LibreSSL will not fake it.

LibreSSL is being reformatted into KNF – the OpenBSD code style. OpenSSL uses whatever style seemed right at the moment. The reformatting makes other problems visible, which is the point. More readable code hopefully means more developer involvement.

The OpenSSL bug tracking RT has been and continues to be a valuable resource.

OpenSSL exposes just about everything via public header files. Lots of the API should probably not be used outside of the library, but who knows who calls what? OpenBSD is finding out through constant integration testing with their ports tree.

The LibreSSL team wants to put the API on a diet so that they can remove potentially dangerous stuff. Their guys are being careful in this by testing against the OpenBSD ports tree. Yes, this conflicts with the “drop-in replacement” goal.

Internally, LibreSSL uses only regular intrinsic functions provided by libc. OpenSSL’s custom APIs remain for now only to maintain compatibility with external apps.

Surprises LibreSSL guys in OpenSSL:

big endian amd64 support

Compile options NO_OLD_ASN1 and option NO_ASN1_OLD are not the same

You can turn off sockets, but you can’t turn off debugging malloc

socklen_t – if your OS doesn’t have socklen_t, it’s either int or size_t. But OpenSSL does horrible contortions to define its own. If the size of socklen_t changes while your program is running, openssl will cope.

OpenSSL also copes if /dev/null moves while openssl is running.

So far:

OpenSSL 1.0.1g was a 388,000 line code base

As of yesterday, 90,000 lines of C source deleted, about 150,000 lines of files

Approximately 500,000 line unidiff from 1.0.1g at this point

Many bugs fixed

The cleaning continues, but they’ve started adding new features (ciphers)

Code has become more readable – portions remain scary

LibreSSL has added the following cipher suites under acceptable licenses – Brainpool, ChaCha, poly1305, ANSSI FRP256v1, and several new ciphers based on the above.

FIPS mode is gone. It is very intrusive. In other places governments mandate use of certain ciphers (Cameilla, GOST, etc). As long as they’re not on by default, and are provided as clean implementations under an acceptable license they will include them. They believe it’s better people who must use these use them in a sane library with a sane API than rolling their own.

If you want to use the forthcoming portable LibreSSL, you need:

modern POSIX environment

OS must provide random data – readiness and quality are responsibility of OS

malloc/free/calloc/realloc (overflow checking)

modern C string capabilities (strlcat, strlcpy, asprintf, etc)

explicit_bzero, reallocarry, arc4random

You can’t replace explicit_bzero with bzero, or arc4random with random. LibreSSL wants a portability team that understands how to make it work correctly.

LibreSSL’s eventual goals:

provide better (replacement, reduced) api

reduce code base even more

split out non-crypto things from libcrypto

split libcrypto from libssl

There’s lots of challenges to this. The biggest is stable funding.

The OpenBSD Foundation wants to fund several developers to rewrite key pieces of code. They want to sponsor efforts of the portability team, and the ports people track the impact of proposed API changes.

They will not do this at the expense of OpenSSH or OpenBSD.

The OpenBSD Foundation has asked the Linux Foundation for support, but the Linux Foundation has not yet committed to supporting the effort. (I previously said that they hadn’t responded to the request, which is different. The LF has received Bob’s email and discussions are ongoing.)

In Summary:

OpenSSL’s code is awful

LibreSSL can be done

They need support

If you’re interested in supporting the effort, contact the OpenBSD Foundation. The Foundation is run by Bob Beck and Ken Westerback, and they manage all funding. (While Theo de Raadt leads the OpenBSD Project, he actually has nothing to do with allocating funding.)