I’ve integrated the last of the tech edits into the FreeBSD Mastery: Storage Essentials manuscript and sent it to tech editing.
With any luck, the final book will be available on ebookstores at the end of the month, with print early December.
FreeBSD “Working copy ‘/usr/src’ locked.”
Poul-Henning Kamp is working with me on some GBDE fixes. Which means he sends me patches and says “Here, try this,” along with very valuable exposition on how GBDE works and the threat model it applies to. This means I’m updating frequently.
My usual update process is:
# cd /usr/src
# make update && make -j8 buildworld && make -j8 kernel && reboot
Half a second after typing this, I realized I’d forgotten to apply PHK’s latest patch. I hit CTRL-C during the make update before building an unsuitable userland. I need to do the source update, apply the patch, and then build everything.
# make update
--------------------------------------------------------------
>>> Updating /usr/src using Subversion
--------------------------------------------------------------
svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for det
ails)
svn: E155004: Working copy '/usr/src' locked.
svn: E155004: '/usr/src' is already locked.
*** Error code 1
Stop.
make[1]: stopped in /usr/src
*** Error code 1
Fine. svn cleanup it is. But wait–I don’t have svn installed!
As a matter of principle, I don’t want to install svn on this test box. svnlite is what FreeBSD offers to users, so it should be able to handle everything.
The good news is, svnlite also have a cleanup feature.
# svnlite cleanup
# make update
And the update proceeds as I would hope.
Is this worth a PR to get the error message changed? Dunno. What do you think?
Now all I must do is master all the GBDE wisdom PHK dumped in my brain…
New novel out: “Immortal Clay”
Most of you follow me for nonfiction, but this book just came out, so I’m gonna tell you about it anyway.
My SF novel, Immortal Clay, just hit Kobo and Amazon. It’s in process at other outlets like Barnes & Noble, iTunes, and so on, and print is forthcoming.
I’d describe this book as an alien invasion tale like Invasion of the Body Snatchers or The Thing, set after we lose.
You can read the opening on my web site. Or you can jump straight to Kobo or Amazon and buy it sight unseen. I’m good either way.
And Ben Baldwin did an absolutely amazing cover. I mean… wow! Click on the preview and take a look. If you have a big screen, look at the high-res version.
Next Project: “Networking for Sysadmins”
FreeBSD Mastery: Storage Essentials is out for tech review. (If you’re reading the pre-pub book, you’ve got a few more days to get comments back to me.) I’ll then make the corrections and send it to copyediting.
So I’m writing another book.
The current title is Networking for System Administrators. (I’d like to work the word “Mastery” in there, but it sounds artificially kludged together, because it would be.) It’s a small book, readable in a couple hours.
I’ve worked in a whole bunch of IT organizations as both a system administrator and a network administrator. In most of them I get sucked into a bridge role because I can speak to both teams in their own language.
It’s hard to teach a network administrator to be a sysadmin. An enterprise often runs a dozen or more different operating system, and who knows how many variants of each. Plus, each team might configure their differently. “You need a password to sudo here, you need a Yubikey to log on here, you need a hole in the head to log on here…” oy vey! Asking a network administrator to learn all this is like asking a sysadmin to configure Cisco, RouterOS, and OpenBSD routers. It just isn’t going to happen.
But the basic principles of networking isn’t hard, and understanding basic networking can save the sysadmin so much time. A sysadmin who wants to learn networking is often referred to books like The TCP/IP Guide or TCP/IP Illustrated. These are awesome books, and some systems administrators (and all non-web app developers) need to read them. For the majority of sysadmins, they’re overkill. An enterprise database administrator who needs to understand TCP/IP window scaling to do his job should call his network administrator.
Instead, most sysadmins learn networking via occasional blog posts, Google searches, and oral tradition. This is a ghastly way to learn any technical topic.
The result? Calls the sysadmin doesn’t want to make and the network administrator doesn’t want to get.
A knowledgeable sysadmin can quickly answer all of these questions for themselves without picking up the phone. And we wouldn’t be in IT if we wanted to talk on the phone.
The table of contents so far is:
This book also contains guidance on detecting an uneducated network administrator. “Filtering all ICMP, because ICMP is bad? Bzzzt!” I don’t put it in quite those terms, but… yeah. You at least need to know what you’re dealing with.
Unlike my earlier Mastery books, the incomplete draft of this book will not be available for pre-order. Sales of books that I offer for pre-order are much lower than books I don’t offer pre-order on. Part of this is the topic–DNSSEC has less popular interest than SSH. But the sudo book is doing much less well than I expected, excluding a spike from the Slashdot review. (Reviews on sites like Slashdot help sales more than anything I’ve found.)
From talking to other indie authors, it seems that an initial surge of sales strongly affects online bookstore’s algorithms. I say seems because most online bookstores do not make their algorithms public–they don’t want clever buggers like you telling me how to game their system.
The only way for me to tell is to test it, however. I won’t be doing preorders for this book and the next FreeBSD Mastery title.
I believe that many of my readers don’t need this book. I do hope that you’ll tell certain people you work with to read it, however. You know the ones I mean.
More updates as events warrant. Or you can check Twitter for the hashtag #n4sa. (I’m not the only one with that hashtag, but it seems pretty rarely used, so I’ll claim it.)
FreeBSD Mastery: Storage Essentials cover
Here’s an early version of the cover for FreeBSD Mastery: Storage Essentials.
The text is subject to change, but the art (from the inimitable Eddie Sharam) is final.
You might also be interested in the painting Liberty Leading the People.
Revoked and Replaced OpenPGP Key
I uploaded a GPG key to subkeys.pgp.net back in 2005. It’s well past time for me to replace it. I covered creating your revocation certificate back in PGP & GPG, but didn’t actually write about using that revocation certificate. Nine years later… yeah, I better figure this out.
So Io to the machine with my keypair, and create my revocation certificate.
# gpg --output oldgpg.revoke.asc --gen-revoke E68C49BC
sec 1024D/E68C49BC 2005-02-21 Michael Warren Lucas Jr (Author, consultant, sysadmin)
Yep, that’s my old key.
Create a revocation certificate for this key? (y/N) y
Please select the reason for the revocation:
0 = No reason specified
1 = Key has been compromised
2 = Key is superseded
3 = Key is no longer used
Q = Cancel
(Probably you want to select 1 here)
Your decision? 2
Why is this key being revoked? Because it’s nine years old. I’ve generated a new key,
Enter an optional description; end it with an empty line:
>
Reason for revocation: Key is superseded
(No description given)
Is this okay? (y/N) y
Nobody cares about the details, so I don’t enter any.
You need a passphrase to unlock the secret key for
user: "Michael Warren Lucas Jr (Author, consultant, sysadmin)
1024-bit DSA key, ID E68C49BC, created 2005-02-21
I enter my passphrase.
ASCII armored output forced.
Revocation certificate created.
I now have a revocation certificate, oldgpg.revoke.asc. To activate it, I import it into my keyring.
# gpg --import oldgpg.revoke.asc
gpg: key E68C49BC: "Michael Warren Lucas Jr (Author, consultant, sysadmin)
gpg: Total number processed: 1
gpg: new key revocations: 1
gpg: 3 marginal(s) needed, 1 complete(s) needed, classic trust model
gpg: depth: 0 valid: 2 signed: 14 trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: depth: 1 valid: 14 signed: 1 trust: 14-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2020-10-13
No passphrase needed–it just happens.
Now: sleep tight, sweet prince.
# gpg --send-keys E68C49BC
gpg: sending key E68C49BC to hkp server subkeys.pgp.net
My old key is dead.
For the record, my new key is 1F2E54A8, for mwlucas at michaelwlucas dot com.
Now if I could only kill 4EBA9723…
“Storage Essentials” first draft complete!
The first draft of FreeBSD Mastery: Storage Essentials is now complete and available on the Tilted Windmill Press site.
My target for a Mastery book is for it to be about 30K words. That seems a fair length for a $10 technology ebook. FMSE is 45K words, or about 50% more than that. At that price point, it’ll be a bargain. The print version will probably run more than the $20 I prefer, but we’ll see what happens.
As it’s a complete draft, the price has been raised to $8.99. Once the book goes through technical review and I correct it, the price will go to its final $9.99. At that point, I’ll get it into Amazon, B&N, and so on, in both print and epub.
So, what’s next?
Next, I look at my pile of outlines and try to untangle them. I’m planning FreeBSD books on jails, on ZFS, specialty filesystems (which might or might not include network filesystems), and more. These topics are all terribly interrelated. As I’m now writing full time, I need to figure out the approach that makes the best use of my time and yet gives me the maximum amount of exposure to everything.
I still intend to do a small OpenBSD book in the near future, but I’m still debating what that should be. I have high hopes for both OpenHTTPD and LibreSSL, but I want both projects to settle first. And I have a whole list of non-BSD books on my list as well.
There’s also the possibility that the market will reject FMSE. If that happens, it will limit how many more FreeBSD Mastery books I do. I think that won’t happen–I expect the book to do well–but I never know. As I’m depending on books to pay my mortgage, I might have to make the hard decision to cancel the series. We’ll have to wait and see.
I’m changing careers
My employer was just bought by another company. I find myself unemployed. This was not unexpected, so I’ve had time to think about what to do next.
I could have another IT job by three PM by picking up the phone and calling my friend Pam. If Pam was out of town, I’d call half a dozen other people and have a job by noon tomorrow. I’d certainly get a raise over what I’m making now–actually, given that I specifically chose a lower-paying job with less stress, I could double my salary. That’s what monster.com tells me.
But I don’t think I want to do that.
Instead, after talking with my family and taking a hard look at our finances, I’ve decided to write full time.
This is a big pay cut for me. Yes, even from my low-stress low-pay job. It means not going out to eat, hoping the car doesn’t drop a transmission, and mowing my own lawn instead of having Chuck the lawn guy do it for me. But it’s work I’ll far enjoy more than being paged at stupid o’clock because some beancounter decided we didn’t need to replace that faulty power supply. I’ll enjoy it a heck of a lot more than attending yet another pointless staff meeting.
Based on my previous book sales, it appears that I can get my income up to what my recently-departed job paid in about 12-18 months of hard writing. It’ll be a spartan year, but that’s okay.
Any number of things could derail this plan. I might wind up working at a big company in six months, regretting ever calling anyone a beancounter.
Writing for a living means I must figure out how quickly I actually write and coming up with a real production schedule. My books have all been written in one-hour stretches, in a variety of inconvenient locations. I have no idea what my sustained output looks like, especially once I no longer have any threat of a phone call waking me up in the middle of the night. (It doesn’t matter how “low-stress” a job is, a faulty email server that taken ten days to get properly fixed means two weeks without writing.)
Writing for a living means I need to write towards money like a hungry rat gnawing through the brick wall of the butcher shop. My family is supportive, but we do like to go out to eat now and then at a fancy place, like Qdoba. I’m going to try a bunch of different projects and see which take off. I have high hopes for the forthcoming FreeBSD Mastery books, and I have a list of thirty other titles to work on.
Writing for a living means I need to be a lot more consistent about, say, mentioning that I have a tip jar at the bottom of technical blog posts. I must overcome my shame at saying “Hey, if I helped you, give me money.”
This means that if you’re one of the organizations that owe me conference reimbursements, I’m gonna knock on your door with my hand out. I have lots of time to do that now.
This means when my clothes wear out I shop at Salvation Army rather than Costco. That’s okay. Old Sal is more fun, even though you probably don’t want to eat any of the free samples.
Speaking of Costco? Yeah… try the farmer’s market in downtown Detroit instead. What’s in season is cheap.
On the other hand, I only have one full-time job now instead of two. I’ll have free time. I’m looking forward to it. I’ve studied the craft of writing for decades now, and given up a lot of things for it. Why, I hear they rebooted Star Trek a few years ago. I grew up watching that show, and I’d really enjoy catching up with it. I can’t see how they’ll have a bald French guy as captain of the Enterprise, but what the heck, I’ll give it a try.
But am really going to miss the lawn guy.
Shuffling Partitions on FreeBSD
I’ve recently moved my personal web sites to https://www.vultr.com/, using virtual machines instead of real hardware. (I’ve caught up to the 2000s, hurrah!) I didn’t track server utilization, so I provisioned the machines based on a vague gut feeling.
The web server started spewing signal 11s, occasionally taking down the site by killing mysql. Investigation showed that this server didn’t have enough memory. How can 1GB RAM not be enough for WordPress and MySQL? Why, back in my day–
<SLAP>
Right. Sorry about that.
Anyway, I needed to increase the amount of memory. This meant moving up to a larger hosting package, which also expanded my hard drive space. After running gpart recover to move the backup GPT table to the end of the new disk, my new disk was partitioned like so:
# gpart show
=> 34 1342177213 vtbd0 GPT (640G)
34 128 1 freebsd-boot (64K)
162 6 - free - (3.0K)
168 666988544 2 freebsd-ufs (318G)
666988712 4096000 3 freebsd-swap (2.0G)
671084712 671092535 - free - (320G)
I have 320 GB of space space at the end of the disk.
The easy thing to do would be to create a new partition in that space. I advocate and recommend partitioning servers. The only reason that this system has one large partition is because that’s what the hosting provider gave me.
I’m writing a book on FreeBSD disk partitioning, however, so this struck me as an opportunity to try something that I need for the book. (As I write this, you can still get FreeBSD Mastery: Storage Essentials at a pre-pub discount.) How would I expand the root partition, with the swap space smack dab in the middle of the disk?
Virtualized systems have no awareness of the underlying disk. Advice like “put the swap partition at the beginning of the disk” becomes irrelevant, as you have no idea where that physically is. On a system like this, how would I use the built-in FreeBSD tools to create a swap partition at the end of the disk, and expand the existing partition to fill the remaining space?
This isn’t as easy as you might think. FreeBSD’s gpart command has no feature to add a partition at a specific offset. But it can be done.
Any time you touch disk format or partitions, you might lose filesystems. Back up your vital files. For a WordPress web server, this is my web directory and the SQL database. (My backup includes a half-complete version of this article. If my repartitioning goes badly, I’ll retitle this piece “How to Not Repartition.” But anyway…) Copy these files off the target server.
Now, what exactly do I want to do?
gpart(8) won’t let me say “create a 4GB partition at the end of the disk.” It will let me create a filler partition that I have no intention of actually using, however. As I’m sure the disk is not precisely 320GB, I’m going to play it safe and give myself 5GB of room for this swap partition. I give this partition a label to remind me of its purpose and role.
# gpart add -t freebsd -s 315GB -l garbage vtbd0
vtbd0s4 added
The partitioning now looks like this.
=> 34 1342177213 vtbd0 GPT (640G)
34 128 1 freebsd-boot (64K)
162 6 - free - (3.0K)
168 666988544 2 freebsd-ufs (318G)
666988712 4096000 3 freebsd-swap (2.0G)
671084712 660602880 4 freebsd (315G)
1331687592 10489655 - free - (5.0G)
Now I can add a swap partition at the end of the disk.
# gpart add -t freebsd-swap -l swap0 vtbd0
vtbd0p5 added
The resulting partitioning looks like this.
# gpart show
=> 34 1342177213 vtbd0 GPT (640G)
34 128 1 freebsd-boot (64K)
162 6 - free - (3.0K)
168 666988544 2 freebsd-ufs (318G)
666988712 4096000 3 freebsd-swap (2.0G)
671084712 660602880 4 freebsd (315G)
1331687592 10489655 5 freebsd-swap (5.0G)
Tell /etc/fstab about the new swap partition, and remove the old one.
/dev/gpt/swap0 none swap sw 0 0
(In looking at the old entry, I realized that Vultr uses glabel(8) labels, where I use gpart(8) labels. Either type is fine, but I need to remember that /dev/label/swap0 is the old swap partition, and /dev/gpt/swap0 is the new one.)
Activate the new swap space. I could reboot, but why bother?
# swapon /dev/gpt/swap0
My swap now looks like this.
# swapinfo -h
Device 1K-blocks Used Avail Capacity /dev/label/swap0 2047996 0B 2.0G 0% /dev/gpt/swap0 5244824 0B 5.0G 0% Total 7292820 0B 7.0G 0%
Turn off the old swap.
# swapoff /dev/label/swap0
The old swap is unused. I can put it out of my misery. Double-check gpart show to learn which partition is your swap space (3) and your temporary placeholder (4). Double double-check these numbers. We’re going to remove these partitions. If you delete your data partition due to your own stupidity you will be… unhappy.
# gpart delete -i 3 vtbd0
vtbd0p3 deleted
# gpart delete -i 4 vtbd0
vtbd0s4 deleted
Triple double-check: do you still have a root filesystem? (Yes, FreeBSD has safeguards to prevent you from deleting mounted partitions. Check anyway.)
# gpart show
=> 34 1342177213 vtbd0 GPT (640G)
34 128 1 freebsd-boot (64K)
162 6 - free - (3.0K)
168 666988544 2 freebsd-ufs (318G)
666988712 664698880 - free - (317G)
1331687592 10489655 5 freebsd-swap (5.0G)
Our swap space is at the end of the disk. And we have 317GB of free space right next to our root filesystem. You have not ruined your day. Yet.
Double-check your backups. Do you really have everything you need to recreate this server? If so, expand the root filesystem with gpart resize. Don’t specify a size, and the new partition will fill all available contiguous space.
# gpart resize -i 2 vtbd0
vtbd0p2 resized
# gpart show
=> 34 1342177213 vtbd0 GPT (640G)
34 128 1 freebsd-boot (64K)
162 6 - free - (3.0K)
168 1331687424 2 freebsd-ufs (635G)
1331687592 10489655 5 freebsd-swap (5.0G)
Now I have a 318GB filesystem on a 636GB partition. Let’s expand that filesystem to fill the partition. You can’t resize a filesystem label such as /dev/label/root0, you must use a partition identifier like vtbd0p2 or /dev/gpt/rootfs0. In FreeBSD 10, you can use growfs on mounted file systems.
# growfs /dev/vtbd0p2
It's strongly recommended to make a backup before growing the file system.
OK to grow filesystem on /dev/vtbd0p2 from 318GB to 635GB? [Yes/No] yes
growfs: /dev/vtbd0p2: Operation not permitted
Not permitted? I activated GEOM debugging mode by setting kern.geom.debugflags to 0x10, but was still denied. I’ve grown mounted filesystems before, so what the heck?
This virtual server has disabled soft updates, journaling, and all the fancy FreeBSD disk performance features. I suspect this error is tied to that. Let’s go to single user mode and grow the filesystem unmounted.
I reboot, and get:
Mounting from ufs:/dev/label/rootfs0 failed with error 19.
Even when you know what’s wrong, this message makes that little voice in the back of my skull simultaneously call me an idiot and scream “You destroyed your filesystem! Ha ha!” Plus, I can no longer make notes in my web browser–the article is on the non-running server.
Fortunately, I know which partition is the root partition. I enter
mountroot> ufs:/dev/vtbd0p2
and get the familiar single-user-mode prompt. Now I can do:
# growfs vtbd0p2
I answer yes, and new superblocks scroll across the screen. The filesystem grows to fill all available contiguous space.
My suspicion is that resizing the partition destroyed the label. Many GEOM classes store information in the last sector of the partition. Let’s use a GPT label instead.
# gpart modify -i 2 -l rootfs vtbd0
Mount the root filesystem read-write.
# mount -o rw /dev/vtbd0p2 /
Create a new /etc/fstab entry for the root filesystem, using the GPT label instead of the glabel(8) one.
/dev/gpt/rootfs / ufs rw,noatime 1 1
And then a reboot to see if everything comes back. It does.
My partitions now look like this:
# df -h
Filesystem Size Used Avail Capacity Mounted on /dev/gpt/rootfs 615G 6.1G 560G 1% / devfs 1.0K 1.0K 0B 100% /dev devfs 1.0K 1.0K 0B 100% /var/named/dev
All installed disk space is now in use. Mission accomplished!
Having written this, though, I have no chance of forgetting that I need to go back and do a custom install to partition the server properly.
Phabricator on FreeBSD installation notes
BSDs generally break their PHP packages into smaller units than most Linux distribution. This means that you need extra packages when following installation guides. I’m installing Phabricator on FreeBSD because I want ZFS under it.
This is the complete list of PHP modules and related stuff I needed to install to get Phabricator to run on FreeBSD 10.0p7/amd64. Don’t use PHP 5.5, as some modules are only available with PHP 5.4.
php5
mod_php5
php5-curl
php5-gd
php5-iconv
php5-mbstring
php5-mysql
php5-pcntl
pecl-APC
php5-filter
pear-Services_JSON
php5-json
php5-hash
php5-openssl
php5-ctype
php5-posix
php5-fileinfo
Restart your web server after installing everything.
Phabricator wants a lot of control over its database. I don’t like giving web applications root privileges on a database. this article by David Antaramian was quite helpful there.
Once you have your user set up, initialize the Phabricator database by running
# ./storage upgrade --user root --password MyRootPassword
This gives the script the access needed to actually create and adjust Phabricator databases.
After that, the Phabricator installer seems to do a good job of walking you through fixing the various setup niggles.