Category: Tech

What most of you probably want

Posted on

WordPress versus FreeBSD

I recently migrated my web site to a new FreeBSD install, configured so I could use ZFS boot environments. This upgrade crossed FreeBSD versions (10.0->10.1), filesystems (UFS -> ZFS), and PHP versions (5.5 -> 5.6).

And my WordPress pointy-clicky upgrades stopped working. Every time I ran an upgrade, the web gui hung with:

Updating Plugin Honketyblatt (1/1)

The web site would site there, forever. Enabling WP debugging gave me no error messages.

If I had the job of running WordPress sites, I would have an automatic tool that processed the upgrades for me. It’s not, so I don’t.

I use the FreeBSD WordPress package to get all of the dependencies, but manage my actual WordPress sites in a separate directory. It turns out that the FreeBSD WordPress package doesn’t list all of the modules that you need for a self-maintaining WordPress install. My old server had a few packages that the new one didn’t.

If you want to use WordPress’ self-updating features, be sure to install the following packages in addition to the FreeBSD-recommended defaults.

php56-tokenizer
php56-zlib
php56-zip

I installed these packages, and everything started working.

Posted on

n4sa Print Sneaking Out

The print version of Networking for Systems Administrators is starting to appear for purchase on Amazon’s web site.

  • n4sa US
  • n4sa UK

    Other Amazon sites should appear shortly.

    Amazon has not yet performed their usual discounting. Usually, if you order a brand-new book they’ll retroactively drop the price. But not always. I’d tell you to add it to your cart today and check tomorrow.

    Vendors like Barnes & Noble and Powells will pick it up in a week or two.

    • Posted on

    FreeBSD Jail Management Tools

    To design “FreeBSD Mastery: Jails” I need to look at the existing jail management tools. Jails have been around about fifteen years now, and FreeBSD has accumulated a whole bunch of wrappers and supporting tools. Many of these have wound up in the ports collection.

    Jails have evolved over the years. Some of these add-on tools are not useful for FreeBSD 9.1 and later.

    Here’s a few things I discovered in my research. I’m hoping that you lot will offer your own comments and help me decide which tools to cover in the book.

    It seems we have five major jail management toolkits.

  • ezjail – perhaps the best known jail management tool. Written entirely in shell.
  • qjail – Designed for managing lots of jails at the command line, based on templates. The examples use ipfilter, which is my third choice of FreeBSD firewall. Does not need ZFS.
  • iocage – supports resource limiting, thin provisioning, cloning, and either vimage or NAT from the host’s main IP.
  • jadm – Python-based jail command shell, uses a bridge interface. Can migrate jails between hosts. ZFS integration. Lets you set global settings for all jails, per-jail settings, jail groups, and so on.
  • cbsd – web-based management of jails. Supports HAST, migration, CARP, etc.

    The question for me is: which should I cover in the jails book? I’ll mention that all of them exist, but I can only give attention to one or two.

    CBSD seems an obvious choice. It integrates CARP and HAST and vimage and just about everything. Plus, people like web GUIs. It seems to be the giant ape of jail management tools.

    But I want to cover a command-line toolkit. Between ezjail, qjail, iocage, and jadm, I find myself leaning towards iocage.

    There’s some other jail-related software in the ports collection. Here’s those I plan to investigate and possibly include. I might find that their functionality is now included in mainline FreeBSD, however.

  • jps and jtop – external wrappers that add jail info to ps and top.
  • jkill – shuts down a running jail and all its processes from outside the jail. I don’t know that this is still needed, but the functionality is important.
  • bsnmp-jails – feed jail info into snmpd.

    Here are some jail-related ports I don’t plan to include, and why.

  • py-ploy_ezjail, bsdploy – ploy for jails. I don’t ploy.
  • py-ezjailremote – a python wrapper around ezjail. I don’t Python.
  • p5-BSD-Jail-Object – a Perl interface for jail management. I do Perl, but… no.
  • pkg_jail – build packages inside a jail. This looks like an old poudriere.
  • jailrc – improved startup/shutdown scripts for pre-9.1 jails. The key words here are “pre 9.1.”
  • pam_jail – drops the user into a jail upon successful login
  • jailme – a modified version of jexec with more sanity checking. Is setuid, lets normal users run jails.
  • jaildaemon – lets the jail talk to the host? I’m sure this solved a problem for someone, but not me.
  • jailctl – for FreeBSD 4.x and 5.x
  • jailaudit – portaudit for inside jails. I’d say this is superceded by pkg audit.
  • jail2 – advanced jail script. Uses /etc/jail.conf. I’m kinda, sorta sure that this or its descendant is the default FreeBSD 9.1 and later.
    • Posted on

    Sudo talk now on YouTube

    My talk Sudo: You’re Doing It Wrong is now live on YouTube. (Thanks to TJ for letting me know.) The talk is based on my book Sudo Mastery.

    This talk went better than my NYCBSDCon talk. Probably because I hadn’t confused “buzzing with caffeine, adrenaline, and sleeplessness” with “raging tonsilitis.” The Q&A at the end took us wildly astray, and ended with the general conclusion that “Lucas needs to present to mug.org about how to use SSH correctly.”

    I gave away a couple books, one Sudo Mastery and one SSH Mastery. The SSH book went to the first person to raise their hand and admit that they used passwords with SSH.

    But I’m sure none of you use password-only authentication with SSH. You’re all good, decent, moral people who wouldn’t do anything that vile.

    Posted on

    Sudo talk at mug.org, 9 Dec 2014

    I’ll be talking at mug.org in Farmington Hills, MI, on 9 December 2014.

    The topic is Sudo: You’re Doing It Wrong. If you use sudo, you need to show up for this. Because you’re doing it wrong. It’s based on Sudo Mastery, as you might guess.

    Come to mug.org. They have cookies.

    They usually record and show their talks, so if you can’t be bothered to go to Farmington Hills in December you can probably catch it on YouTube later. But it won’t be nearly as awesome.

    Also, I’m planning to go to the IT in the D casual social event on 20 November 2014. I’m not speaking, just hanging out. Why?

    Now that I’m a full time writer, this is my staff.
    Tilted Windmill Press staff
    They’re perfectly sociable, and definitely cuter than most of my previous co-workers, but they’re a little short in the techie conversation department.

    So, yeah. Two chances to see me in the near future. No public appearances planned afterwards. I’ll be busy trying to teach my staff how to copyedit.

    Posted on

    FreeBSD “Working copy ‘/usr/src’ locked.”

    Poul-Henning Kamp is working with me on some GBDE fixes. Which means he sends me patches and says “Here, try this,” along with very valuable exposition on how GBDE works and the threat model it applies to. This means I’m updating frequently.

    My usual update process is:

    # cd /usr/src
    # make update && make -j8 buildworld && make -j8 kernel && reboot

    Half a second after typing this, I realized I’d forgotten to apply PHK’s latest patch. I hit CTRL-C during the make update before building an unsuitable userland. I need to do the source update, apply the patch, and then build everything.

    # make update
    --------------------------------------------------------------
    >>> Updating /usr/src using Subversion
    --------------------------------------------------------------
    svn: E155004: Run 'svn cleanup' to remove locks (type 'svn help cleanup' for det
    ails)
    svn: E155004: Working copy '/usr/src' locked.
    svn: E155004: '/usr/src' is already locked.
    *** Error code 1

    Stop.
    make[1]: stopped in /usr/src
    *** Error code 1

    Fine. svn cleanup it is. But wait–I don’t have svn installed!

    As a matter of principle, I don’t want to install svn on this test box. svnlite is what FreeBSD offers to users, so it should be able to handle everything.

    The good news is, svnlite also have a cleanup feature.

    # svnlite cleanup
    # make update

    And the update proceeds as I would hope.

    Is this worth a PR to get the error message changed? Dunno. What do you think?

    Now all I must do is master all the GBDE wisdom PHK dumped in my brain…

    Posted on

    Next Project: “Networking for Sysadmins”

    FreeBSD Mastery: Storage Essentials is out for tech review. (If you’re reading the pre-pub book, you’ve got a few more days to get comments back to me.) I’ll then make the corrections and send it to copyediting.

    So I’m writing another book.

    The current title is Networking for System Administrators. (I’d like to work the word “Mastery” in there, but it sounds artificially kludged together, because it would be.) It’s a small book, readable in a couple hours.

    I’ve worked in a whole bunch of IT organizations as both a system administrator and a network administrator. In most of them I get sucked into a bridge role because I can speak to both teams in their own language.

    It’s hard to teach a network administrator to be a sysadmin. An enterprise often runs a dozen or more different operating system, and who knows how many variants of each. Plus, each team might configure their differently. “You need a password to sudo here, you need a Yubikey to log on here, you need a hole in the head to log on here…” oy vey! Asking a network administrator to learn all this is like asking a sysadmin to configure Cisco, RouterOS, and OpenBSD routers. It just isn’t going to happen.

    But the basic principles of networking isn’t hard, and understanding basic networking can save the sysadmin so much time. A sysadmin who wants to learn networking is often referred to books like The TCP/IP Guide or TCP/IP Illustrated. These are awesome books, and some systems administrators (and all non-web app developers) need to read them. For the majority of sysadmins, they’re overkill. An enterprise database administrator who needs to understand TCP/IP window scaling to do his job should call his network administrator.

    Instead, most sysadmins learn networking via occasional blog posts, Google searches, and oral tradition. This is a ghastly way to learn any technical topic.

    The result? Calls the sysadmin doesn’t want to make and the network administrator doesn’t want to get.

  • “Did that firewall port ever get opened?”
  • “Is my server plugged into the right network?”
  • “What do you mean that service is broken, I can ping it?”
  • “That service isn’t working, I can’t ping it.”
  • “That UDP port isn’t open, I can’t telnet to it!”

    A knowledgeable sysadmin can quickly answer all of these questions for themselves without picking up the phone. And we wouldn’t be in IT if we wanted to talk on the phone.

    The table of contents so far is:

  • Introduction
  • Network layers — the bottom 4 layers, and troubleshooting pointers to later chapters
  • Ethernet
  • IPv4
  • IPv6
  • TCP/IP (protocols, ports, etc)
  • Active traffic (netstat)
  • DNS
  • Checking the Network (sending vs receiving)
  • tcpdump (what we receive)
  • netcat (what we send)
  • packet filtering for sysadmins
  • tracing problems (traceroute & mtr)

    This book also contains guidance on detecting an uneducated network administrator. “Filtering all ICMP, because ICMP is bad? Bzzzt!” I don’t put it in quite those terms, but… yeah. You at least need to know what you’re dealing with.

    Unlike my earlier Mastery books, the incomplete draft of this book will not be available for pre-order. Sales of books that I offer for pre-order are much lower than books I don’t offer pre-order on. Part of this is the topic–DNSSEC has less popular interest than SSH. But the sudo book is doing much less well than I expected, excluding a spike from the Slashdot review. (Reviews on sites like Slashdot help sales more than anything I’ve found.)

    From talking to other indie authors, it seems that an initial surge of sales strongly affects online bookstore’s algorithms. I say seems because most online bookstores do not make their algorithms public–they don’t want clever buggers like you telling me how to game their system.

    The only way for me to tell is to test it, however. I won’t be doing preorders for this book and the next FreeBSD Mastery title.

    I believe that many of my readers don’t need this book. I do hope that you’ll tell certain people you work with to read it, however. You know the ones I mean.

    More updates as events warrant. Or you can check Twitter for the hashtag #n4sa. (I’m not the only one with that hashtag, but it seems pretty rarely used, so I’ll claim it.)

    • Posted on

    Revoked and Replaced OpenPGP Key

    I uploaded a GPG key to subkeys.pgp.net back in 2005. It’s well past time for me to replace it. I covered creating your revocation certificate back in PGP & GPG, but didn’t actually write about using that revocation certificate. Nine years later… yeah, I better figure this out.

    So Io to the machine with my keypair, and create my revocation certificate.

    # gpg --output oldgpg.revoke.asc --gen-revoke E68C49BC

    sec 1024D/E68C49BC 2005-02-21 Michael Warren Lucas Jr (Author, consultant, sysadmin)

    Yep, that’s my old key.

    Create a revocation certificate for this key? (y/N) y
    Please select the reason for the revocation:
    0 = No reason specified
    1 = Key has been compromised
    2 = Key is superseded
    3 = Key is no longer used
    Q = Cancel
    (Probably you want to select 1 here)
    Your decision? 2

    Why is this key being revoked? Because it’s nine years old. I’ve generated a new key,

    Enter an optional description; end it with an empty line:
    >
    Reason for revocation: Key is superseded
    (No description given)
    Is this okay? (y/N) y

    Nobody cares about the details, so I don’t enter any.

    You need a passphrase to unlock the secret key for
    user: "Michael Warren Lucas Jr (Author, consultant, sysadmin) "
    1024-bit DSA key, ID E68C49BC, created 2005-02-21

    I enter my passphrase.

    ASCII armored output forced.
    Revocation certificate created.

    I now have a revocation certificate, oldgpg.revoke.asc. To activate it, I import it into my keyring.

    # gpg --import oldgpg.revoke.asc
    gpg: key E68C49BC: "Michael Warren Lucas Jr (Author, consultant, sysadmin) " revocation certificate imported
    gpg: Total number processed: 1
    gpg: new key revocations: 1
    gpg: 3 marginal(s) needed, 1 complete(s) needed, classic trust model
    gpg: depth: 0 valid: 2 signed: 14 trust: 0-, 0q, 0n, 0m, 0f, 2u
    gpg: depth: 1 valid: 14 signed: 1 trust: 14-, 0q, 0n, 0m, 0f, 0u
    gpg: next trustdb check due at 2020-10-13

    No passphrase needed–it just happens.

    Now: sleep tight, sweet prince.

    # gpg --send-keys E68C49BC
    gpg: sending key E68C49BC to hkp server subkeys.pgp.net

    My old key is dead.

    For the record, my new key is 1F2E54A8, for mwlucas at michaelwlucas dot com.

    Now if I could only kill 4EBA9723…

    Posted on

    Shuffling Partitions on FreeBSD

    I’ve recently moved my personal web sites to https://www.vultr.com/, using virtual machines instead of real hardware. (I’ve caught up to the 2000s, hurrah!) I didn’t track server utilization, so I provisioned the machines based on a vague gut feeling.

    The web server started spewing signal 11s, occasionally taking down the site by killing mysql. Investigation showed that this server didn’t have enough memory. How can 1GB RAM not be enough for WordPress and MySQL? Why, back in my day–

    <SLAP>

    Right. Sorry about that.

    Anyway, I needed to increase the amount of memory. This meant moving up to a larger hosting package, which also expanded my hard drive space. After running gpart recover to move the backup GPT table to the end of the new disk, my new disk was partitioned like so:

    # gpart show

    =>        34  1342177213  vtbd0  GPT  (640G)
          34         128      1  freebsd-boot  (64K)
         162           6         - free -  (3.0K)
         168   666988544      2  freebsd-ufs  (318G)
   666988712     4096000      3  freebsd-swap  (2.0G)
   671084712   671092535         - free -  (320G)

    I have 320 GB of space space at the end of the disk.

    The easy thing to do would be to create a new partition in that space. I advocate and recommend partitioning servers. The only reason that this system has one large partition is because that’s what the hosting provider gave me.

    I’m writing a book on FreeBSD disk partitioning, however, so this struck me as an opportunity to try something that I need for the book. (As I write this, you can still get FreeBSD Mastery: Storage Essentials at a pre-pub discount.) How would I expand the root partition, with the swap space smack dab in the middle of the disk?

    Virtualized systems have no awareness of the underlying disk. Advice like “put the swap partition at the beginning of the disk” becomes irrelevant, as you have no idea where that physically is. On a system like this, how would I use the built-in FreeBSD tools to create a swap partition at the end of the disk, and expand the existing partition to fill the remaining space?

    This isn’t as easy as you might think. FreeBSD’s gpart command has no feature to add a partition at a specific offset. But it can be done.

    Any time you touch disk format or partitions, you might lose filesystems. Back up your vital files. For a WordPress web server, this is my web directory and the SQL database. (My backup includes a half-complete version of this article. If my repartitioning goes badly, I’ll retitle this piece “How to Not Repartition.” But anyway…) Copy these files off the target server.

    Now, what exactly do I want to do?

  • I want 4-5GB of swap space, at the end of the disk. (The server now has 2GB RAM.)
  • I want to remove the current swap space.
  • I want to expand the root partition to fill the remaining space.

    gpart(8) won’t let me say “create a 4GB partition at the end of the disk.” It will let me create a filler partition that I have no intention of actually using, however. As I’m sure the disk is not precisely 320GB, I’m going to play it safe and give myself 5GB of room for this swap partition. I give this partition a label to remind me of its purpose and role.

    # gpart add -t freebsd -s 315GB -l garbage vtbd0
    vtbd0s4 added

    The partitioning now looks like this.

    =>        34  1342177213  vtbd0  GPT  (640G)
          34         128      1  freebsd-boot  (64K)
         162           6         - free -  (3.0K)
         168   666988544      2  freebsd-ufs  (318G)
   666988712     4096000      3  freebsd-swap  (2.0G)
   671084712   660602880      4  freebsd  (315G)
  1331687592    10489655         - free -  (5.0G)

    Now I can add a swap partition at the end of the disk.

    # gpart add -t freebsd-swap -l swap0 vtbd0
    vtbd0p5 added

    The resulting partitioning looks like this.

    # gpart show

    =>        34  1342177213  vtbd0  GPT  (640G)
          34         128      1  freebsd-boot  (64K)
         162           6         - free -  (3.0K)
         168   666988544      2  freebsd-ufs  (318G)
   666988712     4096000      3  freebsd-swap  (2.0G)
   671084712   660602880      4  freebsd  (315G)
  1331687592    10489655      5  freebsd-swap  (5.0G)

    Tell /etc/fstab about the new swap partition, and remove the old one.

    /dev/gpt/swap0 none swap sw 0 0

    (In looking at the old entry, I realized that Vultr uses glabel(8) labels, where I use gpart(8) labels. Either type is fine, but I need to remember that /dev/label/swap0 is the old swap partition, and /dev/gpt/swap0 is the new one.)

    Activate the new swap space. I could reboot, but why bother?

    # swapon /dev/gpt/swap0

    My swap now looks like this.

    # swapinfo -h

    Device          1K-blocks     Used    Avail Capacity
/dev/label/swap0   2047996       0B     2.0G     0%
/dev/gpt/swap0    5244824       0B     5.0G     0%
Total             7292820       0B     7.0G     0%

    Turn off the old swap.

    # swapoff /dev/label/swap0

    The old swap is unused. I can put it out of my misery. Double-check gpart show to learn which partition is your swap space (3) and your temporary placeholder (4). Double double-check these numbers. We’re going to remove these partitions. If you delete your data partition due to your own stupidity you will be… unhappy.

    # gpart delete -i 3 vtbd0
    vtbd0p3 deleted
    # gpart delete -i 4 vtbd0
    vtbd0s4 deleted

    Triple double-check: do you still have a root filesystem? (Yes, FreeBSD has safeguards to prevent you from deleting mounted partitions. Check anyway.)

    # gpart show

    =>        34  1342177213  vtbd0  GPT  (640G)
          34         128      1  freebsd-boot  (64K)
         162           6         - free -  (3.0K)
         168   666988544      2  freebsd-ufs  (318G)
   666988712   664698880         - free -  (317G)
  1331687592    10489655      5  freebsd-swap  (5.0G)

    Our swap space is at the end of the disk. And we have 317GB of free space right next to our root filesystem. You have not ruined your day. Yet.

    Double-check your backups. Do you really have everything you need to recreate this server? If so, expand the root filesystem with gpart resize. Don’t specify a size, and the new partition will fill all available contiguous space.

    # gpart resize -i 2 vtbd0
    vtbd0p2 resized
    # gpart show

    =>        34  1342177213  vtbd0  GPT  (640G)
          34         128      1  freebsd-boot  (64K)
         162           6         - free -  (3.0K)
         168  1331687424      2  freebsd-ufs  (635G)
  1331687592    10489655      5  freebsd-swap  (5.0G)

    Now I have a 318GB filesystem on a 636GB partition. Let’s expand that filesystem to fill the partition. You can’t resize a filesystem label such as /dev/label/root0, you must use a partition identifier like vtbd0p2 or /dev/gpt/rootfs0. In FreeBSD 10, you can use growfs on mounted file systems.

    # growfs /dev/vtbd0p2
    It's strongly recommended to make a backup before growing the file system.
    OK to grow filesystem on /dev/vtbd0p2 from 318GB to 635GB? [Yes/No] yes
    growfs: /dev/vtbd0p2: Operation not permitted

    Not permitted? I activated GEOM debugging mode by setting kern.geom.debugflags to 0x10, but was still denied. I’ve grown mounted filesystems before, so what the heck?

    This virtual server has disabled soft updates, journaling, and all the fancy FreeBSD disk performance features. I suspect this error is tied to that. Let’s go to single user mode and grow the filesystem unmounted.

    I reboot, and get:

    Mounting from ufs:/dev/label/rootfs0 failed with error 19.

    Even when you know what’s wrong, this message makes that little voice in the back of my skull simultaneously call me an idiot and scream “You destroyed your filesystem! Ha ha!” Plus, I can no longer make notes in my web browser–the article is on the non-running server.

    Fortunately, I know which partition is the root partition. I enter

    mountroot> ufs:/dev/vtbd0p2

    and get the familiar single-user-mode prompt. Now I can do:

    # growfs vtbd0p2

    I answer yes, and new superblocks scroll across the screen. The filesystem grows to fill all available contiguous space.

    My suspicion is that resizing the partition destroyed the label. Many GEOM classes store information in the last sector of the partition. Let’s use a GPT label instead.

    # gpart modify -i 2 -l rootfs vtbd0

    Mount the root filesystem read-write.

    # mount -o rw /dev/vtbd0p2 /

    Create a new /etc/fstab entry for the root filesystem, using the GPT label instead of the glabel(8) one.

    /dev/gpt/rootfs / ufs rw,noatime 1 1

    And then a reboot to see if everything comes back. It does.

    My partitions now look like this:

    # df -h

    Filesystem         Size    Used   Avail Capacity  Mounted on
/dev/gpt/rootfs    615G    6.1G    560G     1%    /
devfs              1.0K    1.0K      0B   100%    /dev
devfs              1.0K    1.0K      0B   100%    /var/named/dev

    All installed disk space is now in use. Mission accomplished!

    Having written this, though, I have no chance of forgetting that I need to go back and do a custom install to partition the server properly.

    • Posted on

    Phabricator on FreeBSD installation notes

    BSDs generally break their PHP packages into smaller units than most Linux distribution. This means that you need extra packages when following installation guides. I’m installing Phabricator on FreeBSD because I want ZFS under it.

    This is the complete list of PHP modules and related stuff I needed to install to get Phabricator to run on FreeBSD 10.0p7/amd64. Don’t use PHP 5.5, as some modules are only available with PHP 5.4.

    php5
    mod_php5
    php5-curl
    php5-gd
    php5-iconv
    php5-mbstring
    php5-mysql
    php5-pcntl
    pecl-APC
    php5-filter
    pear-Services_JSON
    php5-json
    php5-hash
    php5-openssl
    php5-ctype
    php5-posix
    php5-fileinfo

    Restart your web server after installing everything.

    Phabricator wants a lot of control over its database. I don’t like giving web applications root privileges on a database. this article by David Antaramian was quite helpful there.

    Once you have your user set up, initialize the Phabricator database by running

    # ./storage upgrade --user root --password MyRootPassword

    This gives the script the access needed to actually create and adjust Phabricator databases.

    After that, the Phabricator installer seems to do a good job of walking you through fixing the various setup niggles.