my .cwmrc

I need a window manager that doesn’t take up desktop space with lots of icons and permits me to work without removing my hands from the keyboard. (I do use the mouse, mind you, but I don’t want to be required to use the mouse for routine tasks.) I’ve used cwm on OpenBSD as my window manager for about a year, and it’s fulfilled my needs perfectly well.

I have made some changes. cwm is very easy to modify via the .cwmrc file. Here’s mine, with comments interspersed to show why I’ve made these changes.

I want a narrow blue border around my windows, and I want the active window to be highlighted. Maximized windows leave a 180-pixel gap at the right-hand side of the screen, so my xclock remains visible. (Without a clock, I sink into an authorial coma and lose awareness of time, sound, light, and so on.)

borderwidth 2
color activeborder blue
color inactiveborder darkblue
gap 0 0 0 180

My most heavily-used big applications are OpenOffice and Firefox. Both of these highlight text a word at a time when you use CTRL and an arrow key. By default, cwm maps CTRL-arrow to moving the pointer. I’m not going to train my fingers to make my applications work differently on OpenBSD than on any other operating system, so I needed to make cwm pass CTRL-arrow through to applications and use another key sequence for moving the pointer.

bind CS-Left unmap
bind CS-Right unmap
bind CS-Up unmap
bind CS-Down unmap
bind C-Left unmap
bind C-Right unmap
bind C-Up unmap
bind C-Down unmap
bind 4S-Left bigptrmoveleft
bind 4S-Right bigptrmoveright
bind 4S-Up bigptrmoveup
bind 4S-Down bigptrmovedown
bind 4-Left ptrmoveleft
bind 4-Right ptrmoveright
bind 4-Up ptrmoveup
bind 4-Down ptrmovedown

The 4th option key (which probably appears as the Windows key on your keyboard) and an arrow now controls pointer movement.

I experimented at length with my cwm configuration, and wanted to be able to make cwm reread .cwmrc with a keyboard shortcut, such as CTRL-ALT-r.

bind CM-r reload

I have a preferred screensaver, activated on a CTRL-ALT-DELETE.

command lock '/usr/X11R6/bin/xlock -mode flow'

Finally, I want to be able to start commonly-used programs via a right mouse click on the background. I don’t have to start them this way, but the option is convenient.

command firefox /usr/local/bin/firefox
command xpdf /usr/local/bin/xpdf
command OOo /usr/local/bin/soffice

With these modifications, cwm stays out of my way and lets me work.

And while I’m babbling about cwm: when you spawn an SSH session with CTRL-. you can put SSH command-line arguments before the hostname. Very useful for when you want to enable, say, X11 forwarding for a particular session.

OpenBSD, Firefox, and Flash

An OpenBSD advocacy article led me to a mailing list posting on how to get Flash playing transparently in Firefox on OpenBSD. You could get Flash (and other clunky media formats) to work on Unix-like platforms that Adobe doesn’t support for some time now, using a combination of players in packages and nearly random hacks in Firefox. This process makes everything simple, however.

I’m using 4.9/i386 and mozilla-firefox-3.6.13p3.

Set a package path in your shell. A few tests with ping and traceroute showed ftp3.usa.openbsd.org is my closest mirror. I use tcsh, so my .cshrc has:

PKG_PATH=ftp://ftp3.usa.openbsd.org/pub/OpenBSD/4.9/packages/i386/

Then run:

$ sudo pkg_add -r gecko-mediaplayer

This installs a whole bunch of packages for playing not just Flash, but other complicated media formats.

Browse to https://addons.mozilla.org/en-US/firefox/addon/flashvideoreplacer/ and install the plugin. The plugin handles replacing Flash with appropriate third-party players from packages.

Restart your browser. And everything just works.

FreeBSD iSCSI panic

I woke up today to find a console with:

panic: _mtx_lock_sleep: recursed on non-recursive mutex iscsi-io @ /usr/src/sys/modules/iscsi/initiator/../../../dev/iscsi/initiator/isc_sm.c:324

The initiator is a FreeBSD-current amd64 from 8 May 2011. The iSCSI target is an inexpensive iomega NAS. Other hosts attached to this iSCSI NAS have also had errors, though. The errors clear when I reboot the NAS.

Unfortunately, the FreeBSD box is a diskless system. Dumps aren’t exactly simple. While I heard some rumours about a network dump facility coming soon at the FreeBSD BSDCan devsummit, that’s the future.

How to fix this?

I attended the High Performance FreeBSD Clusters talk at BSDCan 2011. The presenter had originally used FreeBSD servers, then tried OpenSolaris to get better performance. He had OpenSolaris problems, but found that they could not access the bug information without a support contract. They’re now moving towards FreeBSD with EIT, and are happier.

I intend to learn from their mistakes, and replace the iomega with a FreeBSD EIT server. I’ll keep the iomega for, say, a central ports and packages NFS server, where a reboot won’t impact my uptime.

Why bother to blog this? So that the next poor bugger who gets this panic message gets at least one search engine hit.

BSDCan 2011

BSDCan 2011 was great. The problem with a conference that’s routinely great is that great becomes routine, and hence boring. Several presentations struck me as notably interesting for a variety of reasons, and I wanted to comment on three of them. These are only my personal opinions, of course. BSDCan had three tracks, and I could only be in one talk at a time.

Mark Linimon’s talk on How not to build a lights-out facility discussed the FreeBSD Project’s efforts to mirror its core infrastructure in datacenter space donated by New York Internet. As a chronicle of lessons learned and things that should be done differently next time, it’s valuable listening for anyone who thinks that building heavy-duty project infrastructure is easy.

I’m not going to name the people, the projects, or the code involved in the second talk, because the talk itself is less important than what happened during it. A committer from one large BSD project presented on a new piece of infrastructure he had developed. The audience included people associated with a variety of BSD projects. At the end of the talk, a senior developer from a different BSD project asked a few questions. The presenter and the developer had several rounds of completely civil back-and-forth technical discussion, and at the end the presenter agreed that the developer had some strong points and that some parts of his infrastructure needed additional work. I’m told that this happened in more than one talk. Despite discussion of disagreements between various BSDs projects, it’s clear that technical correctness is still most important.

The presentation I found most technically interesting was Randall Stewart’s work in data center congestion control. Stewart did real-world testing of data center congestion control with ECN and SCTP, and presented his results. It wasn’t until hours later that I realized exactly why I found the talk so interesting: he had essentially done “Mythbusters” for a specific part of TCP/IP. He’d bought a bunch of $50 servers on eBay, repeatedly adjusted SCTP’s response to packets with ECN set, and graphed the results. This was real-world stuff suspiciously close to academic research, done in a basement. And this sort of research is something that almost anyone could do. Lots of claims are made for our network stacks, but very few people actually experiment to measure performance with their workloads.

I’m glad to see open source projects learning lessons. I’m glad to see different BSD camps politely testing their ideas against each other, creating better software for everyone. But I’m really really happy to see real-world experiments.

I see all sorts of claims for different BSD’s network stacks, disk performance, and so on. Please, put them to the test. Make changes. Measure the results. While this work requires real hardware rather than virtualization, it’s something that anyone can do. You know your workload. Read about benchmarking. While naive benchmarks aren’t useful, it’s not that hard to design valid benchmarks. Buy used hardware, run your own tests. Make changes, and test again. Measure and document everything. Capture packets, and keep the pcap files so that you can go back and answer interesting questions. Publish your results. You’ll get interest. Perhaps your results will be as you expect. Maybe they won’t. But you’ll never know until you try.

As a BSDCan committee member, I would love to see more work like this. I can’t guarantee that your paper would be accepted, but I can say I’m much more likely to vote for a paper with a real investigation than yet another talk on well-understood features. Even if your results say “Yes, the fooBSD disk I/O system works exactly as expected,” it’s still interesting. And if you discover weak spots, you’ll have evidence the developers will need to improve performance.

Populating OpenSSH chroot on OpenBSD

I need to confine the user jrlodden to his home directory on this OpenBSD 4.9/i386 system, but give him a shell prompt and access to a couple of specific commands. While the SFTP server has built-in chroot support, a shell environment is more complicated. The /etc/ssh/sshd_config part is pretty simple…

...
#ChrootDirectory none
...
Match User jrlodden
ChrootDirectory %h

This chroot directory is nonfunctional. I must create device nodes and add necessary programs. Start by creating the the user account with adduser(8), creating standard device nodes, and removing unnecessary nodes.

# cd ~jrlodden
# mkdir dev
# cd dev
# /dev/MAKEDEV std
# ls
arandom klog ksyms null stdin tty zero
console kmem mem stderr stdout xf86
# rm console klog kmem ksyms mem xf86
# ls
arandom null stderr stdin stdout tty zero

He’ll need a statically-linked shell, such as /bin/ksh.

# cd ~jrlodden
# mkdir bin
# cd bin/
# file /bin/ksh
/bin/ksh: ELF 32-bit LSB executable, Intel 80386, version 1, for OpenBSD, statically linked, stripped
# cp /bin/ksh .

A chrooted user should not have write access to his own root directory. He will need a home directory in the chroot, however.

# chown root:wheel /home/jrlodden/
# mkdir -p /home/jrlodden/home/jrlodden
# chown jrlodden:jrlodden home/jrlodden

Now he can log in:

# ssh jrlodden@chroothost
ksh: No controlling tty (open /dev/tty: Device not configured)
ksh: warning: won't have full job control
$

jrlodden is logged in and cannot access anything beyond his cell. While I’d like to clean up the /dev/tty warning, I can’t seem to create /dev/ttypc in the chroot’ed /dev. For now, I can copy statically-linked versions of his necessary programs into /home/jrlodden/bin and get on with my life.

Absolute OpenBSD, 2nd Edition

I promised I’d announce the title of my next No Starch Press book in my BSDCan talk. That happened. The rest of you had to wait until now to hear that I’m rewriting Absolute OpenBSD. The technical reviewer is Peter Hansteen, author of The Book of PF.

Most of the book does not exist yet. Best guess for a release date is some time in 2012.

Why did a second edition take so long?

I will only write books about tools I use in production, out in the real world. (Desktop use does not count.) In my previous job, senior network engineer at a global automotive supplier, I had no opportunity to use OpenBSD. That meant I couldn’t offer advice about using OpenBSD, or discuss how it fit into my infrastructure. I could have written the book, but it would have sucked.

I’m also working on a second nonfiction project, but I’ll announce that separately.

new package system coming for FreeBSD

From the BSDCan FreeBSD developer summit:

The ports team has developed new package management tools and methods to simplify FreeBSD package management. The hope is to have these as the default in FreeBSD 10. Erwin Lansing has posted slides from his brief presentation, and a Web search for “pkgng FreeBSD” will get you all sorts of details.

BSDCan this week

BSDCan! Are you going? Why not? Sorry, that excuse isn’t good enough. Get there. I arrive Tuesday. I will be looking for you. Do not make me come looking.

As a result of BSDCan, as well as preparing to sell my house, various stuff has been delayed. If you’re waiting on me, I’ll get to you soon. Really.

The good news is, the house painting is finished. All that remains is to pack. While not fun, packing can be done in smaller chunks of time than painting. Hoping to get book writing back on track as a result.

New FreeBSD Installer test and walkthrough

For those who missed the announcement, FreeBSD 9 has replaced sysinstall with a new installer. It’s based on the PC-BSD installer’s back end and a text-based front end by Nathan Whitehorn. So, what is this new installer we’ve waited sixteen years for? I decided to find out.

As an official snapshot with the BSD installer isn’t yet available, I grabbed one of nwhitehorn’s recent snapshots. The back end is not integrated, but let’s see what the front end looks like. I assume there will be changes between this February snapshot and the next official snapshot. For my testing, I used a VM running Microsoft Virtual PC (for ease of screenshots) with ACPI disabled (so the install would finish).

Booting the image, I saw:

First Boot Screen
Select Install

A recovery shell and a live CD are great, but I want to install.

The next screen tells me to select a keyboard. By virtue of alphabetical order, the default is Armenian. When I arrow down to find the US keyboard, there’s several different options. I know exactly what kind of keyboard I have, it’s an Endurapro with the mouse pointer between the G and the H. But that’s not an option. Fortunately, I use a Dvorak layout, and that is an option.

I’m now asked for a hostname, and then allowed to choose optional components.

what optional parts to install
Distribution Choices

Of course I want source code!

I will miss the ability to choose “the smallest distribution possible,” but in reality, upgrading added the entire base system anyway.

Now we partition the disk.

Partition Methods
Partitioning Tools

Ooooh, a “guided” method. That sounds newbie-friendly. Let’s try that.

Use Entire Disk?
Conquer, or Divide?

We want to use the whole disk.

Partitions
Partitions

Wait a minute… where are /var, /tmp, /usr, and all the other partitions? Don’t tell me that one of the last survivors of proper partitioning is stumbling into the one large root filesystem trap? Ick. (I assume that the other partitioning methods will let you create proper partitions.)

Note that these are GPT partitions, not MBR partitions (slices).

Highlighting a partition and pressing ENTER shows details.

partition details
partition details

We can set a label in the installer, which is a nice feature. And it appears that two ESC takes you out of this submenu into the main menu. That seems odd, but perhaps it’s related to this being in Virtual PC. Arrow over to EXIT and hit ENTER to get the final confirmation screen.

are you sure?
you're committed now

“Are you sure?” Save and Abort aren’t exactly answers to that question – actually “save” is kind of ambiguous in this context. Save your existing data? Yes, we’re sure, choose Save. The filesystem is initialized:

newfs
newfs running

The installer verifies the files on disk and begins installing.

unpacking the archive
Oooh, a lock order reversal!

We must watch for a panic in the background, versus a simple lock order reversal. This is -current, either one can happen and both are displayed by default.

While watching this, I wanted to hit ALT-F2 and see just how well the ports were extracting. That’s apparently no longer an option.

After extracting everything, I was prompted for the root password, then for a network interface.

Choose Interfaces
Choose Interfaces

I took the default.

Yes, please
Use DHCP?

Why yes, I would like DHCP with my host, thank you.

Now I’m asked for the services I want enabled at boot.

just a few choices
services to start at boot

Compared to sysinstall there aren’t many choices, but I’m all right with being offered only the lowest common denominators. Far more vexing is how the network configuration gracelessly handled the unplugged network interface. While CTRL-L redrew the screen, a new link down message reappeared every second or so. I’d never tried Virtual PC before, and apparently I need to decipher the network configuration.

Once I choose the essential services, I’m asked if I want to add users. When I say yes, it drops me to text mode for the usual adduser(8) dialog.

adding our first user
adding a user, plus broken networking

All right, the network message is getting pretty annoying now.

After adding my user, I get the final screen.

Final Install Options
last chance to change things

You can change many of your optional settings, plus set the time zone. You get the usual warnings about removing the CD-ROM from the drive, and then reboot into the new install.

Overall, I like many of the changes. It doesn’t have many of the libdialog annoyances that plague sysinstall. I dislike the guided partitioning system, or more exactly the lack of actual partitioning therein. I do not want /tmp files filling my hard disk!

The good news is, with the separation between the front end and the back end, changing the installer will be much simpler than it ever was with sysinstall. For all the petty annoyances in this installer, it’s a big step in the right direction. I want to commend the creators for actually pushing Sysinstall Replacement Attempt #82,319 to completion.

[emerg] (13)Permission denied: couldn’t grab the accept mutex

I installed Apache on a diskless FreeBSD-9/amd64 server. Once I added SSL, the web server wouldn’t start. It died with:

[Mon Mar 21 15:37:16 2011] [emerg] (13)Permission denied: couldn't grab the accept mutex
[Mon Mar 21 15:37:16 2011] [emerg] (13)Permission denied: couldn't grab the accept mutex
[Mon Mar 21 15:37:16 2011] [emerg] (13)Permission denied: couldn't grab the accept mutex
[Mon Mar 21 15:37:16 2011] [emerg] (13)Permission denied: couldn't grab the accept mutex
[Mon Mar 21 15:37:16 2011] [emerg] (13)Permission denied: couldn't grab the accept mutex
[Mon Mar 21 15:37:17 2011] [alert] Child 1733 returned a Fatal error... Apache is exiting!

Google says that the fix for this is to define AcceptMutex flock somewhere in httpd.conf. Google shows dozens of mailing list discussions that give this advice, but without explanation. Every so often, someone whinges about it not working, and doesn’t get a response.

This is an example of what I call “Occult IT.” There are certain recipes that people just follow, without understanding why. We stumble around, grasping for invocations and incantations that will fix our problem. I don’t just want the ritual that will solve the problem; I want to know why the problem happened and deepen my understanding of the issue.

Besides, adding AcceptMutex flock didn’t work.

To identify the problem, I set LogLevel to debug in httpd.conf and tried starting httpd again.

[Mon Mar 21 16:03:18 2011] [debug] prefork.c(1018): AcceptMutex: flock (default: flock)

I didn’t have that configuration setting in my httpd.conf, but it’s the Apache default anyway. So, what are my other choices?

Apache documentation says that the AcceptMutex setting determines how Apache serializes incoming requests. The flock setting dictates how Apache locks the LockFile.

My configuration doesn’t have a defined LockFile, so choosing how we lock it isn’t going to help.

I don’t like lock files. Bad stuff happens to them. Let’s try a locking method without a lockfile. The documentation lists two different classes of locking mechanisms. flock and fcntl work on lock files. posixsem, pthread, and sysvsem use the in-memory mechanisms of semaphores and/or mutexes to provide locking.

As I don’t like lock files, I’ll try one of the in-memory mechanisms.

AcceptMutex posixsem

And Apache starts and runs perfectly.

I can’t find any details on the differences between these in-memory mechanisms, from a system administrator’s point of view. I imagine that the System V mechanism wouldn’t work if you’d removed that support from your kernel. But the point is:

Do not rely on occult IT. Read The Fine Manual.