OpenSSH community reviewers wanted

UPDATE: I have all the reviewers I can handle.

I have about half of the OpenSSH book written. I can start getting feedback on the manuscript. If you’re interested in providing feedback, first read the review process article on my web site.

If you’re still interested after reading that article, send me an email with the subject “OpenSSH review” and tell me that a) you won’t share the review manuscript, and b) why you’d be a good reviewer. I can only manage so many reviewers, so I try to pick readers of every experience level. My email address is m w lucas at black helicopters dot org.

And before you ask: four chapters of the OpenBSD book are finished. Not enough to solicit reviewers. It is proceeding apace, though. I usually work on multiple projects simultaneously, so this is not unusual.

How Community Tech Review Works

I’ve received quite a few questions about how I do community-based tech reviews on forthcoming books, as well as offers for one or both of the projects I have underway. I’ve put up a public Web page about the process I follow. I expect to request community reviewers for one book later this week.

Realistically, my brain is limited. I can only manage about 20 prepub reviewers for a given project. I choose the best people from the pool of volunteers.

I hope that all of them will return useful comments. I expect that about 10 of them will return nothing. Another 5 will drop out halfway through.

So, if you volunteer and I don’t pick you, you can feel superior in knowing that you would have been one of the 25% to stay all the way through, except that I wasn’t smart enough to know that beforehand.

desktop OpenBSD pf.conf

I have an OpenBSD 4.9/i386 desktop sitting naked on the Internet, and found people poking at my TCP ports. While PF is enabled by default, it’s configured to permit everything except remote X11. I need a policy that will block incoming traffic from everywhere except a few key IP addresses, while allowing me to make any outbound connections I desire.

mgmt="{192.0.2.0/24, 192.168.8.0/24}"
set skip on lo
block
pass proto icmp
pass from $mgmt to self
pass from self to any
block in on ! lo0 proto tcp to port 6000:6010

Disabling ICMP globally is a bad idea. Lots of stuff will break. I could specify permitted ICMP types, but that would be a lot of work and probably break my connectivity to random in a really obscure but educational manner.

I retained the X11 block, even for those known-good addresses, because if I’m trying to open an X11 connection to my home desktop without forwarding it over SSH, I am clearly solving the wrong problem.

If I permit password-based SSH connections from “known good” hosts, such as my house and the office, then if one of those machines is penetrated, the intruder will be able to get into my machine. I protect my desktop by requiring SSH public key auth, even from trusted IP addresses. An intruder could be savvy enough to hijack my agent forwarding, but in that case it’s someone specifically targeting me.

The current PF syntax is as close to painless as a firewall can be.

SSHv1 and PuTTY

One of the advantages of writing books is that you must double-check everything you thought you knew about a topic. PuTTY is probably the most widely deployed SSH client in the world. I’ve used it for years. It’s good software. (I also use the OpenSSH client, of course.)

To my surprise, PuTTY accepts both version 1 and 2 of the SSH protocol. It prefers version 2, but will accept 1.

Version 1 of the SSH protocol has irremediable problems. If a client accepts SSHv1, an attacker can intercept a new SSH connection and force it to downgrade to SSHv1. He can inject arbitrary commands into the SSHv1 stream. These problems have been known since 1998. Increases in computing power have made executing these attacks much simpler.

Worst of all, Ettercap can decode SSHv1 in real time. If Wireshark cannot decode SSH now, I suspect it will soon.

In my mind, this puts SSHv1 into the same category as Telnet and unencrypted read-write SNMP; stuff that Just Should Not Be On My Network.

I absolutely understand why PuTTY supports SSHv1 by default. The generous people who spend their free time writing PuTTY aren’t interested in supporting folks who can’t be bothered to read the instructions. I might make the same decision in their place.

And yes, host key verification helps eliminate MITM attacks. But do your users really verify host keys? Really and truly? The PuTTY FAQ lists “How do I turn off the annoying host key verification prompt?” as a question. As a sysadmin, I translate this as “yours users don’t verify host keys, and mine don’t either.”

There’s no reason for anyone who actually reads this blog to routinely permit SSHv1, and the appearance of security is worse than no security. I encourage you to disable SSHv1 by default in your and your users’ clients. Users can override the default on a host-by-host basis, but at least they must make the conscious effort. They’ll probably ask you for help. This will help you find lingering SSHv1 servers. If you have some embedded device that only speaks SSHv1, well, you have a job to do. That job should include replacing that device or yelling at the vendor.

How do you disable SSHv1 in PuTTY? Open PuTTY. On the left side, go to Connection->SSH. Select “2 only.” On the left side, select Session (at the top). Highlight “Default Settings.” Click Save. PuTTY saves its configuration in the registry, so you can export this setting and apply it to your client PCs through whatever method you use.

The most annoying part of this change is that PuTTY’s default settings do not propagate to all of the previously saved sessions. You must update them by hand or recreate them. I suspect that you could use some sort of script to update your saved sessions from your registry, but I can’t find such a thing. (This would be a great add-on tool for some Windows programmer looking for a way to contribute to the community.)

I will continue to highly recommend PuTTY to my Windows-based friends, with a note on how to disable SSHv1. As a lowly user who has no right to complain and who doesn’t have to listen to users whinge, though, I’d like to say to the PuTTY folks: researchers broke SSHv1 thirteen years ago. It’s time to stop accepting it by default.

Summer 2011 nonfiction project: OpenSSH

I have a problem with Absolute OpenBSD, 2nd edition. It’s too big. The outline is 26 chapters. This brings the book close to 300,000 words, well over a thousand pages. I don’t want to write books that I don’t like. I don’t like huge books that I cannot comfortably read in the bathtub.

One component of OpenBSD is OpenSSH. People have written books about OpenSSH, but they contain more information than most people need. (Not all, but most.) I write for the most common user, which means that those books are perhaps 2x-3x the size of what I would write. And OpenSSH is widely used outside OpenBSD: I could argue that it’s their most widely deployed component. Lots of people who will never use OpenBSD need a swift sharp smack with the OpenSSH clue-by-four.

AO2e book needs some OpenSSH stuff in it, of course. But I have about 30,000 words of OpenSSH. For comparison, the smallest nonfiction book I’ve written is about 60,000 words. It’s too small for a traditional publisher.

So I’m removing the intermediate-level OpenSSH material from AO2e, and doing a small independent book tentatively titled “OpenSSH: Your Next Steps.” It will cover the OpenSSH server, and the OpenSSH and PuTTY clients.

Many people believe they know how to use SSH: they download PuTTY, enter a hostname, username, and password, and “poof!” — they’re secure. It’s not that simple. If you search for documentation and articles about using OpenSSH, you’ll find years and years of accumulated cruft. Many of the most visible articles and posts are obsolete. A single source of up-to-date information at an inexpensive price would find readers, and would let me satisfy my self-publishing curiosity. I’ve contemplated independent publishing for some time now, and this looks like a realistic opportunity.

I’ll have to pay for copyediting, design, and so on. And I’ll have to manage the whole project. Worst case, I lose a lot of time and money, but develop a new appreciation for my publisher. (I still cannot see leaving them for major books like AO2e, however; it’s just too much work.)

I believe that there’s a market for inexpensive, small, single-topic books, as Cisco Routers for the Desperate is one of my best-selling books on Kindle. Not bad for a two-year-old book. I could also make the case that if you want a book called “Cisco Routers for the Desperate,” you want it NOW. (The publisher could probably double the price without impacting Kindle sales, but that would be gauche.)

I will do a community-based tech review. I’ll announce here when I’m looking for reviewers.

Finally: at Austin Hook’s request, I’ll also do a print-on-demand version. (I’m not terribly interested in distribution of physical copies of such a short book, but the expense and annoyance of setting up POD isn’t that great, so why not?) Austin handles the sale and distribution of the various OpenBSD merchandise, so I have confidence he can manage this.

As an independent, I cannot do pre-orders for books that don’t yet exist. There are Web sites that manage this for authors, but I really don’t want to bother. But Austin does that all the time, and he’s a trusted community figure. He will be the official source for pre-orders. While POD books need to be paid for in advance, the money doesn’t come to me until the books are ready to print and ship. This will get some extra cash in the hands of the folks who develop OpenSSH, which is a good thing.

No, I don’t have numbers or prices yet. I’m taking a course on this stuff, as well as POD, though, so I’m not even trying to figure that out yet. I can assure you that the paper version will be more expensive than the electronic version, however.

Will this succeed? There’s only one way to find out.

UPDATE: I’m not shelving AO2e for the summer. I normally work on multiple projects simultaneously. When I’m sick of one, I work on the other.

my .cwmrc

I need a window manager that doesn’t take up desktop space with lots of icons and permits me to work without removing my hands from the keyboard. (I do use the mouse, mind you, but I don’t want to be required to use the mouse for routine tasks.) I’ve used cwm on OpenBSD as my window manager for about a year, and it’s fulfilled my needs perfectly well.

I have made some changes. cwm is very easy to modify via the .cwmrc file. Here’s mine, with comments interspersed to show why I’ve made these changes.

I want a narrow blue border around my windows, and I want the active window to be highlighted. Maximized windows leave a 180-pixel gap at the right-hand side of the screen, so my xclock remains visible. (Without a clock, I sink into an authorial coma and lose awareness of time, sound, light, and so on.)

borderwidth 2
color activeborder blue
color inactiveborder darkblue
gap 0 0 0 180

My most heavily-used big applications are OpenOffice and Firefox. Both of these highlight text a word at a time when you use CTRL and an arrow key. By default, cwm maps CTRL-arrow to moving the pointer. I’m not going to train my fingers to make my applications work differently on OpenBSD than on any other operating system, so I needed to make cwm pass CTRL-arrow through to applications and use another key sequence for moving the pointer.

bind CS-Left unmap
bind CS-Right unmap
bind CS-Up unmap
bind CS-Down unmap
bind C-Left unmap
bind C-Right unmap
bind C-Up unmap
bind C-Down unmap
bind 4S-Left bigptrmoveleft
bind 4S-Right bigptrmoveright
bind 4S-Up bigptrmoveup
bind 4S-Down bigptrmovedown
bind 4-Left ptrmoveleft
bind 4-Right ptrmoveright
bind 4-Up ptrmoveup
bind 4-Down ptrmovedown

The 4th option key (which probably appears as the Windows key on your keyboard) and an arrow now controls pointer movement.

I experimented at length with my cwm configuration, and wanted to be able to make cwm reread .cwmrc with a keyboard shortcut, such as CTRL-ALT-r.

bind CM-r reload

I have a preferred screensaver, activated on a CTRL-ALT-DELETE.

command lock '/usr/X11R6/bin/xlock -mode flow'

Finally, I want to be able to start commonly-used programs via a right mouse click on the background. I don’t have to start them this way, but the option is convenient.

command firefox /usr/local/bin/firefox
command xpdf /usr/local/bin/xpdf
command OOo /usr/local/bin/soffice

With these modifications, cwm stays out of my way and lets me work.

And while I’m babbling about cwm: when you spawn an SSH session with CTRL-. you can put SSH command-line arguments before the hostname. Very useful for when you want to enable, say, X11 forwarding for a particular session.

IP Tables and VoIP

Here’s an iptables ruleset for a VoIP server with a Web interface. The goals are to allow management hosts to communicate with them freely, allow VoIP and HTTP(S) from the public, and drop everything else. It’s designed to be used as /etc/iptables.rules, and loaded with

# iptables-restore < /etc/iptables.rules

In Linux, you’re supposed to adjust the firewall at the command line. This implies an ability to retain the firewall ruleset in your head, as well as an ability to type correctly. Neither of these is true for me. My /etc/iptables.rules


*filter
#management
-A INPUT -s 192.168.0.0/16 -i eth0 -j ACCEPT
-A INPUT -s 10.0.0.0/8 -i eth0 -j ACCEPT

#Web interface
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

#VoIP
-A INPUT -p udp -m udp --dport 5080 -j ACCEPT
-A INPUT -p udp -m udp --dport 5061 -j ACCEPT
-A INPUT -p udp -m udp --dport 5060 -j ACCEPT
-A INPUT -p udp -m udp --dport 10000:20000 -j ACCEPT
-A INPUT -p udp -m udp --dport 1025:65534 -j ACCEPT

#keep state
-A INPUT -p tcp -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p udp -m state --state ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -j DROP

#allow outbound
-A OUTPUT -p tcp -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -p udp -m state --state NEW,ESTABLISHED -j ACCEPT
COMMIT

The section labeled “management” is where the rules allowing access from my management network goes. Management hosts may connect to this server on any port desired. Add additional lines for additional subnets.

The Web interface rules permit inbound HTTP(S) connections, and the VoIP section supports phone calls.

After working with iptables for a while, I feel perfectly qualified to say: I vastly prefer PF. Or even ipfilter. But now that I have the ruleset worked out, I can easily replicate it across all my VoIP servers.

OpenBSD, Firefox, and Flash

An OpenBSD advocacy article led me to a mailing list posting on how to get Flash playing transparently in Firefox on OpenBSD. You could get Flash (and other clunky media formats) to work on Unix-like platforms that Adobe doesn’t support for some time now, using a combination of players in packages and nearly random hacks in Firefox. This process makes everything simple, however.

I’m using 4.9/i386 and mozilla-firefox-3.6.13p3.

Set a package path in your shell. A few tests with ping and traceroute showed ftp3.usa.openbsd.org is my closest mirror. I use tcsh, so my .cshrc has:

PKG_PATH=ftp://ftp3.usa.openbsd.org/pub/OpenBSD/4.9/packages/i386/

Then run:

$ sudo pkg_add -r gecko-mediaplayer

This installs a whole bunch of packages for playing not just Flash, but other complicated media formats.

Browse to https://addons.mozilla.org/en-US/firefox/addon/flashvideoreplacer/ and install the plugin. The plugin handles replacing Flash with appropriate third-party players from packages.

Restart your browser. And everything just works.

FreeBSD iSCSI panic

I woke up today to find a console with:

panic: _mtx_lock_sleep: recursed on non-recursive mutex iscsi-io @ /usr/src/sys/modules/iscsi/initiator/../../../dev/iscsi/initiator/isc_sm.c:324

The initiator is a FreeBSD-current amd64 from 8 May 2011. The iSCSI target is an inexpensive iomega NAS. Other hosts attached to this iSCSI NAS have also had errors, though. The errors clear when I reboot the NAS.

Unfortunately, the FreeBSD box is a diskless system. Dumps aren’t exactly simple. While I heard some rumours about a network dump facility coming soon at the FreeBSD BSDCan devsummit, that’s the future.

How to fix this?

I attended the High Performance FreeBSD Clusters talk at BSDCan 2011. The presenter had originally used FreeBSD servers, then tried OpenSolaris to get better performance. He had OpenSolaris problems, but found that they could not access the bug information without a support contract. They’re now moving towards FreeBSD with EIT, and are happier.

I intend to learn from their mistakes, and replace the iomega with a FreeBSD EIT server. I’ll keep the iomega for, say, a central ports and packages NFS server, where a reboot won’t impact my uptime.

Why bother to blog this? So that the next poor bugger who gets this panic message gets at least one search engine hit.

BSDCan 2011

BSDCan 2011 was great. The problem with a conference that’s routinely great is that great becomes routine, and hence boring. Several presentations struck me as notably interesting for a variety of reasons, and I wanted to comment on three of them. These are only my personal opinions, of course. BSDCan had three tracks, and I could only be in one talk at a time.

Mark Linimon’s talk on How not to build a lights-out facility discussed the FreeBSD Project’s efforts to mirror its core infrastructure in datacenter space donated by New York Internet. As a chronicle of lessons learned and things that should be done differently next time, it’s valuable listening for anyone who thinks that building heavy-duty project infrastructure is easy.

I’m not going to name the people, the projects, or the code involved in the second talk, because the talk itself is less important than what happened during it. A committer from one large BSD project presented on a new piece of infrastructure he had developed. The audience included people associated with a variety of BSD projects. At the end of the talk, a senior developer from a different BSD project asked a few questions. The presenter and the developer had several rounds of completely civil back-and-forth technical discussion, and at the end the presenter agreed that the developer had some strong points and that some parts of his infrastructure needed additional work. I’m told that this happened in more than one talk. Despite discussion of disagreements between various BSDs projects, it’s clear that technical correctness is still most important.

The presentation I found most technically interesting was Randall Stewart’s work in data center congestion control. Stewart did real-world testing of data center congestion control with ECN and SCTP, and presented his results. It wasn’t until hours later that I realized exactly why I found the talk so interesting: he had essentially done “Mythbusters” for a specific part of TCP/IP. He’d bought a bunch of $50 servers on eBay, repeatedly adjusted SCTP’s response to packets with ECN set, and graphed the results. This was real-world stuff suspiciously close to academic research, done in a basement. And this sort of research is something that almost anyone could do. Lots of claims are made for our network stacks, but very few people actually experiment to measure performance with their workloads.

I’m glad to see open source projects learning lessons. I’m glad to see different BSD camps politely testing their ideas against each other, creating better software for everyone. But I’m really really happy to see real-world experiments.

I see all sorts of claims for different BSD’s network stacks, disk performance, and so on. Please, put them to the test. Make changes. Measure the results. While this work requires real hardware rather than virtualization, it’s something that anyone can do. You know your workload. Read about benchmarking. While naive benchmarks aren’t useful, it’s not that hard to design valid benchmarks. Buy used hardware, run your own tests. Make changes, and test again. Measure and document everything. Capture packets, and keep the pcap files so that you can go back and answer interesting questions. Publish your results. You’ll get interest. Perhaps your results will be as you expect. Maybe they won’t. But you’ll never know until you try.

As a BSDCan committee member, I would love to see more work like this. I can’t guarantee that your paper would be accepted, but I can say I’m much more likely to vote for a paper with a real investigation than yet another talk on well-understood features. Even if your results say “Yes, the fooBSD disk I/O system works exactly as expected,” it’s still interesting. And if you discover weak spots, you’ll have evidence the developers will need to improve performance.