New reviews

There’s been a few new reviews out lately. First, two from Grant Taylor, on Sudo Mastery and SSH Mastery. Thank you, Grant!

Yesterday, a review of Sudo Mastery appeared on Slashdot. I haven’t been reviewed on Slashdot since Absolute OpenBSD came out. No, not the second edition–the original, in 2003. So this is cool. Thank you, “Saint Aardvark.” (Yes, I can figure out his real name, but if he goes by that, who am I to argue?)

As a result of these reviews, I now simultaneously have the #1 and #4 best-seller slots in Amazon’s Unix category.

I really want to thank everyone who takes the time to review my books — or, indeed, any books. Reviews drive sales. Sales mean that authors can afford to write books instead of washing dishes at the Burger Hut (which is all that most of us are qualified for in the real world). If you enjoy a book, and want to thank the author, take a moment to do so publicly.

And now back to writing more books…

2 titles in Amazon's top 10

2013 Failures and 2014 Goals

I set goals for 2013. And I failed to meet them. I promised three short nonfiction books, Absolute OpenBSD 2nd edition, and a novel. You got AO2e and two short nonfiction books, DNSSEC Mastery and Sudo Mastery.

While setting goals is important, exploring why you fail to meet those goals is just as important. Driving factors behind these goals boil down to three things.

  • These were pretty ambitious goals
  • Traveled to EuroBSDCon in September
  • January’s emergency appendectomy
  • I knew this was ambitious beforehand, but decided to try for it anyway. So, the first I accept as my own inability to realistically predict what I can do.

    I spent two weeks in Europe, both for EuroBSDCon and meeting with other writers and publishers. If I had to fly for eight hours one way (which I detest), and shift my body clock (which I find very difficult), I was going to make the trip worthwhile. But between preparing for teaching at EuroBSDCon, physical preparations for the trip, and recovering from the trip (both physically and real life), that cost me at least a month.

    You cannot predict something like an appendolith. That’s life. I didn’t merely have an appendolith, though. I had fever and infection and all sorts of horrible ghastly things. Proper recovery took months. Plus, general anaesthesia is insidious. Even when you wake up, it muddles your brain for weeks or months afterwards.

    When life derails your goals, you get back up as soon as you can and get back on track. Maybe you can’t complete the entire goal, but you can sure do a whole bunch of it. Or maybe the deadline slips into the next year. Whatever you do, you don’t quit.

    So: I failed.

    With those things in mind, let me set some goals for 2014. I already let part of this out at NYCBSDCon, so the rest of you might as well know.

    1) I will write at least three short nonfiction books. At least one will be on OpenBSD, at least one will be on FreeBSD. At least two will see print by the end of the year.

    2) Last year’s novel will get out of my house. A couple of my author friends are encouraging me to run the novel through a publisher and have offered introductions. Their faith in my work is sincerely touching. I’m inclined to self-publish, but am keeping an open mind. We’ll see what happens. (I waited to publish this list until I finished the first draft, for those who wonder.)

    3) I’ll write at least 120,000 words of fiction. (See FAQ 9.)

    4) I will not change time zones for a conference. EuroBSDCon was great, and I’m sure that the Sofia conference will be just as grand, but that kind of travel messes me up too badly to write. I’ll be at BSDCan, but this year I’m taking the train. Because I really, really abhor flying.

    5) I’m a candidate for my dojo’s red sash test this year. If selected, I will do my best to pass. This means much practice and sweat, as the test lasts several hours. For example, my green sash test included over four hundred falls. The falling isn’t bad, but getting up again gets pretty rough. The red sash test is worse.

    My deadline for these goals in February 2015. Because my birthday is in February. Using my personal year for goals always feels better than using the calendar year.

    In a more general sense:

    I’m starting a series of short FreeBSD books, each dedicated to a single topic. Which topics will I cover? Whatever I’m working with at the moment, that’s holding still long enough for me to write about it. For example, at this moment it doesn’t make sense for me to write a book about pkgng, because pkgng is developing quickly.

    Eventually, I’ll create enough FreeBSD content to “remix” into a big FreeBSD book, probably a 3rd edition of Absolute FreeBSD.

    The small books will use the 6×9 form factor, and all be about the size of SSH Mastery. People have taken well to this size of book at the $10 ebook/$20 print price point.

    This will also let me judge which material should go into a big book. If nobody buys, say, a small FreeBSD virtualization book, it’s clear I shouldn’t put that topic into a big book, because nobody cares.

    Ideally, I’ll be able to produce a slipcase for a complete collection of small FreeBSD books. At this time, I’m planning to give them themed covers based on old pulp magazines, minus the blatant sexism and racism. (It’s been suggested by more than one person that I keep both elements but make them funny. It CAN be done, just as it is possible to make thoughtful, incisive, and honestly funny jokes about any other painful or horrifying topic. But it’s extraordinarily hard, especially for someone who looks utterly “privileged white male.” I choose to spend my energy elsewhere.) But Beastie as a hard-boiled private eye, Beastie swinging on a vine through the jungle, Beastie as the flying ace, and so on? I think that’s going to look fantastic.

    What will the OpenBSD book be? I have three ideas. I’ve caught wind of other OpenBSD books in progress, however. I need to meet with my fellow BSD authors at BSDCan 2014 and hash things out with them. It’s very important that we not step on each other’s else’s projects, especially when it’s simple enough to avoid with five minutes at the bar. That’s why I won’t do, say, a pfSense book — Chris and Jim have that territory covered quite well. I’m confident that at least one of my three ideas will be free, if for no other reason than we don’t have that many OpenBSD authors.

    I expect to let the FreeBSD Foundation have books at cost for PBS-style donation prizes. “Donate $100, and we’ll send you this $20 book!”

    I have a clever idea for using the OpenBSD book to support OpenBSD. Theo and I discussed it briefly at EuroBSDCon. I don’t know if it will actually work, mind you. But worst case, they’ll have my book in the OpenBSD bookstore, with proceeds going to OpenBSD. (For anyone who is wondering, Austin Hook is very very easy to work with. The hardest part of getting books to the OpenBSD bookstore is figuring out how to cram all the shipping information onto the CreateSpace web form, which is certainly not Austin’s fault.)

    So, is this a cynical scheme to get you to give me more money? No… and yes.

    You’ll have the option to give me any amount of money you wish, from zero up to over a hundred bucks. There’s a couple people that I suspect will buy every book, in every version. I suspect others will get a few of the small books. Others will wait for a big book. Some will buy all the small books just so they can fill a slipcase. This is about options. It’s about getting content into reader’s hands as quickly as possible.

    But if you want to give me money, I’m certainly not going to argue.

    The good news is, I now know exactly what an appendolith feels like. The next time my appendix blows up, I’ll jump on it at the earliest possible moment. Why, just today I’ve felt three twinges that might have been a faulty appendix. Catching these things early is the key to quick recovery, after all.

    Crazed Ferrets in a Berkeley Shower – 2014 edition

    With Richard Stallman’s recent raising of the flag against LLVM and Clang, I’ve heard a lot of people talking about how the “FreeBSD people are whining about the GPLv3 terrorizing them.”

    Back in 2000, I wrote an essay for Linux.com about why I like the BSD license. It’s actually stood up fairly well to the test of time, but it’s fourteen years old now. Times have changed. So have licenses.

    Normally I object to taking something with warts, dosing it with Compound-W, and sending it back out. The structure of the original still holds up pretty well, but experience and history has added new information and context. Any new essay I wrote would look an awful lot like this one.

    So, rather than try to write out my opinions from scratch, I’m taking advantage of the Internet and updating my previous essay.

    Crazed Ferrets in a Berkeley Shower (2014)

    When USL sued Berkeley over BSD UNIX, it would have been easiest and cheapest for the university to drop the BSD tapes down a deep well and forget the Computer Science Research Group ever existed. Instead, combining the academic tradition with Berkeley’s well-known liberal spirit, they fought to give their work to the world. In a decision that has caused more arguments than most in the open-source software world, they released the code under the BSD license.

    For anyone who has avoided free software licensing for the last five years, the BSD license is very simple: give the authors credit in the source code, and don’t sue the authors if it breaks. There is no requirement to distribute any modifications to anyone. Alternately, the GPL version 2 requires that source code and any modifications be made available when modified code is redistributed. GPL version 3 has additional terms addressing patent indemnification, Tivoization, and more.

    Every week sees a new argument on some public forum about how the GPL is free, and the BSD license isn’t. Someone responds that the BSD license is more free than the GPL. Eventually, someone drags out the word “communist,” someone else fires up “corporate exploitation,” and all hope for rational discussion vanishes like free Jolt cola on the trade show floor. It’s better than mud wrestling or Jerry Springer.

    Anyone familiar with my work knows that I’m on the BSD side of the fence. The GPL is not a bad thing — as an author I would argue that it’s poorly written as either a software license or a political manifesto, but the whole “share and share alike” ideal is grand. We try to teach our children the importance of sharing. I live sharing. I try to do it myself.

    The people who use the BSD license want something very different than the people who use the GPL. The license has different goals and different motivations than those of the GPL. The end goals require different licenses.

    Let’s go back in time, to the early days of BSD.

    Why would a university, especially such a famously liberal university, struggle to turn a top-notch operating system over to corporations for their pillaging pleasure? BSD UNIX gave thousands of developers the chance to innovate on a solid infrastructure. It raised the bar for minimal acceptable performance in a computer system.

    An organization can take a chunk of BSD-licensed code, shrink-wrap it, and resell it. They might make money. But if that’s all that they do, any competitor with an enhancement can take over their market and destroy them. Without real improvements and innovation, BSD-licensed code isn’t as useful as you might think.

    On the other hand, an organization can use BSD-licensed code in specific components, free programmers, and improve their products.

    If you want to build, say, a networked fax machine you can a) build an operating system from scratch, b) invest in an embedded OS such as QNX, c) use GPL-licensed code, or d) use BSD-licensed code. Most programmers would find building an operating system interesting the first time, tedious the second, and mind-numbingly dull the third. And the organization has to pay for the programmers’ time while he does it. Purchasing an operating system also raises the product’s cost. While the GPL is slowly becoming more accepted, the thought of giving away code still gives most corporate lawyers a bad case of rotating heads and the pea-soup-spews.

    By using BSD-licensed code, however, the programmer can spend his time working on the fax machine interface. He isn’t reinventing the wheel yet again. Competitors must improve, or die. The cost to the organization is reduced. The end-users get a device that is cheaper and more reliable.

    The organization must do lots of work to make this happen. Developing hardware and software for a fax machine, or any other networked device, is no easy task. All that the BSD-licensed code does is make a few select parts easy.

    Today, Intel has network appliances based on BSD. Apple has OS X. So do IBM, Netflix, and a lot of other household names. Entire businesses and careers have been built on BSD-licensed code. They’re big names now, but many of them started with some guy, a basement, and BSD. If the University of California, Berkeley, had not opened their code base under such a non-restrictive license, none of this would be possible. Such intelligent appliances would be far more expensive for both manufacturers and consumers.

    Organizations with special code, or some other kind of secret sauce, do not want to release that code to the world. If they’ve spent millions researching how to shuffle bits in a certain way, they want to get their money back. They will not use a GPL code base that requires they release their code to the world. It’s just not going to happen.

    While organizations are not required to return code to BSD-licensed projects, many do. Why?

    The biggest reasons are cheap debugging, wide test base, and ease of merging future releases.

    Open-source developers are quick to point out flaws in code, especially in a project that focuses so much attention on correctness. They will happily point out your boneheaded mistakes.

    Open-source end-users frequently run some of the hardest-working machines on the Net, and put donated code through stress tests the original authors never intended.

    Organizations don’t have to return improments to their source BSD project, but it is in their best interest — for purely commercial reasons. Hardware changes over time, as do operating systems, and this bites organizations in the future. Here’s a true story about a company’s involvement with FreeBSD. I’ve anonymized it and changed some minor details so as to not embarrass them further. (If you were there and watched the fun, please don’t name them.)

    Company X produced a product based on FreeBSD 2. It did well. They built a thriving business. Their product required changes to the FreeBSD kernel. Some of these changes were key intellectual property, while others were improvements to related systems that anyone could have done.

    They kept all their changes as a single massive patch set.

    Time passed. Hardware changed. Company X found that FreeBSD 2 wouldn’t work well for their new hardware. They needed to move to FreeBSD 4.

    All of the improvements they made needed to be ported to the new version. Their “secret sauce” had to be rewritten for the new kernel — but so did the less important changes. In many cases FreeBSD developers had improved the systems that Company X’s code patched, but in a way that was totally incompatible with their designs.

    Company X realized that if they interacted with the FreeBSD community more and sent their non-proprietary patches back to the project, future merges would be simpler. They could implement the next generation of their proprietary stuff in a way that would fit the FreeBSD development roadmap.

    They knew this… but didn’t manage to execute.

    Time passed. They had to redo their entire forward porting process for FreeBSD 6.

    I don’t know how many times these poor bastards went through this, but I’m glad to say they’re now consistent and regular FreeBSD contributors. (Pain is a wonderful teacher, but nobody wants to go to his school.)

    For me, knowing that Company X “gets it” and donates code back to FreeBSD makes me look upon them much more kindly. I know other organizations have taken BSD code and not returned anything. They’re not required to, but they don’t stick in my mind as one of the good guys. I also chuckle when I imagine them merging their code forward.

    Some people fear that the BSD license allows companies to take their work and make it proprietary. Source code isn’t a limited resource; any number of people can have it, use it, or improve it. The two programs that hold the Internet together, BIND and Sendmail, had BSD licenses for many years. If a company could seize control of one or another of these programs, the Internet would be their oyster and we would be their slaves. People have tried. People have spent lots of money trying to execute this nefarious master plan. It hasn’t happened. It hasn’t happened because it can’t.

    Even if a Big Evil Software Company (tm., pat. pend.) wanted to assimilate BIND, they would have to convince the current BIND users to pay for something they could still get for free. Any vital, but proprietary, improvement to BIND will be duplicated quickly by open-source developers. A closed-source development process cannot withstand the onslaught of open source when it comes to vital core functions, especially when you start with the same code base. Those of us watching the race would have to content ourselves with pointing, laughing, and buying the volunteers beer.

    Look at TCP/IP. It was originally implemented in BSD. The BSD implementation became the standard. Everybody uses it. If you write your own stack, it’s expected to be compatible with BSD TCP/IP. Or everyone calls you a loser and says your IP stack sucks. Which it does.

    Closed source companies can and do compete on the bells-and-whistles front. And people are willing to pay for the pretty point-and-click interfaces. Heck, more than once I’ve snarled obscenities while trying to make Sendmail relay mail for a specific type of client, and if you asked me at the right moment I would have plopped down a credit card to make the problem go away.

    Now, users without source code will have a hard time fixing bugs or adding features themselves. Most organizations today have trouble finding people who can make reports off their Access databases, let alone people who could fix device drivers. Organizations who have that talent use it. Most organizations fix problems by asking the vendor.

    And no code, whether GPLd or BSD’d, can make up for an organization that does not respond to its customers. Starting with known working code can give them a head start, but a non-responsive organization still won’t go far. You probably gripe about Microsoft and Apple, but the average user wants an operating system where they can easily install a kitten screen saver with extra spyware. The Windows interface might not work well, but it is easy. The average corporate customers want simple more than they want performance.

    From the developer’s viewpoint, code under the BSD license provides a certain “minimal acceptable level” of software quality. Can you write an IP stack better than the one in OpenBSD? If not, why bother? Isn’t there something you’d rather do than handle exceptions for SYNs, ACKs, and RSTs, then return every six months to deal with the latest denial-of-service attack?

    Hundreds of organizations have made this same decision; the BSD IP stack has found its way into countless products. Almost any network device that isn’t labeled Linux or System V uses BSD. Some products even combine them; the RTEMS real-time operating system, which is distributed under the GPL, incorporated the BSD IP stack.

    Things could have been very different.

    Imagine someone travels back in time to the point where the Berkeley regents were choosing the license for their UNIX code. Our time traveler is a very persuasive man, and convinces several regents that they should use the GPL. Most of the others receive a cup of coffee spiked with Richard Stallman’s messenger RNA. The last holdout is distracted on the morning of the final vote by the half dozen crazed ferrets that somehow got in his shower. BSD UNIX receives the GPL.

    Consider how the GPL would have been received in the corporate boardroom of the 1980s. BSD UNIX never would have been used.

    Organizations frequently build software features such that they work. That’s very different from “they work correctly,” but that’s what we would have gotten.

    Imagine SunOS shipping without BSD’s vi; you’d have ed as the default editor. Imagine Digital UNIX with a “good enough” virtual memory system. SVR4 wouldn’t have had long filenames, or job control. The standard Unix File System is a BSD invention; SVR4 would have shipped with the S51K filesystem. A power failure would have meant hours of work using tools like fsdb and clri. If you’ve never heard of fsdb and clri, you don’t want to know about them. Trust me on this one.

    If all that didn’t make you flinch: imagine the toll in sheer human suffering if Microsoft hadn’t used the BSD TCP/IP stack.

    If Windows was the only competition faced by Linux, Linux would not have come so far. Today, the Linux and BSD camps work every day to outpace each other. We don’t know where that race will end. We do know that the software at the end of it will work.

    Today, the attitude towards open source software is quite different than it was in the 1980s. The corporate world accepts open source, and even the GPL. Many will not release their code under the GPL, but they contribute many improvements back to to the BSD world and from there to everyone.

    We can never be sure what the world will think tomorrow, or next year. After all, forty years ago everyone knew you needed source code to do anything useful. Twenty years ago, source code was irrelevant for many organizations. We’ve returned to the open source world, but the wheel might just keep turning through open source and back the other way.

    Think it can’t happen? Lots of people would like to make Vernor Vinge’s Ubiquitous Law Enforcement real. Many of them have a lot of money. There are alternative scenarios that would crush or marginalize open source operating systems, much as car tinkerers are a fringe group today.

    In 2020, or 2030, we might hear the words “Anyone remember the open source fad?” I don’t think it will happen, but history is littered with the rotting corpses of invulnerable juggernauts.

    Even if the open source movement collapses, BSD-licensed code will still be used. The open-source BSD groups will have support from people smart enough to recognize the benefits of open source, but who are hamstrung by organizational policy or legal problems.

    Average, everyday people, whose only interest in a computer is from 9 to 5, Monday through Friday, benefit. Imagine if your code could, every day, save one thousand people five minutes of annoyance, frustration, or downright hysteria, without them having to know or do anything. BSD-licensed code saves more than that, at no further cost to anyone.

    The BSD license is not about open source. Open source is simply a tool to get better software. It’s not about ideology. It’s about correct, stable, and above all, better software, not just for computer-literate individuals but for everyone. It’s about giving users peace of mind, and letting developers do original work.

    BSD is about making the world a better place. For everyone.

    updated 2014-01-31: corrected some GPLv3 terms

    Why I don’t have ads here

    Another “put this up so I can point to it later” post.

    Now and then someone tells me that I should put ads on my blog. Some of the articles get a ridiculous number of search engine hits, and I could probably add another (small) income stream there.

    I’m not morally opposed to the very idea of advertising-supported web sites or, indeed, advertising in general. I’m opposed to leaking information about my readers, however. And that’s what the advertisers actually pay for.

    I don’t oppose sharing information, mind you. If I could easily customize this blog so you only saw what you wanted without hooking you into the government/corporate exploitation network, I would. (It can be done, mind you, but it’s not trivial, so it’s not worth my time.)

    That’s why I let people send me money over at my online bookstore. I’d rather perform on a streetcorner with my hat in front of me than feed human flow data to anonymous information leeches.

    Of course, that raises the question of “why don’t you have a tip jar link on every blog post?” I write this blog to remind me what I did and to promote my work. If you can make use of either, fine — but that’s incidental to my purposes. Rather like OpenBSD, actually.

    Jan 2014 Java update broke me

    So I’m trying to upgrade my Ansible server to the newest OpenBSD snapshot, which involves working at the console. I go to my virtual server control panel, click on the link to the Java applet, and get told that Java won’t run this application.

    Turns out that Java has trusted self-signed certificates for applications until now, relying on blacklists rather than whitelists. I simultaneously applaud this move away from enumerating badness and condemn them for temporarily inconveniencing me.

    To whitelist a specific site, open the Java configuration applet. For Windows users, this is the Java Control Panel. Open the Security tab. About 2/3 of the way down, there’s an “Edit site list” option. Add the desired web site.

    Java will then run applets from that web site.

    Ansible and PF, plus NTP

    It seems that ntpd has turned into the latest DDOS amplifier. I run a lot of servers, and most of them use the standard ntp client. I need to verify that none of my servers can be used for DDOS amplification. To do this, I need to give all the clients a standard NTP configuration, pointing at my personal NTP servers.

    While my internal addresses need access to the port 123 on my servers, the public doesn’t. And I occasionally add internal addresses. Automating PF and NTP configuration via Ansible will simplify my life in the future.

    I’ve used Ansible templates to configure services before, but packet filters are a little different. Packet filtering rules involve lots of information about the local host, such as interface names and the various system roles. It’s entirely possible to write an Ansible template that expresses your PF ruleset, it just took a little work.

    First, I define an Ansible group for the NTP servers (as well as other server duties). The time servers run FreeBSD 9.2.

    Here’s the playbook, with added NTP.

    ---
    - hosts: ntp-servers
      user: ansible
      sudo: yes
    
      tasks:
    
      - name: enable ntpdate
        action: command sysrc ntpdate_enable=yes
      - name: enable ntpdate server
        action: command sysrc ntpdate_hosts=pool.ntp.org
      - name: enable ntpd
        action: command sysrc ntpd_enable=yes
    
      - name: copy ntp.server.conf to servers
        action: copy src=/home/ansible/freebsd/etc/ntp.server.conf
          dest=/etc/ntp.conf owner=root group=wheel mode=0644
        notify:
          - restart ntpd
    
      - include: tasks/pf-compile.yml
    
      handlers:
    
        - include: handlers/restarts.yml

    Simple enough, no?

    Except there’s nothing in here about the packet filter. Or restarting ntp.

    These functions are pretty common, so I’ve moved them to separate files. I might need to rebuild the packet filter rules for any number of playbooks, after all.

    The file tasks/pf-compile.yml looks like this.

    ---
    #build pf.conf from template
    
      - name: configure firewall
        template: src=/home/ansible/freebsd/etc/pf.conf.j2
          dest=/etc/pf.conf owner=0 group=0 mode=0444
          validate='/sbin/pfctl -nf %s'
        notify:
          - reload pf

    This task uses a jinja2 template to build a pf.conf specifically for this host, copies it to the host, validates its syntax, puts it in place, and triggers a PF reload. Always validate your files before deploying them. Ansible doesn’t prevent mistakes, but rather allows you to deploy mistakes faster than ever.

    Similarly, I’ve split the “restart services” handlers off into the file handlers/restarts.yml. Here are the relevant bits.

    ---
    #restart assorted services
    
      - name: restart ntpd
        service: name=ntpd state=restarted
    
      - name: reload pf
        action: shell /sbin/pfctl -f /etc/pf.conf

    So, where is this firewall template? That’s probably what dragged you here.

    #{{ ansible_managed }}
    #$Id: pf.conf.j2,v 1.2 2014/01/16 16:10:54 mwlucas Exp $
    
    ext_if="{{ ansible_default_ipv4.device }}"
    
    include "/etc/pf.mgmt.conf"
    include "/etc/pf.ournets.conf"
    
    set block-policy return
    set loginterface $ext_if
    set skip on lo0
    
    scrub in all
    block in all
    
    pass in on $ext_if proto icmp all
    pass in on $ext_if proto icmp6 all
    
    #this host may initiate anything
    pass out on $ext_if from any to any
    
    #mgmt networks can talk to this host on any service
    pass in on $ext_if from  to any
    
    #Allowed services, in port order
    
    {% if inventory_hostname in groups['dns'] %}
    #DNS access
    pass in on $ext_if proto {tcp, udp} from any to any port 53
    {% endif %}
    
    {% if inventory_hostname in groups['tftpd'] %}
    #allow tftp from the world
    pass in on $ext_if proto udp from any to any port 69
    {% endif %}
    
    {% if inventory_hostname in groups['ntp-servers'] %}
    #allow time from our networks
    pass in on $ext_if proto udp from  to any port 123
    {% endif %}
    
    #end of services

    The first bit of new (to me) trickery in this is getting the interface name. I use an Ansible-provided variable for this. Get the complete list of Ansible-provided variables for a host by running ansible -m setup hostname. The variable ansible_default_ipv4.device contains the network interface name. (If your host has multiple network-facing interfaces, you’ll need to modify this.)

    This PF ruleset pulls in two external files, one containing a list of management addresses and one containing the complete list of my internal networks.

    I allow access from my management networks, allow ICMP, default block, all the routine packet filter stuff. The next interesting bit is the allowed services. I check for the host’s presence in a group, and if it’s there’ I add a rule to permit the access that protocol needs.

    One detail that gave me trouble made me use inventory_hostname rather than ansible_fqdn or ansible_hostname to check group membership. I manage systems in several domains, and many of them have one name in our management systems and another in DNS. I put machines in Ansible by their fully qualified domain name. Using ansible_fqdn to get a hostname returns the hostname given in reverse DNS. inventory_hostname returns the hostname as it appears in the hosts file. If ansible_fqdn doesn’t match the hostname in the hosts file, the group comparison fails. Using inventory_hostname gave me one consistent set of hostnames for comparisons.

    So now I can easily deploy a secure NTP configuration to my servers. When I have to deploy some other service that requires updating the packet filter, I can include the same task file. And the handlers are now similarly reusable.

    Configuring the clients across several different operating systems will probably require Ansible roles, however. I’d best get on that next…

    Wanted: a VAX

    No, not for me. If I bring another piece of obsolete hardware into this house, it can have my chair because I’ll be sitting out at the curb.

    The OpenBSD Project builds all their packages on native hardware. Yes, it might take a month to build a complete package set on some of their platforms, but that’s okay.

    Their VAX recently died. They need a new one.

    I know that some of you have a VAX in storage, that you’ve been reluctant to surrender because it’s a cool toy. Now you can send it to a good home. You can tell everyone that you, personally, saved OpenBSD’s VAX support.

    How often can your pack rat tendencies make you a hero? This is your chance. Take it.

    The Desktop of Doom

    I’m sick of scrounging hardware for writing books. I’m sick of waiting for things to compile, managing disk space, and running out of memory. I finally got so sick of it that I decided to invest some serious cash in a research machine, in the hope that I wouldn’t need to hunt hardware piecemeal for the next five to six years.

    I solicited hardware advice from my modest horde of Twitter followers. After diving through realms of documentation, I came to realize that I didn’t want to build my own computer. I rarely configure, buy, or even touch hardware these days unless something goes unspeakably wrong, and there’s enough options out there that were I to attempt to do so, I’d waste both money and time. I certainly could figure it out, but that’s not where my strength is and the expertise I would acquire would be largely wasted outside this one task.

    I wanted eight hard drives, so that when the time is right I can write about assorted ZFS configurations. I wanted some flash disk on top of that. I wanted enough oomph to run umpteen virtual machines simultaneously. And I didn’t want to spend a whole lot.

    After wandering around the Internet, using assorted online system configuration tools to develop a rough idea of price, I contacted iX Systems and bought a system.

    I now have a shiny new desktop system that can “portsnap fetch extract” in two minutes. Because SATA3 is pretty sweet. My only problem is that the portable version of cwm that @blooouuup found for me is really slow & clunky when resizing windows, but I’m sure if I whine at the right people someone will fix it for me. (Now to figure out who the right people are.)

    For all those who asked me to share my shopping results: here’s a FreeBSD 11 dmesg. Thanks to bhyve, I can now run whatever I need. It lives under my desk, and is surprisingly quiet when I’m not building x11/xorg in 30 minutes or less. With WITNESS enabled.

    iX Systems might not advertise desktop systems, but they’ll do a nice one for you. Even if yours isn’t as awesome as mine.


    Copyright (c) 1992-2013 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 11.0-CURRENT #0 r260064: Mon Dec 30 16:28:06 UTC 2013
    root@grind.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
    FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
    WARNING: WITNESS option enabled, expect reduced performance.
    CPU: Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz (3700.08-MHz K8-class CPU)
    Origin = "GenuineIntel" Id = 0x306e4 Family = 0x6 Model = 0x3e Stepping = 4
    Features=0xbfebfbff
    Features2=0x7fbee3ff
    AMD Features=0x2c100800
    AMD Features2=0x1
    Standard Extended Features=0x281
    TSC: P-state invariant, performance statistics
    real memory = 34368126976 (32776 MB)
    avail memory = 33227055104 (31687 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: < >
    FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
    FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
    cpu0 (BSP): APIC ID: 0
    cpu1 (AP): APIC ID: 1
    cpu2 (AP): APIC ID: 2
    cpu3 (AP): APIC ID: 3
    cpu4 (AP): APIC ID: 4
    cpu5 (AP): APIC ID: 5
    cpu6 (AP): APIC ID: 6
    cpu7 (AP): APIC ID: 7
    ioapic0 irqs 0-23 on motherboard
    ioapic1 irqs 24-47 on motherboard
    kbd1 at kbdmux0
    random: initialized
    acpi0: on motherboard
    acpi0: Power Button (fixed)
    acpi0: reservation of 0, a0000 (3) failed
    cpu0: on acpi0
    cpu1: on acpi0
    cpu2: on acpi0
    cpu3: on acpi0
    cpu4: on acpi0
    cpu5: on acpi0
    cpu6: on acpi0
    cpu7: on acpi0
    attimer0: port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    atrtc0: port 0x70-0x71 irq 8 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    hpet0: iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 950
    Event timer "HPET" frequency 14318180 Hz quality 550
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
    pcib0: port 0xcf8-0xcff on acpi0
    pci0: on pcib0
    pcib1: irq 27 at device 1.0 on pci0
    pci1: on pcib1
    pcib2: irq 33 at device 2.0 on pci0
    pci2: on pcib2
    pcib3: irq 33 at device 2.2 on pci0
    pci3: on pcib3
    pcib4: irq 43 at device 3.0 on pci0
    pci4: on pcib4
    pcib5: irq 43 at device 3.1 on pci0
    pci5: on pcib5
    pcib6: irq 43 at device 3.2 on pci0
    pci6: on pcib6
    pci0: at device 4.0 (no driver attached)
    pci0: at device 4.1 (no driver attached)
    pci0: at device 4.2 (no driver attached)
    pci0: at device 4.3 (no driver attached)
    pci0: at device 4.4 (no driver attached)
    pci0: at device 4.5 (no driver attached)
    pci0: at device 4.6 (no driver attached)
    pci0: at device 4.7 (no driver attached)
    pci0: at device 5.0 (no driver attached)
    pci0: at device 5.2 (no driver attached)
    pcib7: irq 16 at device 17.0 on pci0
    pci7: on pcib7
    isci0: port 0xe000-0xe0ff mem 0xfa47c000-0xfa47ffff,0xfa000000-0xfa3fffff irq 16 at device 0.0 on pci7
    pci0: at device 22.0 (no driver attached)
    pci0: at device 22.1 (no driver attached)
    ehci0: mem 0xfbb23000-0xfbb233ff irq 16 at device 26.0 on pci0
    usbus0: EHCI version 1.0
    usbus0 on ehci0
    pcib8: irq 17 at device 28.0 on pci0
    pci8: on pcib8
    pcib9: irq 18 at device 28.6 on pci0
    pci9: on pcib9
    em0: port 0xd000-0xd01f mem 0xfba00000-0xfba1ffff,0xfba20000-0xfba23fff irq 18 at device 0.0 on pci9
    em0: Using MSIX interrupts with 3 vectors
    em0: Ethernet address: 00:25:90:db:d5:94
    pcib10: irq 19 at device 28.7 on pci0
    pci10: on pcib10
    em1: port 0xc000-0xc01f mem 0xfb900000-0xfb91ffff,0xfb920000-0xfb923fff irq 19 at device 0.0 on pci10
    em1: Using MSIX interrupts with 3 vectors
    em1: Ethernet address: 00:25:90:db:d5:95
    ehci1: mem 0xfbb22000-0xfbb223ff irq 23 at device 29.0 on pci0
    usbus1: EHCI version 1.0
    usbus1 on ehci1
    pcib11: at device 30.0 on pci0
    pci11: on pcib11
    vgapci0: mem 0xf9000000-0xf9ffffff,0xfb800000-0xfb803fff,0xfb000000-0xfb7fffff irq 16 at device 4.0 on pci11
    vgapci0: Boot video device
    isab0: at device 31.0 on pci0
    isa0: on isab0
    ahci0: port 0xf050-0xf057,0xf040-0xf043,0xf030-0xf037,0xf020-0xf023,0xf000-0xf01f mem 0xfbb21000-0xfbb217ff irq 18 at device 31.2 on pci0
    ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
    ahcich0: at channel 0 on ahci0
    ahcich1: at channel 1 on ahci0
    ahcich2: at channel 2 on ahci0
    ahcich3: at channel 3 on ahci0
    ahcich4: at channel 4 on ahci0
    ahcich5: at channel 5 on ahci0
    ahciem0: on ahci0
    pci0: at device 31.3 (no driver attached)
    pci0: at device 31.6 (no driver attached)
    pcib12: on acpi0
    pci255: on pcib12
    pci255: at device 8.0 (no driver attached)
    pci255: at device 9.0 (no driver attached)
    pci255: at device 10.0 (no driver attached)
    pci255: at device 10.1 (no driver attached)
    pci255: at device 10.2 (no driver attached)
    pci255: at device 10.3 (no driver attached)
    pci255: at device 11.0 (no driver attached)
    pci255: at device 11.3 (no driver attached)
    pci255: at device 12.0 (no driver attached)
    pci255: at device 12.1 (no driver attached)
    pci255: at device 13.0 (no driver attached)
    pci255: at device 13.1 (no driver attached)
    pci255: at device 14.0 (no driver attached)
    pci255: at device 14.1 (no driver attached)
    pci255: at device 15.0 (no driver attached)
    pci255: at device 15.1 (no driver attached)
    pci255: at device 15.2 (no driver attached)
    pci255: at device 15.3 (no driver attached)
    pci255: at device 15.4 (no driver attached)
    pci255: at device 15.5 (no driver attached)
    pci255: at device 16.0 (no driver attached)
    pci255: at device 16.1 (no driver attached)
    pci255: at device 16.2 (no driver attached)
    pci255: at device 16.3 (no driver attached)
    pci255: at device 16.4 (no driver attached)
    pci255: at device 16.5 (no driver attached)
    pci255: at device 16.6 (no driver attached)
    pci255: at device 16.7 (no driver attached)
    pci255: at device 19.0 (no driver attached)
    pci255: at device 19.1 (no driver attached)
    pci255: at device 19.4 (no driver attached)
    pci255: at device 19.5 (no driver attached)
    pci255: at device 22.0 (no driver attached)
    pci255: at device 22.1 (no driver attached)
    pci255: at device 22.2 (no driver attached)
    acpi_button0: on acpi0
    uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
    uart2: <16550 or compatible> port 0x3e8-0x3ef irq 10 on acpi0
    orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0
    sc0: at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    ppc0: cannot reserve I/O port range
    est0: on cpu0
    p4tcc0: on cpu0
    est1: on cpu1
    p4tcc1: on cpu1
    est2: on cpu2
    p4tcc2: on cpu2
    est3: on cpu3
    p4tcc3: on cpu3
    est4: on cpu4
    p4tcc4: on cpu4
    est5: on cpu5
    p4tcc5: on cpu5
    est6: on cpu6
    p4tcc6: on cpu6
    est7: on cpu7
    p4tcc7: on cpu7
    ZFS filesystem version: 5
    ZFS storage pool version: features support (5000)
    Timecounters tick every 1.000 msec
    random: unblocking device.
    usbus0: 480Mbps High Speed USB v2.0
    usbus1: 480Mbps High Speed USB v2.0
    ugen0.1: at usbus0
    uhub0: on usbus0
    ugen1.1: at usbus1
    uhub1: on usbus1
    uhub1: 2 ports with 2 removable, self powered
    uhub0: 2 ports with 2 removable, self powered
    ses0 at ahciem0 bus 0 scbus7 target 0 lun 0
    ses0: SEMB S-E-S 2.00 device
    ses0: SEMB SES Device
    da2 at isci0 bus 0 scbus0 target 2 lun 0
    da2: Fixed Direct Access SCSI-5 device
    da2: Serial Number WD-WCAW36477223
    da2: 300.000MB/s transfers
    da2: Command Queueing enabled
    da2: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
    da0 at isci0 bus 0 scbus0 target 0 lun 0
    da0: Fixed Direct Access SCSI-5 device
    da0: Serial Number WD-WCAW36478143
    da0: 300.000MB/s transfers
    da0: Command Queueing enabled
    da0: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
    da3 at isci0 bus 0 scbus0 target 3 lun 0
    da3: Fixed Direct Access SCSI-5 device
    da3: Serial Number WD-WCAW36477062
    da3: 300.000MB/s transfers
    da3: Command Queueing enabled
    da3: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
    da1 at isci0 bus 0 scbus0 target 1 lun 0
    da1: Fixed Direct Access SCSI-5 device
    da1: Serial Number WD-WCAW36498185
    da1: 300.000MB/s transfers
    da1: Command Queueing enabled
    da1: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
    ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
    ada0: ATA-8 SATA 3.x device
    ada0: Serial Number WD-WCAW36478129
    ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
    ada0: Command Queueing enabled
    ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
    ada0: Previously was known as ad4
    ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
    ada1: ATA-8 SATA 3.x device
    ada1: Serial Number WD-WCAW36477352
    ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
    ada1: Command Queueing enabled
    ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
    ada1: Previously was known as ad6
    ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
    ada2: ATA-8 SATA 3.x device
    ada2: Serial Number WD-WCAW36477139
    ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada2: Command Queueing enabled
    ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
    ada2: Previously was known as ad8
    ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
    ada3: ATA-8 SATA 3.x device
    ada3: Serial Number WD-WCAW36477141
    ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada3: Command Queueing enabled
    ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
    ada3: Previously was known as ad10
    ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
    ada4: ATA-8 SATA 2.x device
    ada4: Serial Number 20131111AA300000000E
    ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada4: Command Queueing enabled
    ada4: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
    ada4: Previously was known as ad12
    ada5 at ahcich5 bus 0 scbus6 target 0 lun 0
    ada5: ATA-8 SATA 2.x device
    ada5: Serial Number 20131111AA3000000007
    ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada5: Command Queueing enabled
    ada5: 61057MB (125045424 512 byte sectors: 16H 63S/T 16383C)
    ada5: Previously was known as ad14
    ugen0.2: at usbus0
    uhub2: on usbus0
    ugen1.2: at usbus1
    uhub3: on usbus1
    Netvsc initializing... done!
    SMP: AP CPU #1 Launched!
    SMP: AP CPU #3 Launched!
    SMP: AP CPU #4 Launched!
    SMP: AP CPU #2 Launched!
    SMP: AP CPU #6 Launched!
    SMP: AP CPU #7 Launched!
    SMP: AP CPU #5 Launched!
    Timecounter "TSC-low" frequency 1850039224 Hz quality 1000
    WARNING: WITNESS option enabled, expect reduced performance.
    GEOM_RAID: Intel-ee147409: Array Intel-ee147409 created.
    GEOM_RAID: Intel-ee147409: Disk ada4 state changed from NONE to ACTIVE.
    GEOM_RAID: Intel-ee147409: Subdisk Volume0:0-ada4 state changed from NONE to ACTIVE.
    GEOM_RAID: Intel-ee147409: Disk ada5 state changed from NONE to ACTIVE.
    GEOM_RAID: Intel-ee147409: Subdisk Volume0:1-ada5 state changed from NONE to ACTIVE.
    GEOM_RAID: Intel-ee147409: Array started.
    GEOM_RAID: Intel-ee147409: Volume Volume0 state changed from STARTING to OPTIMAL.
    GEOM_RAID: Intel-ee147409: Provider raid/r0 for volume Volume0 created.

    Jailing FreeBSD 4 on FreeBSD 10

    We have an in-house application that was written for FreeBSD 4 and antediluvian versions of PHP, Perl, OpenSSL, and so forth. Most of the features have migrated into other applications, but a few critical functions remain.

    An old operating system isn’t sufficiently bad, though. The hardware terrifies me. Not only is it over a decade old, it’s repurposed desktop hardware.

    Virtualize it? Maybe. But device drivers have changed over the intervening decade, and a ten-year-old de(4) or fxp(4) driver works poorly on any of my virtualization systems. Virtio is right out.

    Port it to a current OS, PHP, and perl? That would be a painful prospect if I knew what I was doing. I’m a sysadmin, not a programmer. I have no bloody clue what I’m doing.

    But theoretically, FreeBSD 4 systems should run almost unchanged in a jail on FreeBSD 10. Can they? Let’s find out!

    First I tarred up the entire 4.10 system, except for /proc. (Yes, FreeBSD 4.10 used /proc. Those were the days, eh?) I did no preparatory cleaning, and even included port work directories, /usr/obj, /var/tmp, and so on, as I have NO idea what I might need. Yes, I’m sure I can find a PHP 5.0.whatever tarball out on the Internet, but that would involve work.

    Create my jail directory, and untar the copied server

    # mkdir /var/jail/oldserver
    # cd oldserver
    # tar -xvpf $HOME/oldserver.tgz

    Be sure to use the -p flag, to preserve permissions.

    Now to edit some configuration files, to change this system from hardware to jail.

  • Check rc.conf. Remove functions handled by the host server: firewalls, timekeeping, SSH, and so on. This jailed host only needs the functions that directly support the application (the Apache web server and the associated MySQL database).
  • Check /etc/fstab for anything you need to retain, like NFS filesystems or /proc. Comment out everything else.
  • Remove the old /dev. You cannot use FreeBSD 4 device nodes on a FreeBSD 10 host. (I spent an hour repeatedly bashing my head on the brick wall intensively testing these two sentences, so you can rest assured that it’s true.)

    Now on to the jail server. FreeBSD 9 and above has a jail-specific configuration file, /etc/jail.conf. I define some basic characteristics here.

    While that’s running, do some basic jail setup. I would normally recommend ezjail, but this is a fairly special case: no ZFS, the server will never be upgraded, and I specifically want a very minimal system. Also, ezjail doesn’t seem to have been updated for the Jail New World Order. So I’ll configure this the hard way, which isn’t terribly hard.

    FreeBSD 9 and above has a jail-specific configuration file, /etc/jail.conf. Here’s a configuration for my old server.

    exec.start = "/bin/sh /etc/rc";
    exec.stop = "/bin/sh /etc/rc.shutdown";

    oldserver {
    path = /var/jail/oldserver;
    host.hostname = oldserver.mwlucas.org;
    ip4.addr = 10.0.16.31;
    interface = vtnet0;
    };

    Now for the real test:

    # service jail start oldserver
    Starting jails: oldserver
    #

    Is it that easy?

    Of course not. Several processes didn’t start. I had to edit some configuration files to account for the change of IP address. That’s pretty minor, though.

    To log into the jail, get the jail ID and run a shell prompt in that server.

    host# jls
    JID IP Address Hostname Path
    7 10.0.16.31 oldserver.mwlucas.org /var/toolkit
    host# jexec 7 /bin/tcsh
    oldserver#

    The advantage to hosting this ancient system as a jail is that I can use the host operating system to control access to the legacy application. I don’t have to use the ancient IPF included on FreeBSD 4.10; I protect this host with PF from FreeBSD 10. (Admittedly, there’s newer PF out there, but even the older version in FreeBSD 10 is better than IPF.) FreeBSD 4.10 includes OpenSSH 3.5, which has roughly the same security as Windows XP. To log into the old server now, you must authenticate to the up-to-date FreeBSD 10 system and get a jail command prompt.

    The disadvantage to this system is that you can’t run ps(1) inside the jail. The ps command reads kernel data structures. While the FreeBSD 10 kernel includes FreeBSD 4 compatibility, that compatibility doesn’t extend to ps(1) and similar commands. You must use ps(1) from the jail host.

    That’s an acceptable trade-off, if it means I don’t have to touch actual PHP code. That stuff has cooties.