Category: bsd

Posted on

FreeBSD: portmaster with pkgng

I recently tried FreeBSD’s pkgng, based on Ivan Voras’ blog post. Days after getting the new machine set up, though, I got this in my daily status mail:


Checking for packages with security vulnerabilities:
Database fetched: Fri Aug 3 03:02:57 EDT 2012
apache-2.2.22_5 is vulnerable:
Apache -- Insecure LD_LIBRARY_PATH handling

WWW: http://portaudit.FreeBSD.org/de2bc01f-dc44-11e1-9f4d-002354ed89bc.html

php5-5.4.4 is vulnerable:
php -- potential overflow in _php_stream_scandir

WWW: http://portaudit.FreeBSD.org/bdab0acd-d4cd-11e1-8a1c-14dae9ebcf89.html

Where did this come from? A bit of poking around my system leads me to /usr/local/etc/periodic/security/410.pkg-audit. My first question is, where did this file come from? pkgng includes an equivalent to the old pkg_info -W:

$ pkg which /usr/local/etc/periodic/security/410.pkg-audit
/usr/local/etc/periodic/security/410.pkg-audit was installed by package pkg-1.0.r4

pkgng gives you an audit of your packages in the daily mail. Excellent. It’s a DragonFly feature that I really like, and long overdue in FreeBSD.

So, how to upgrade the insecure ports? Unfortunately, I’ve had to build apache from source. I can’t use packages. That means I need to build the new version from ports. I get the new version ports tree with:

# portsnap fetch extract

I like to use portmaster for managing my ports. I’m told it works with pkgng. Let’s find out. First, tell ports that we’re running under pkgng by setting a variable in /etc/make.conf:

WITH_PKGNG=yes

Now install portmaster:

# cd /usr/ports/ports-mgmt/portmaster
# make all install clean

But portmaster doesn’t seem to work with pkgng:

$ portmaster -L
===>>> Root ports (No dependencies, not depended on)
===>>> 0 root ports

===>>> Trunk ports (No dependencies, are depended on)
===>>> 0 trunk ports

===>>> Branch ports (Have dependencies, are depended on)
===>>> 0 branch ports

===>>> Leaf ports (Have dependencies, not depended on)
===>>> 0 leaf ports

===>>> 0 total installed ports
===>>> There are no new versions available

A bit of research shows that portmaster needs to be patched before it works with pkgng. (Portmaster has a large install base, the portmaster maintainer is very careful about not messing up existing users, and pkgng is still very young. I’m confident that portmaster will work with pkgng by the time pkgng becomes the default.) You can get the patch on github.

Now build portmaster with the patch.

# cd /usr/ports/ports-mgmt/portmaster/
# make patch
# cd work/portmaster-3.13.13
# patch < $HOME/patch-portmaster-pkgng
# cd ..
# cd ..
# make all install

I keep the patch in my home directory, because any time I want to rebuild portmaster I must reapply the patch. And if the patch fails, I must check for a new patch.

Try the new portmaster:

# portmaster -L
===>>> Root ports (No dependencies, not depended on)
===>>> pkg-1.0.r4
===>>> New version available: pkg-1.0.r5_1
===>>> portmaster-3.13.13
...

Much better.

I’ll start by upgrading pkgng itself, then my other ports. I use:

# portmaster -d --no-confirm pkg

Use whatever portmaster options you prefer, of course. With the pkgng patch, portmaster seems to behave exactly as you expect. But the only way we’ll know for sure is if you test pkgng in your environment and file bug reports with the appropriate maintainer.

Posted on

BSDTalk #218, featuring… Me!

Will Beckman interviewed me at BSDCan. That interview is now available as BSDTalk #218.

Some of the issues I mention in the podcast are now solved. SSH Mastery is easily available in print in Europe. (You want the print copy as well as the ebook. You know you do.)

Posted on

FreeBSD Ports Annoyance

Ivan Voras’ article on FreeBSD’s pkgng prompted me to try pkgng. pkgng works exactly as advertised, with a couple of minor annoyances. But this brought to head a problem with FreeBSD that I’ve had for a while. I’ve talked to various ports guys about it over the years. It’s an engineering problem that’s begging for someone to solve.

Before folks in other Unix-like operating systems start snickering at “Lucas turning like a rabid dog on the community he came from”: you guys have your own problems. Go deal with them.

First, some context:

When the Ports Tree started it was pretty durn good. It might have started life as a temporary workaround, but it came out all right. Ports was the envy of many operating systems, and it got dragged into other BSDs and copied elsewhere. And in many ways, it’s actually aged pretty well. I run all three “Big BSDs” and a DragonFly box. I administer CentOS and Ubuntu machines. I have a couple of OpenSolaris servers slated to be replaced with FreeNAS. I feel qualified to offer an opinion on how things compare. FreeBSD trumpets its performance as a network operating system, as well it should. It passes packets like “whoa.” It makes lots of complicated stuff easy.

But some simple tasks are harder than they should be.

I’m standing up a new server to replace my old Web server. The old server originally ran 6-current, and has been updated to -current every few months. Now and then, Apache hangs up. I’m fairly confident that some lingering cruft from a previous aeon is responsible, and while I could exhaustively search the old server for the problem, I’d rather send it off to a well-deserved retirement and put some new hardware in play.

So, I have a virgin server. I want sendmail, mutt, and a WordPress server. (Your choices might differ, and that’s fine.) I don’t care which Web server WordPress runs on, anything vaguely modern would do. I’d like to install all software from packages, as I don’t have the time or energy to build an artisan server. Ideally, I won’t even need to install the Ports Collection. This is an extremely common set of software.

I can do this easily on OpenBSD, because OpenBSD has One True Web Server. It’s Apache 1.3, yes, which makes my bowels churn a bit, but the OpenBSD Project stands behind it, so: fine. I can do this with CentOS or Ubuntu or OpenSolaris — I have to install new repositories, which brings up all kinds of trust issues, but when you run those operating systems you decide to evaluate those risks on an ongoing basis. Fine. (I must confess, I’ve never installed a Web server on my NetBSD palmtop. That’s too geeky even for me.)

But I just can’t do this with FreeBSD.

The Ports System is massively flexible. If I’m willing to build from source, I can configure my system in dang near any way I like. That’s an awesome feature. It really is. The Apache 2.2 port has about eighty configuration knobs. I can tweak them all. I can build Apache exactly as I want and be highly confident that it will Just Work.

This is amazing.

But I can’t install a WordPress server with precompiled packages.

The problem is in the PHP port. PHP doesn’t know which Web server you’re running. The Ports System gives you the flexibility to build PHP with or without the Apache module. You can build with or without every dang module in the world. The PHP port doesn’t tie your hands.

But without these — comparatively tiny — bits of precompiled glue, I can’t make FreeBSD Just Work.

And it’s not just WordPress. I run an LDAP server at work. Apache 2.2 can be compiled to include the OpenLDAP module. But I cannot install this common corporate configuration from packages, I must compile the software myself. If I want OpenLDAP and Postgres, well, that’s a different build. And if an application absolutely requires MySQL, it’s back to the Ports Tree.

Multiply this by the thousands of combinations of options in over 23,000 ports, and… wow. Building a package for every possible combination would take a long time.

I can — and do — build my own package repositories. But this is an ongoing annoyance. And if I need an Apache 2.2 server that supports OpenLDAP, MySQL, and Postgres, I’m back to building it myself.

The Ports System makes the difficult manageable. I’d much rather maintain my own FreeBSD package repo than a CentOS or Ubuntu/Debian repo. But I really shouldn’t have to.

I’m not saying “copy Linux.” By all that’s sacred, no. NO. But they’ve solved this problem without building umpteen bajillion packages. We have some really brilliant people in the community. Surely there’s a realistic solution to this?

I pimp BAMP (BSD, Apache, Mysql, PHP) as an alternative to LAMP. This particular example of this issue stops those efforts cold.

Just in case you’re one of the brilliant people in our community, and this motivates you to do something, as a sysadmin who wants to run a lot more FreeBSD, here’s what I’d like to see:

  • Don’t rely on external package repositories. Every time I need to add a new repository to make some Linux package work, my blood turns to icewater and the voices in the back of my head start gibbering again. FreeBSD has a centralized model. I trust FreeBSD’s central package builds; they’re highly consistent, and if they get compromised, I will hear about it.
  • I’d like to be able to set variables in, say, /etc/make.conf or /usr/local/etc/pkg.conf that say what combination of software I use. Say, WEBSERVER=apache22 and DATABASE=postgres91. There are ports that will configure the ports-building process for me, such as ports-mgmt/portconf; I’d like something similar for packages.
  • When I install a new server, I want to be able to copy the config file in, run pkg add wordpress, and have the package system say “Aha! The Boss wants to run this webserver, and this database, but the software package doesn’t support his chosen database, so I’ll install MySQL anyway.” It should then just Do The Right Thing.

    • Why don’t I fix this? I’m a sysadmin, not a programmer. Hogs, when they look at my code, vomit. My spare time is spent writing, most often in support of the various BSD communities. I could learn to fix it, but I believe that others are better qualified to do it first. But those most likely to be able to fix it are also those most likely to just build the software themselves.

    We run operating systems to use applications. (If that wasn’t true, I would have truly mastered my Sinclair ZX80 by now.) The new user, confronted with a requirement to install the Ports Tree to build a simple PHP webserver, might well say “ick” and install something else instead. It’s a serious barrier to entry.

    So, if you’re looking for a new problem to solve, here it is.

    Posted on

    Writing New Editions

    This post is, “how is the new edition of Absolute OpenBSD coming along?” with a bit of musing on the craft of writing a second edition added in.

    I’m always shocked by the number of systems administrators ignorant of networking basics. I don’t care that they don’t know how to choose between BGP and OSPF, or that they don’t know what those acronyms stand for. That’s not relevant to most servers. But lots of them don’t know what an IP address is, or how to recognize a valid netmask, or the difference between TCP and UDP, or why there’s an /etc/protocols file. Any sysadmin who doesn’t know these things is still an amateur. My goal in writing a book is to drag people a little closer to professional.

    So, I include a chapter on networking basics in my introductory sysadmin texts, just like I include chapters on user management.

    I have a chapter on IPv4 networking in three published books. I’m writing this same chapter for the fourth time. I can’t just copy-and-paste from earlier editions. First, that would be rude. Second, my understanding of TCP/IP has changed in the last ten years, and that changes how I approach the material.

    But I can use the earlier efforts as models. Some text I can almost reuse, because it’s still the best way I know of to explain the specific topic. This will be the third time I use the dinner table analogy, for example. I still pass it through my brain to my keyboard, however, freeing myself to tweak a few words in the process.

    The most recent IPv4 chapter I wrote was for Absolute FreeBSD. In this incarnation, the chapter included a couple pages of basic binary and hexadecimal math. I looked at this, and thought “Why did I cover this? Doesn’t everybody know it?”

    Then I thought back, and realized that I included those pages because at the time I wrote the book, I spent a fair amount of energy teaching that material to my coworkers.

    I flipped back through the earlier editions of these books. Each book had one or more sections that I included because coworkers didn’t know it.

    At the moment, I’m not responsible for teaching anyone anything. I have no tech minions, and am molding nobody. It’s definitely changed my mind about what topics I cover. I suspect that the new edition of Absolute OpenBSD will contain less basic material. I’m definitely assuming that you know how to do binary and hexadecimal math, for one thing. This leaves room for more advanced topics.

    My conclusion seems to be: if you find the new edition of Absolute OpenBSD moves too fast, I suggest you get a copy of Absolute FreeBSD as well, or reread the one you have.

    (Mind you, the no-minions-to-mold is about to change. After two years of minion-free peace, I have been given a minion to mold. He’s on vacation at the moment, and has no idea what awaits him on his return. I have no desire to ruin his last few days of freedom, so we’re waiting for his first day back to tell him what he’s been sentenced to. The poor bastard.)

    So, where am I on the second edition of Absolute OpenBSD?

    If you want the minutia of my progress, search for the #ao2e hashtag on Twitter. But at a larger level, I’m writing the chapter on IPv4.

    This seems to be about 40% through the first draft of the book. The manuscript has proceeded quickly, now that I’m not moving into a house that needs work to be habitable. I’m hoping that this pace continues.

    I’ve received initial feedback on chapters 1-8 from Henning Brauer. Then the chapters go to Nathan Houle, my editor at No Starch Press, then back to me for corrections and discussion. Then they go to Peter Hansteen for formal technical review. Then back to me for correction. Then copyedit, back to me for correction, page layout, and back to me for correction.

    So, it’s not 40% done. The first draft is the hardest part, however. Doing the math, though, I see that I’ve been through an IPv4 chapter at least twenty times, given all the cycles. No wonder writing it is causing me nausea and chest pain.

    The original outline calls for a book about 400,000 words. For reference, Absolute FreeBSD is close to 300,000 words. This is too dang long. One of my goals is that my books be small enough to read in the bathtub.

    As SSH Mastery was successful, I have a resolution. I think I’ve figured out topics I can extract from the book and publish separately, without damaging the integrity of the book or its usefulness. Not everybody needs to know about, say, OpenBSD’s wireless features, but it’s certainly a topic worth covering. I can do small books on those topics and publish them as an aside, making the content available to interested readers. Assuming that the reader is a competent OpenBSD sysadmin (e.g., they’ve read Absolute OpenBSD 2nd ed or have an equivalent combination of education and experience) will let me do these books almost as easily as if it was integrated into the book. And my initial market research indicates that my readers are amenable to smaller, single-topic books.

    In summary: book is underway. More books coming.

    Posted on

    I’m in BSD Magazine

    The July 2012 issue of BSD Magazine has an article by yours truly: freebsd-update as an Intrusion Detection System.

    It also has a code to get you 30% off of Absolute FreeBSD at No Starch Press. If you don’t have your copy of this book, here’s your chance.

    It has other good articles too. None as awe-inspiring as mine, of course, but definitely worth a read.

    Posted on

    Death of a Web Server

    My first day at BSDCan, my Web sites died. Hard drive failure. The latest backups are defective. I think I’ve recovered the blog, but some links have changed, dang it. I’ll have to learn more about mod_rewrite to fix them. Web site is next. RSS readers will see some repeats, sorry.

    Other than that, BSDCan was awesome. As usual. In fact, it was just awesome as expected. So it was kind of routine. But still awesome.

    Posted on

    Absolute OpenBSD reviewers

    Lots of people have offered to tech review the second edition of Absolute OpenBSD before it goes to print. Peter Hansteen is doing the final tech edit, but I still need a reality check before it goes to him.

    Henning Brauer has offered to do this for me. He’s reviewed a few chapters already, and he’s caught a vast array of my inadequacies. I’ve decided to rely on Henning for fact-checking, rather than my usual volunteer community review process.

    This will probably disappoint those of you who really wanted to volunteer. I appreciate you more than words can say. But the community review process is the single biggest time consumer of my time in book production outside of actual testing. Henning certainly knows OpenBSD, is more than capable of catching my errors, and is highly motivated to have the book be accurate. (Or, if you prefer, he’d rather not have some doofus author misrepresent his hard work. I’m good either way.)

    I’m highly confident in their ability to point out every single mistake I make. If I make an error and both Henning and Peter miss it, well, then, apparently it was meant to be that way.

    Posted on

    Configuring OpenBSD to use RADIUS auth

    I have a love-hate relationship with RADIUS. RADIUS is the cheap white glue of authentication. Just about everything speaks it, so you can use it as cheap glue to unify passwords across your gear. But it’s a finicky protocol, with lots of edge cases, and those edges can be SHARP.

    Okay, perhaps it’s more of a tolerate-hate relationship. But still.

    OpenBSD supports using RADIUS to authenticate user accounts. Why would you possibly want to do this? For one thing, if you’re using authpf, it gives you a way to easily synchronize firewall passwords with your Windows domain through the Microsoft Internet Authentication Service. It’s not ideal — ideal would be making all of your users use public key auth — but it’s better than nothing, reduces your support workload, and helps convince management that your firewall is a real solution.

    I’m assuming you already have a RADIUS server. Mine is freeRADIUS-2.1.12, but any basic RADIUS server will work.

    OpenBSD uses BSD authentication, a competitor to PAM. The RADIUS auth process is documented in login_radius(8), but I’ll walk through the basics.

    Configure your RADIUS server to allow access from your OpenBSD box, and assign it a shared secret. My RADIUS server is 192.0.2.2, and my secret is the string Insubordinate. (It’s a lousy secret, but it’s just an example, and I’m tired.)

    Create a directory for the server list, and set the permissions as per the man page.

    # mkdir /etc/raddb
    # chgrp _radius /etc/raddb/
    # chmod 755 /etc/raddb/

    Create the file /etc/raddb/servers. List each server, and its secret, on its own line.

    192.0.2.2 Insubordinate

    Now switch your default authentication scheme in login.conf.

    #auth-defaults:auth=passwd,skey:
auth-defaults:\
        :auth=radius:\
        :radius-port=1812:\
        :radius-server=192.0.2.2:

    The change should take effect immediately. Be sure you have a window logged in as root beforehand, so you can switch back if necessary. If you have trouble, check your RADIUS server’s debugging log, or use a packet sniffer to examine the actual RADIUS dialog.

    Your host will now check every password against the RADIUS server. That includes root. You probably want to set up an auth-su class, or only have the authpf class use RADIUS, or some combination thereof. I really recommend requiring anyone who can get an actual shell to log in with public key authentication.

    And for those who care, you now know exactly what part of Absolute OpenBSD I’m writing.

    Posted on

    The Purpose of Tech Books

    I just got asked one too many times, “What’s in this book that’s not in the man pages?” And I’ve snapped.

    I’m blogging my answer, so I can point here and save myself from typing the answer again.

    I’m best known for writing about BSD technologies, a field where the developers are notoriously detailed in their documentation. If you look at the man pages for any open-source BSD, you’ll see that everything is included. If something is missing, it’s a bug. In addition there are extensive, lovingly-maintained FAQs and community-supported handbooks. How could I possibly add anything to than knowledge?

    The short answer is: integration and context.

    The man pages almost certainly contain everything you want to know. But man pages are not examples. Man pages do not provide context for the use of that knowledge. The ability to read disparate manuals and assemble that knowledge into a working, cohesive whole is a very specific skill. Programmers, in particular programmers who learn new technologies, have that skill. Many systems administrators develop that skill, after years of practice.

    Some people can take a whole pile of man pages, assimilate their contents, integrate that knowledge together, and create a holistic understanding of the field they cover. They can extrapolate from documents into use cases, and reverse-extrapolate from actual uses into configuration. If you are one of these people, I have two things to say to you:

    1) You do not need my books.
    2) You are smarter than me.
    3) By attempting to convince me of things I already know, you are wasting your own time.

    I also concede that many technology books are nothing more than recitals of man pages. Others are nothing more than collections of screenshots, saying “click the Next box” four hundred times. These books are a waste of electrons and wood pulp. I blame them for dragging down the reputation of technology writing. (I also writhe in envy because these books sell far, far better than mine. But that’s a separate issue)

    Good technology writing provides context for the information, and guides the reader to create a holistic understanding. Yes, some people can do that purely by reading man pages. Others need help.

    Why should I write a book that competes directly with, say, the FreeBSD Handbook or the OpenBSD FAQ? Not everybody learns in the same way. Discussing the same facts in different language, with a different organization, makes the knowledge take a different path through the reader’s mind. The reader’s job is to use new information to make new connections in their brain, and seeing the same information presented very differently can help.

    On a personal level, I do my best to make the job of getting that information easy, and present the reader with a whole bunch of ready-made connections.

    If you want me to listen to your proclamation of superiority, I have to say: put your money where your mouth is. Donate the list price of one of my books to an open-source project that I write about. If you feel the uncontrollable need to advertise your superiority, write “That Moron Lucas Is Wasting His Time” in the note field. Copy me on the emailed receipt. At that time I will pay attention to you, in direct proportion to the size of the donation. I won’t change what I do, mind you — I probably won’t even answer the email — but I’ll pay attention to you. And I promise you, the recipient project won’t mind.

    Update 5/2/2013: With the OpenBSD book coming out, I’m getting more of these. What really amuses me is that people think it’s important that I know the book is not useful.

    Posted on

    SSH course at BSDCan 2012

    I will be at BSDCan 2012. In addition to poking my nose where it doesn’t belong, spouting insouciance and irrelevance, and derailing important technical discussions with tediously pointless anecdotes about my pet rats, I am teaching a course on SSH.

    If you don’t have time to work your way through the SSH book, take a couple hours and and get dragged through it. The course will include material not included in the book, unless it goes too long, in which case I’ll spout off about extra material upon request.

    The new FreeBSD Device Drivers book should be out by then, and the author will be there. (Pity poor Joe Kong. He came to BSDCan in 2007, just after his Designing BSD Rootkits came out. I spent the rest of the conference badgering him into writing the FreeBSD Device Drivers book. He spent five years on this project, and is done at long last. I’m sure he’s ready to shout “Ha! Lucas, I’m rid of your stupid project! I’m a free man!” Sadly for Joe, I have another three books he needs to write. Preferably immediately. If he resists my persuasive powers, I’m prepared to use brainwashing techniques.)

    If you’re a reader of my blog, you need to go to BSDCan. We received so many excellent paper proposals this year that cutting the list down was physically painful. If you don’t yet have IPv6 up and running, attend the IPv6 tutorial. Or learn about PF. Not to mention two days of really interesting papers. And beer.