Gleefully Malicious Books
FreeBSD Mastery: ZFS just went to the FreeBSD developer community for technical review.
I’ve also made it available for pre-order on the Tilted Windmill site.
This is the first of two books on ZFS.
I also plan a 4-book bundle once FreeBSD Mastery: Specialty Filesystems is done.
One of Solaris’ ZFS features is boot environments, where you can install multiple versions of the core operating systems and choose which one to boot into. Boot environments take a lot of the risk out of upgrades; if an upgrade goes bad, you can easily roll back to the previous version.
Boot environments are built on top of ZFS. Recent FreeBSD installers create fine-grained datasets rather than a single large dataset, which means you can leverage boot environments on FreeBSD.
Boot environments will go into FreeBSD Mastery: Advanced ZFS. But the first book needs to be boot-environment compatible–especially the chapter on manual ZFS installations. So here are my rough notes on boot environments, both from my own researches and from poking Allan Jude on Twitter.
If you’re going to play with boot environments, it’s best to start with a new system. You can migrate an existing system to a boot environment friendly one, but as Dan Langille discovered, it’s a right pain.
I started by installing a new 10.1/amd64 host. Your initial datasets look like this.
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot 465M 188G 128K none
zroot/ROOT 463M 188G 128K none
zroot/ROOT/default 462M 188G 462M /
zroot/tmp 149K 188G 149K /tmp
zroot/usr 570K 188G 128K /usr
zroot/usr/home 186K 188G 186K /usr/home
zroot/usr/ports 128K 188G 128K /usr/ports
zroot/usr/src 128K 188G 128K /usr/src
zroot/var 703K 188G 128K /var
zroot/var/crash 128K 188G 128K /var/crash
zroot/var/log 192K 188G 192K /var/log
zroot/var/mail 128K 188G 128K /var/mail
zroot/var/tmp 128K 188G 128K /var/tmp
1
This list is deceptive. The dataset zroot/usr exists, but it isn’t mounted. Go ahead, run mount(8) or zfs mount and check. I’ll wait. The dataset must exist for you to have child datasets like /usr/ports, but it doesn’t actually contain any data. The files in /usr that aren’t in specific child datasets like /usr/ports or /usr/src are all in the root dataset.
Run pkg install once to install the package management system, and then install beadm from packages.
# pkg install
# pkg install -y beadm
Now see which boot environments you have.
# beadm list
BE Active Mountpoint Space Created
default NR / 494.0M 2015-04-08 07:18
The only boot environment is named default. Under active, N means the environment is active now. An R means the environment will be active on reboot.
I need to upgrade this host to the latest version of FreeBSD 10.1, p9. This is where we need a new boot environment. I’ll name it after the release.
# beadm create 10.1-p9
Created successfully
# beadm list
BE Active Mountpoint Space Created
default NR / 646.0M 2015-04-08 07:18
10.1-p9 - - 10.7K 2015-04-08 11:43
Activate the new boot environment.
# beadm activate 10.1-p9
Activated successfully
# beadm list
BE Active Mountpoint Space Created
default N / 186.0K 2015-04-08 07:18
10.1-p9 R - 646.2M 2015-04-08 11:43
While the default environment has an N, indicating it’s active now, the 10.1-p9 environment has an R, so it will be active after a reboot.
Reboot. After the reboot, you’ll see the new environment is running.
# beadm list
BE Active Mountpoint Space Created
default - - 538.0K 2015-04-08 07:18
10.1-p9 NR / 646.3M 2015-04-08 11:43
Now I can install the latest FreeBSD patches without damaging my default system. If it fails, I can fall back by activating the default boot environment.
Remember how the zroot/usr partition isn’t mounted, so that files on /usr fall through to the root dataset. Your files in /usr and /usr/local are in the boot environment, so you can create new boot environments for package upgrades as well as base system updates. The /var partition works similarly.
Using boot environments opens another whole crate of weasels, though. Of course you want the core system databases in /var/db to be part of the boot environment–but what about /var/db/mysql? Similarly, postgres stashes data in /usr/local/postgres. You don’t want reverting the core operating system to give you an old version of the system database!
When using boot environments, create a new dataset for your database data. This also lets you tune ZFS for that application.
(PS: While you’re here, take a moment to admire how the header image changes every 10 seconds. I am immensely proud of myself for finding the button that made that happen. That’s, like, real web design! And for changing the density of the bookshelf photo so it looked okay in the header. That makes me a graphic designer too, right? Sweet!)
I’ve been busy producing words, just not blog words.
The short bits first. I hit 125K words in the first 3 months of 2015, putting me on track to break half a million words in 2015. While in the past I normally hit about a thousand words an hour, that required leveraging my day job as “research time.” I need to research during my work day now. My nonfiction speed is about 500 words/hour.
Recent releases include a new Montague Portal novella and Tarsnap Mastery. Both seem to be well-received, although neither has accumulated many reviews yet.
Sunday Morning Linux Review did a nice long review of Networking for Systems Administrators.
FreeBSD Mastery: ZFS is starting to take shape. The shape is unexpected.
For the first time, I’m working with a co-author. Allan Jude, one of the brains behind BSDNow and the original author of the FreeBSD Handbook’s ZFS chapter, is working with me to produce this book. He uses a huge amount of ZFS, and his knowledge and experience have been invaluable. Plus, he’s producing words.
This book won’t read like the Handbook, however. It will unquestionably be done in a light and engaging style, with an information flow designed for comfortable reading. It means I have to beat Allan. A lot. But at EuroBSDCon 2013 I demonstrated to Allan that I can kill him in five different ways with a butter knife, so he’s doing as he’s told. Sort of. Mostly.
(No, don’t come running up to me asking if you can co-author a book. This is a very special circumstance.)
We’ve had some unexpected developments along the way. ZFS is huge, and it includes features that we must cover but aren’t in corresponding books. A book on UFS wouldn’t need to cover rsync–but ZFS includes sending and receiving datasets! A book on UFS wouldn’t need NFS, but NFS is built into ZFS. ZVOLs and snapshots? While some BSD UFS’ have snapshots, ZFS snapshots are far more featureful.
All told, this book will run about 80,000 words.
The Mastery books are deliberately limited to 30,000-45,000 words. People tell me that’s about the right price point for that amount of knowledge.
In traditional publishing, when a book runs long you raise the price. That’s fair, assuming the book is worth reading.
Sadly, I can’t charge more than $9.99 for an ebook, thanks to Amazon’s daft pricing structure. I’ve written about this before, but it comes down to the fact that Amazon clearly has no interest in supporting highly-researched nonfiction with a limited audience.
Given the length, we’re splitting FreeBSD Mastery: ZFS into two books. The first book will be on basic ZFS, while the second will have more complicated ZFS information.
Now I just need to get Allan to stop writing like a sysadmin, and start writing so people will learn from his work. Hmmm… where did I leave that butter knife?
This year, BSDCan passed a critical threshold: we got too many good proposals. As a conference, this is an excellent problem to have. As a concom member, it kind of sucks. When your papers are mostly excellent, how do you decide which proposals to take and which to reject?
What follows is my highly personal take on BSDCan proposals and the evaluation procedure. It should not be taken as BSDCan concom gospel.
Generally speaking, a person who gets persuaded to present at BSDCan submits in later years. This is great–it means we don’t have to track them down again. It means we’re doing a good job at running an educational, friendly, fun, worthwhile event. Repeat attendees and speakers mean you’re doing something right.
In some previous years, we’ve had difficulty filling the conference. In prior years the convention committee did a lot of speaker hunting and trapping. So let’s talk about the committee.
When BSDCan started the BSD projects were fairly small. The handful of us could track what was happening everywhere. Over the last decade the BSD projects expanded. We needed people who keep up on the innards of these projects, and who could give perspective on papers from the project’s perspective. We have George Neville-Neil for FreeBSD, and Bob Beck for OpenBSD. We have Dru Lavigne, because she not only knows bloody everyone, she knows if people are decent speakers or not. David Maxwell originally gave us our NetBSD insight. He’s somewhat moved on from NetBSD, but he still works hard for the BSDCan committee, so we’re not about to let him go. The concom could use a leading NetBSD person to both judge NetBSD papers, and to help drag the NetBSD community into proposing more.
Dan Langille is the chairman, but most of his work is in the area of keeping the BSDCan servers and sites running, arranging physical space, and such. While he evaluates papers, he clearly expects the concom to make decisions on what gets included and what doesn’t.
There’s also this Lucas dude. I have no idea why they keep him around, as he’s far too self-centered to contribute much of any use.
Another part of each concom member’s role is to advocate for projects and proposals they believe in. I expect Bob to stand up for interesting OpenBSD papers. I expect George to pimp interesting FreeBSD papers. I expect both of them to be able to rank proposals from their posse, so that we can have a useful give-and-take and reach the best possible combination of presentations.
So how do we evaluate proposals? We use Pentabarf, which lets each concom member rate proposals, contacts submitters, and helps organize the schedule.
But the concom has to evaluate the papers. I’ve put some criteria that we use, in random order. Others apply them differently, or have additional criteria, but I think none of them actively disagree with mine.
There’s more to it than this, of course. I dug around trying to get more women to submit. I wouldn’t upvote a proposal because of the submitter’s gender, but I want to at least see them submit. After my efforts to let women know they’re welcome to submit, I was really pleased to see two very good kernel-related proposals from women. Having to say “we want women, but not you” would have really sucked.
So, after all that, let’s talk about proposals.
Having too many good proposals means that the average proposal quality increased. Proposals that would have been accepted in previous years based solely on the speaker’s reputation suddenly didn’t look so good next to the others.
I expect the proposals to continue to improve in the future. I’m writing this blog post so that some of our usual suspects have a useful model for writing good proposals.
Ideally, I would present both accepted and rejected talks here, and talk about why each met their fate. There’s problems with that, though. BSDCan doesn’t say up front that rejected talks will be made public. Organizations like USENIX say “we will do anything we want with your proposal,” but BSDCan doesn’t do that. (Whether we should or not is a separate discussion, and one the concom will have.) I went so far as to ask several rejected contributors if I could use their rejected proposals as examples. I’ve decided to not do that in this article, however. For one, it’s already too long, and I really need to generate some words on the ZFS book today. For another, negative criticism is both easy and less useful. Anyone can kick a dead dog. It’s much more useful to say “This was a great proposal, and here’s why.”
We have two separate sets of events: tutorials and presentations. On the tutorial side, we’ve learned the hard way that sysadmin tutorials attract more attendees than networking talks. The farther you go from a Unix command line, the fewer BSDCan people are interested. The attendance only supports 2 days of tutorials, in one room. This year, we got twice that many tutorial proposals. If you want to submit a tutorial, a half-day one is much more likely to fly than a full day one. We took Peter’s PF tutorial, because it’s well attended every year. Sysadmins do DNS, and DNSSEC is absolutely booming in some parts of the world. Despite my passionate argument against the doofus doing the FreeBSD storage tutorial, they took it. And Luigi’s netmap proposal was well done.
So, what’s in a good proposal? Here’s a couple examples I really liked. The details are not yet posted, but these will be public soon.
First, here’s “Expanding RDMA (Remote Direct Memory Access) capability over Ethernet in FreeBSD” by Shany Michaely. Mrs. Michaely has never presented at BSDCan before.
RDMA (Remote Direct Memory Access) is growing in popularity in Linux and Windows systems as a way to transfer large amounts of data with low latency and minimal involvement from the CPU. However RDMA InfiniBand drivers in FreeBSD were not updated, requiring users to create or port their own implementation of RDMA, and RDMA over Ethernet was not available in FreeBSD. This talk will describe how RDMA works and review the new addition of RoCE (RDMA over Converged Ethernet) network drivers in FreeBSD, allowing easier implementation of rapid data transfers with low CPU utilization over Ethernet and InfiniBand. This also enables the use of iSCSI over RDMA via the iSER (iSCSI Extensions for RDMA) protocol.
full description:
One of InfiniBand’s valuable capabilities is its support for RDMA (Remote Direct Memory Access) operations across a network, which enable rapid data transfer without involvement of the host CPU in the data path, and data placement to the responder memory without requiring its CPU awareness.
RoCE (RDMA over Converged Ethernet) is a standard for RDMA over Ethernet.
It provides true RDMA semantics for Ethernet and allows InfiniBand transport applications to work over an Ethernet network.
FreeBSD is frequently used for storage purposes and RDMA capability has a high potential of improving performance in such storage applications.
A good example for that is iSER (iSCSI Extensions for RDMA), a module being developed nowadays for FreeBSD, which enables the use of iSCSI over RoCE.
The main idea of this talk is a short overview of RDMA – Its principles, key components and its main advantages. Additionally, it will cover the use of RoCE – implementation architecture, obstacles we overcame in the development, and a quick browse of RoCE’s different capabilities and milestones.
So, what’s in this?
For a second example, here’s Matt Ahren’s proposal for “New OpenZFS features supporting remote replication.” Again, this will be public soon. Mr. Ahrens is a BSD conference veteran.
OpenZFS send and receive forms the core of remote replication products, allowing incremental changes between snapshots to be serialized and transmitted to remote systems. In the past year, we have implemented several new features and performance enhancements to ZFS send/receive, which I will describe in this talk.
Full description:
This talk will cover:
– Resumable ZFS send/receive, which allows send/receive to pick up where it left off after a failed receive (e.g. due to network outage or machine reboot).
– ZFS receive prefetch, which is especially helpful with objects that are updated by random writes (e.g. databases or zvols/VMDKs).
– ZFS send “rebase”, which can send changes between arbitrary snapshots; the incremental source is not restricted to being an ancestor of the snapshot being sent.
In this talk, I will cover the impact of these changes to users of ZFS send/receive, including how to integrate them into remote replication products. I will also give an overview of how zfs send/receive works, and how these enhancements fit into the ZFS codebase.
What do we have here?
So, here’s two good proposals.
While I don’t want to talk about specific rejected papers, I will touch on some mistakes people made.
One annoyance I had was people giving us multiple proposals. Don’t get me wrong–multiple proposals are good. They let us pick the talk that fits best with the conference. But the people who write good proposals sent multiple good proposals. They’re a pain to pick between!
I’m going to name Peter Hessler here, because he said I could.
Peter offered a wickedly excellent tutorial proposal on OpenBSD routing daemons, plus two good OpenBSD talks. Routing daemons are interesting to me, but BSDCan has learned that the further you get from the system, the fewer people attend the talk. We turned him down. That disappointed me, because I would have heckledattended. I hereby officially encourage Peter to submit that tutorial to a more network-related conference.
Of his talk proposals, we chose the most highly scored in Pentabarf. Now, Bob Beck is the OpenBSD rep on the concom. If he had said “No, we really want his other talk,” we’d probably go along with it. That’s a part of why Bob is on the concom.
We don’t accept BSDCan talks solely on the proposal grade in Pentabarf. If we had just said “Accept the top 40 talks, as rated by the concom in Pentabarf,” this would have been a very different conference, and many people would have been extremely unhappy with the results. But a strong proposal with a lot of information makes it easy for the concom to rate your proposal highly. If you’re willing to spend four hours making slides for your talk, spend a little time making a solid proposal for those slides.
I recommend creating an outline for your slides, and using that outline as the basis of your proposal. That’s what I do.
I can also say that the committee will be having discussions on how to cope with the “too many good talks” problem. If we’re lucky, this problem will only grow worse every year. I look forward to the day that we get so many excellent talks that we have to tell Kirk McKusick that we’re turning him down.
This post has gone on far longer than I intended. Thank you for reading this far, if you did. I hope that some of you who were rejected this year try again in 2016 and beyond, with proposals that make me say “oh, hell yes, we need this talk!”
To design “FreeBSD Mastery: Jails” I need to look at the existing jail management tools. Jails have been around about fifteen years now, and FreeBSD has accumulated a whole bunch of wrappers and supporting tools. Many of these have wound up in the ports collection.
Jails have evolved over the years. Some of these add-on tools are not useful for FreeBSD 9.1 and later.
Here’s a few things I discovered in my research. I’m hoping that you lot will offer your own comments and help me decide which tools to cover in the book.
It seems we have five major jail management toolkits.
The question for me is: which should I cover in the jails book? I’ll mention that all of them exist, but I can only give attention to one or two.
CBSD seems an obvious choice. It integrates CARP and HAST and vimage and just about everything. Plus, people like web GUIs. It seems to be the giant ape of jail management tools.
But I want to cover a command-line toolkit. Between ezjail, qjail, iocage, and jadm, I find myself leaning towards iocage.
There’s some other jail-related software in the ports collection. Here’s those I plan to investigate and possibly include. I might find that their functionality is now included in mainline FreeBSD, however.
Here are some jail-related ports I don’t plan to include, and why.
Welcome to 21 January 2015. Here’s the news.
I finished the first draft of Tarsnap Mastery yesterday. Today I went through it one more time, then shipped it off to Colin Percival for his comments on the last few chapters. Once I have his corrections, I’ll solicit reviewers.
Networking for Systems Administrators is at the copyeditor. It’s due back Friday. I expected this book to be pretty easy, but the tech reviewers savaged it. The end result will be a much better book, but it still wasn’t much fun and took longer to repair than I expected. With any luck, though, I’ll be able to get the electronic version out before the end of January and print in mid-February.
These two books are not available for pre-order through the Tilted Windmill Press web site. Books that I offer direct pre-orders on have done much more poorly than books without direct pre-orders. (Part of that might be topics, of course.) Other authors tell me that Amazon uses a book’s initial sales velocity to compute a book visibility to other buyers. More than 90% of my TWP book sales come from Amazon, so I care what happens on that platform.
The sequel to Immortal Clay is rolling along.
Now that Tarsnap Mastery and Networking for System Administrators are in a lull, I’ve started seriously pulling material together and filling in the outlines for my next FreeBSD Mastery books:
If you watched my most recent BSDNow interview, this is not a surprise.
The books will assume you know what’s in FM: Storage Essentials, and there is a certain amount of interconnection between all three. For example, to use jails you should know about devfs and unionfs. The Jails book will include the incantations to perform the devfs and unionfs tasks needed for jails, but the explanations for them will be in the Specialty Filesystems book. Similarly, the Jails book will have ZFS rituals in it, but the ZFS book will have the knowledge behind those rituals.
So, if you know some of these systems but not all you only need buy exactly what you need.
I might split the Specialty Filesystems book into two parts, one for local filesystems and one for networked filesystems, depending on how long the book gets and the final content. My goal for these books is to make them about 30,000-40,000 words.[1] FMSE was 45,000 words, and N4SA is about that long. The sudo and Tarsnap books are closer to 30K, while SSH is right around 35K.
I expect that once I finish them, I’ll have a bundle at Tilted Windmill Press much like the existing Security Bundle.
When will these be finished? I really want to take the finished print books to BSDCan in June.
[1] Why restrict book length? While the various ebook platforms do not restrict how long books can be, they do restrict how much I can charge for them. Most traditional publishers do not have that restriction. I must stagger roughly around that fuzzy intersection between “give good value,” “include what the reader needs,” and “can’t pay the mortgage.”
Note that the on-demand printers do limit the size of print books. The bindings at the larger size books are not great. I refuse to release shoddy print books.
Two new reviews on FreeBSD Mastery: Storage Essentials yesterday.
First, from Justin Sherrill, on DragonflyBSD Digest.
Then on Slashdot, which got featured on Reddit, from Saint Aardvark the Carpeted. (The nice thing about the Internet is, even a carpeted aardvark can make his mark on it.)
I’d like to thank both gentlemen for taking the time to review my latest. Reviews from disinterested third parties can make or break an author.
I just submitted a talk and a tutorial to BSDCan 2015.
If you’ve done anything in BSD technology over the last year more sophisticated than “maintained respiration and blood flow,” you should submit as well.
As a committee member, I can say we are always looking for unusual work. Yes, we all know that Henning Brauer is working on OpenBSD, and Luigi Rizzo is doing something clever with the network stack, and Max Stucchi is pimping IPv6. That’s all well and good, but we always want something outside our usual suspects. You’re the network administrator of a salt mine, or the frozen North, or an oil rig? You’ve wedged BSD into an alarm clock? Submit! Tell us about it!
Otherwise, they’re gonna let that Lucas character talk. Again. Nobody wants that.
The Sunday Morning Linux Review folks have a review of FreeBSD Mastery: Storage Essentials in show 141. The review starts at about 39:30, but the whole show is worth listening to. As always.
For my own reference, here’s a couple key quotes that I’ll probably use for marketing later. (I either write them down here, or have to go listen to the show again when digging up blurb quotes later.)
“Lucas lays a solid foundation about disks.”
“The devil is in the details, and the details are in the book.”
And when it comes to slices versus partitions: Mary is right. Listen to her, guys.
You can now get FreeBSD Mastery: Storage Essentials in print on Amazon.
If you buy the print from Amazon, you can get the Kindle version for $2.99. Sadly, that’s the closest thing to a proper print/ebook combo I’ve been able to do.
For completeness’ sake: you can also buy it directly from my CreateSpace store. As sales go, that’s where I make the most money. It’s also the most expensive version. If you want to pay extra so I make more I won’t object, but I will suggest you avoid the middleman and go straight to me.
It will appear in places like Powell’s in the coming weeks.
Now to finish “Networking for Systems Administrators,” complete the design on the next tech book, finish the outline for the Immortal Clay sequel, and finish outlining the intertwined morass that is my next three FreeBSD Mastery books.