Blog

Posted on

FreeBSD security report on successful logins

By default, FreeBSD sends a daily security report listing all sorts of good stuff, and failed logins.

I don’t care about poorly-programmed password gropers fumbling at a service that doesn’t accept passwords. I don’t want to read pages of stupidity. Over the years I’ve trained myself to skip the stupidity, which is bad practice. If I get automated email it better contain only things I care about.

I care about successful logins. The number of folks who log onto my hosts is minuscule. I want to skim a short list of logins, recognize them all, and move on with my day.

I’ve trivially modified the failedlogin script to recognize successful logins. No, I’m not going to put this on github. I quit using github several years ago.1 Drop it into /usr/local/etc/periodic/security and enable it in /etc/periodic.conf.local.

security_status_loginfail_enable=NO
security_status_loginsuccess_enable=YES

This only catches SSH logins, though. If anyone has suggestions for improving the regex catching assorted logins for the services you use, I’m open to it.

Will I submit it as a PR? Uh, maybe? Depends if anyone cares.

Posted on

Original N4SA2e cover art for sale!

The original cover painting for Networking for System Administrators, 2nd edition is on sale at my bookstore.

Minimum price is $600, but I’m sure Eddie would appreciate a couple extra bucks if you’ve got them.

I make nothing on this. Your entire payment (minus processing fees) goes to Eddie and he ships you the painting.

There’s only one, so grab it quick if you’re interested.. As an aside, this is the first time I’ve enabled stock management on my store. Supposedly this will disappear immediately when someone buys it, but I’ll be watching on the off chance it doesn’t.

114: 98% Free from Arthropod Infestation

My new FreeBSD Journal column just went to the editor.

Oh AWOL, my succulent summer child,

You’re the latest model of Human(tm), a digital native, theoretically enlightened by access to the collected wisdom of our species. Are your sources of wholesome, cogent, and actionable advice so limited that reaching out to a grizzled sysadmin seemed wise? True, this is an advice column. I offer advice. I also offer trepanning and experimental epidemiology but you don’t see people queuing for either of those do you? The Internet has Captain Awkward and Doctor Nerdlove and Dear Prudence and Ask A Manager and Dear Abby and you decided that no, you want to live abhorrently so you’re beseeching a grant of “wisdom” from a system administrator known for enhancing rat operated vehicles with nitrous oxide boosters. If you’re from a civilized country you even have health care so you could access counseling, or perhaps I should say intensive counseling because a few minutes into your first session the therapist would hit the big red button and you’d have a comfortable new home with three hot meals a day, your own bed guaranteed 98% free from arthropod infestation, and a meticulously personalized medication regimen because, after all, you think that sysadmins dispense sensible advice for a better life.

For more of my ill-advised advice, grab a copy of Dear Abyss.

Posted on

Fundraising Auction Over

The fundraising auction is over.

Seth Hanford won with a bid of $777. I wouldn’t normally declare 777 to be a suitable solution to any problem, but I am compelled to sign off on this one.

Seth, send me your receipt and you will get your pig in a poke–uh, your prize. I don’t care which of the suggested charities you donate to, they’re all worthy.

By the way, spectators and other bidders: the anonymous person who bid $550 also donated their bid to charity. If you felt like doing the same, they’d appreciate it.

113: Destroying Performance

Talking pools in OpenZFS Mastery, which means yet again talking about blocks and sectors and alignment.

GPT partitions fill a number of sectors. If you partition a drive assuming 512-byte sectors, you can easily create a partition that would not cleanly start and end on 4K sector boundaries. Take a drive that has 4K sectors, but claims to have 512-byte sectors. Create a partition that fills, say, seven 512-byte sectors, then add another partition that fills the rest of the drive. Make a ZFS pool with 4K blocks on that large partition. Each filesystem block touches two physical sectors. The misalignment causes write amplification, destroying performance. For average spinning drives, assuming that all your disks use 4K physical sectors is safest. Certain SSDs also expect partitions to be aligned along 128 KB or 1 MB boundaries. Some enterprise drives use 8K sectors, however, and a handful of specialty devices even larger.

Avoid alignment problems. Make all GPT partitions begin and end on megabyte boundaries.

My whole stupid career is built on filesystems. You can still get your name in this book, though.

Posted on

Kansas or Minneapolis Fundraiser

My country is in trouble, and I’m just a tiny rat dude making a marginal living writing about stuff very few folks care about. Not much I can do directly.

My 1 April Kickstarter required some unusual tests, so I produced a physical artifact of interest to a certain subset of sysadmins. I have that artifact. It’s a real thing. I have to make a couple tweaks and add tidbits here and there, but the thing exists.

You want proof it’s real? Here you go!

Mildly redacted to preserve the surprise. Also, this photo is legitimately a hint.

Folks have started asking what it is. I’m not saying until the Kickstarter launches.

I am, however, auctioning off the prototype for charity.

Yes, I’m asking folks to give money for a sysadminny thing sight unseen. The money doesn’t go to me, however! I want you to support a worthy cause. I have a choice of worthy causes: supporting Minnesota folks trapped by ICE, or helping trans folks in Kansas.

What do you get?

When you send me the receipt for your donation, I will mail you the thing and send you a Kickstarter preview link. I will also include a letter declaring that your direct financial support of charity grants you moral superiority. Once the final product escapes, I’ll ship you one of those. So really, the physical good will be unique for only a few weeks.

Intangibly, though? Ah, the intangible benefits! BRAGGING RIGHTS. You’ll know what the thing is before anyone else! You’ll have grounds to call me a dumbass before anyone else! People will call me a dumbass anyway, but your comments will be evidence-based and thus folks will take you seriously.

I will pay for standard Priority Mail shipping. That’ll be three days within the US, and a couple weeks overseas. (If you’re outside the US and want fast shipping, I’ll ask that you send me a few bucks for the upgrade. Sorry, but overnight to Germany or Australia ain’t cheap.)

All I ask is that you don’t ruin the surprise for folks. If you blab, I can’t do much. Sure, I’ll never do a fundraiser like this again and I’ll call you a jerk, but that’s about it. Unless you live within wedgie distance.

I am, of course, perfectly fine if you post something like “holy crap Lucas is a jerk this thing is a total ripoff like I’ve never seen before don’t you dare go to https://mwl.io/ks and follow it you’ll only encourage his next lame travesty.”

Bid by leaving a comment on this page.

The auction runs from now until 5PM EDT on 9 March 2026. If the bidding goes nuts in the last few minutes, I’ll leave it open until it settles down. There’s no sniping this auction at the last moment, as I want bids to escalate beyond all sensible limits.

The winner gets to pick a charity off of https://www.standwithminnesota.com/ or Trans Continental Pipeline and donate their bid. Send me the receipt and I’ll send you the thing. I want you to be able to enjoy knowing the secret for as long as possible, so I’ll ship it ASAP.

Bid early! Bid often! Bid to be the first one disappointed!

112: A Special Uberblock

OpenZFS Mastery is staggering along. Here we talk about how ZFS maintains uber-integrity.

Not having dedicated special index blocks sounds great, but every data tree needs a root. ZFS stores a pointer to the filesystem root in a special uberblock. Every pool has a queue of 128 uberblocks stored at algorithmically-predictable locations. In keeping with the copy-on-write design (Chapter XXX), uberblocks are never edited. Every time a new transaction group gets written to disk, ZFS records the new root information in the next uberblock in line. When the 128th uberblock is used, ZFS loops back to the beginning. At boot, the system searches for the uberblock with the highest transaction group number and uses that to find the pool’s root. If the newest uberblock appears damaged or incomplete, ZFS falls back to the newest usable uberblock. A badly timed failure might cost you the latest transaction group’s worth of data, but the pool itself will be coherent and will not require an integrity check.

OpenZFS Mastery is still open for sponsorship.

111: Artifically Prolonged, Unnecessarily Stressful

Here’s OpenZFS Mastery on physical labeling. I have strong feelings on this.

Develop a consistent naming and numbering scheme for your storage arrays, and use it dogmatically. Many storage arrays have a standard naming scheme, often printed on the equipment. If your equipment already has numbered shelves, use that numbering. Otherwise, make simple rules like “shelf 0 is always at the top and disk 0 is always at the left.” You might use the prefix “f” for the front and “b” for the back, or whatever works for you.

Record the serial number of each drive as you install it in the array. Physically label each drive tray with the drive serial number and physical location. Use good labels that remain stuck over years of being ignored in a dry, dusty datacenter. Yes, this is tedious—but when a drive fails you must have this information. You can do this work in peace and quiet at your own pace, or you can desperately rush through it during an artificially prolonged, unnecessarily stressful outage.

OpenZFS Mastery is open for sponsorships.

110: Resorting to Extraordinary Means

Work is underway on OpenZFS Mastery.

ZFS can run on anything the operating system presents as a block device. The most common are disks. Spinning rust, SSD, NVMe? Sure. Virtual disk files stored on another filesystem? If that’s what you’ve got, ZFS will cope with the extra overhead and with the right settings can still protect your data. USB flash drives? The performance will be terrible, but maybe you don’t care. Memory disks? Uh… presumably you have a reason for wanting a robust ephemeral filesystem, but fine, I guess? We won’t delve into specific differences between physical media types, as any advice we might offer will be overtaken by reality. We will discuss how to use that storage, however.

You want specific hardware advice? Fine. Spinning rust is slow and inexpensive, SSDs cost more but are faster, and NVMe is blazing fast but pricier still and you can only fit a handful on a system without resorting to extraordinary means.

Sponsorships are open at https://sponsor.mwl.io.

Posted on

January’s Jammed Sausage

This See the Sausage Being Made post goes to Patronizers in January and becomes public in February. Not a Patronizer? You could be! $12 a year gets you my latest updates, occasional free tidbits, and the completely pointless MWL Footnote Fortune File, freshly updated for the new edition of Networking for System Administrators.

I have one “job.” Yes, I write books. I get paid for that. But the thing I get a “paycheck” for is writing one blog post a month, at the beginning of the month. For Patronizers.

Here we are at January 28, and I am just getting to writing that blog post.

Absolutely zero excuses, only a procedural failure.

I like to write the blog post when something happens. What happened in the last six weeks?

Nothing.

Well, there’s The Longest Dark, the orcish solstice holiday. I took time off for that. Come January, I started to get to work and realized that my test lab was, at best, suboptimal. Many of the topics I write about can be tested and deployed on virtual machines, but I like to do filesystem books on real hardware or at least real disks. I have a host for that–it’s old, but who cares? Disks are disks, and this thing has eight SATA drives and a couple SATADOMs for the OS. Eight drives lets me test most common ZFS configurations. The host is large and noisy, but I have a basement and it has a BMC so I can kick it as needed. Creating a hard drive failure means two flights of stairs and opening a case, but I can live with that.

Mind you, I had stolen some of the hard drives for other purposes. More drives were needed. I descended into the Parts Closet. Well, I call it the Parts Closet. Other people call it a trash heap, or perhaps a Do-It-Yourself Toxic Gas Cloud Kit, Add Your Own Fire. Old computers, old printers, all precariously balanced. Laptops preinstalled with Windows XP with wobbly lids, cat5 wire crimping kits, pristine Unicomp Model M keyboards with button mice, Keyboardio Model 100s with Dvorak keycaps, a wall of small storage boxes with labels like SOEKRIS and PCCARD and SCSI TERMINATORS. The POWER OVER ETHERNET box sits right next to ETHERNET OVER POWER, which I do recommend as a convenient trick if the electrical system in your home doesn’t date from 1949 and wasn’t engineered by a PTSD-stricken pipefitter with delusions of grounding. (Fortunately cable TV is historical, so I can run Ethernet over the old coax someone paid big money to install.) Once I started digging for hard drives and discovered several expensive 250GB IDE disks, the necessity of a purge overcame my unwillingness to perform said purge.

Finding the cable crimpers reminded me that I needed to check some of the long-haul cabling in the house. The cable tester showed that I shouldn’t ask questions I don’t want answered.

Now let’s detour into backups.

I have a iX Systems miniXL running FreeNAS for local backups, and rely on Dropbox and Tarsnap for external backups. All the backups get tested monthly, mainly because I routinely screw up and must recover from backup. The miniXL also serves as my MP3 storage, because cleaning out the parts closet demands a soundtrack.

Naturally, I’m halfway through the ghastly purge when the music dies.

The miniXL’s boot SATADOM was no longer a boot device in the BIOS.

Fine. I’ve never worked with SATADOM hardware before, but how bad could it be? I ordered an inexpensive replacement.

The Supermicro power cable didn’t fit. Apparently that’s a well-known example of vendor lock-in, so I ordered a more expensive guaranteed compatible Supermicro SATADOM that was not at all compatible and immediately went back to the dealer. Fortunately, some ex-iX folks mentioned that the miniXL had space for an SSD. I had found a couple SSDs in the Parts Closet.

But then I realized that the miniXL has eight removable hard drive bays, is much quieter than my test system, and would easily fit under my standing desk (pic). Causing a hard drive failure with this host doesn’t require going to the basement and opening a case.

The critical service, music, is now being shared from my Mac desktop.

Backups? I have external USB cases for local hosts to back up to. I used the FreeNAS box because it was there. The miniXL is now running raw FreeBSD 15 and bhyve.

I now have a less unsuitable test environment and the toxic waste accident has been relocated from my home to the recycling center. Well, technically, I’ve converted one large toxic waste accident into two smaller accidents and added one of those accidents to a full scale disaster, but I’m not responsible for that disaster so I’m going to be a good citizen and ignore it.

In other news: last month I said I had completed fulfillment of the new Networking for System Administrators by packing and shipping a few hundred books. Uh… not so much. Yes, I finished packing the books. The US Postal Service hauled them off my porch. All good?

No.

More than one backer contacted me to say that their books were weirdly delayed. The USPS hauled my overseas books from Detroit to the big processing center Chicago, where most of them sat for a month. A handful ricocheted between Chicago and NYC for weeks. Several folks told me that this is unacceptable. I agree. I have no leverage nor any ability to fix this, however. My country is increasingly hostile to small business. I’m pretty sure that if I used USPS’ expensive two-day shipping, they would arrive speedily. I could use UPS or Fedex, except they’re expensive and might charge $50 to collect $3 in VAT depending on criteria so complicated as to be effectively unpredictable.

There’s also been other surprises. US Export Compliance returned one package because they require a full recipient name. A recipient like “A D Smith” is not acceptable; the shipping label must spell out Anonymous Doof. I’d expect the destination address to establish an internal protocol for demultiplexing initials, but apparently my government needs to know exactly who backs me. This has never been a problem before, but fine, rules change and there’s probably an announcement somewhere.

All I can think of to do is: warnings on my crowdfunding campaigns. Lots of warnings.

Much like I do when I warn you Patronizers that backing me is a terrible deal. January had no progress, but many tiny fixes. Yes, yes, that’s its own kind of progress but it doesn’t get words on the page. I do appreciate y’all. Thank you.