Category: bsd

Posted on

Fediverse Servers, plus mac_portacl on FreeBSD

One of my business mantras is “control your platform.” If you build your business around a site like Facebook, they can de-prioritize you and disappear you. Twitter’s implosion served as a fierce reminder of that, so I’m blogging more here.

Before Twitter’s implosion, the Fediverse (Mastodon, PixelFed, and all the other ActivityPub-powered systems) drove just as much traffic to my site as Twitter. Other social networking sites are negligible. If I want to follow my business mantra, I must run my own Fediverse server. I tested three options: Mastodon, pleroma, and GoToSocial.

Mastodon is huge, clunky, and handles like a tank made out of chicken wire, tar, and lobsters. I spoke with a few Mastodon operators, and none of them recommended it.

Pleroma? I followed the instructions. They didn’t work. I went looking into support, but I discovered that Pleroma seems to be the server of choice for TERFs, racists, and related jerks. Their recommended servers for new users are all on my personal blocklist. I don’t care to help those folks debug their instructions.

GoToSocial was a joy. Except it’s not only in development, it’s in alpha. They are very clear about this. The features that exist are beautifully done, but certain features I find critical are incomplete.

I have decided to wait to deploy a production fediverse server until GoToSocial enters beta.

For incomplete software, though, GoToSocial is surprisingly complete. It has its own web server and Let’s Encrypt implementation. If it can bind to ports 80 and 443, you don’t need a web server or ACME agent. The catch is, gotosocial(8) runs as an unprivileged user. It can’t bind to privileged ports.

Enter mac_portacl(4).

In the BSD tradition, the man page details everything you can do with this Mandatory Access Control kernel module, but in short it lets you permit particular users or group to bind to privileged network ports. I don’t care for mac_portacl in production, as the rules are hard to read when you’re debugging. If you want me to use an access control program, the output better be no harder to read than pfctl -sr. But here’s how you do it.

Enable the module in /boot/loader.conf.

mac_portacl_load="YES"

You can now write port ACL rules. Each rule has four parts:

uid or group : numerical identifier : tcp or udp : port number

The gotosocial user has uid 209. I want uid 209 to be able to bind to TCP ports 80 and 443, so I need these rules.

uid:209:tcp:80
uid:209:tcp:443

Set the access control rules in /etc/sysctl.conf.

net.inet.ip.portrange.reservedhigh=0
security.mac.portacl.rules=uid:209:tcp:443,uid:209:tcp:80

The first sysctl disables the traditional “reserved port” behavior and allows unprivileged programs to bind to ports below 1024.

The second sysctl installs our rules in the kernel. When you write to this sysctl you must include all rules you want active, separated by commas.

Would I use this in production? If the software has a solid security track record and is designed to be directly exposed to the Internet, sure. If you’re running a web server, some program has to listen on port 80. GoToSocial is brand new, though, and I’d like to see a bit of a track record before I completely trusted it.

When GoToSocial enters beta next year and I deploy it for real, I’ll put an nginx or httpd in front of it so I can filter when needed.

Are there other options other than Mastodon, pleroma, and GoToSocial? Sure. But I’m out of time, and really need to make some words this week.

Posted on

Two pieces by me in this month’s FreeBSD Journal

Yes, I’m trying to use the blog more, rather than dumping everything to multiple social media outlets. Yes, this is in part in response to Comic Book Supervillain purchasing Twitter and kneecapping the moderation team. If you want me on social media, I’m on the fediverse as @mwlucas@bsd.network.

Anyway.

The latest issue of the FreeBSD Journal has two articles by me: one on PAM tips & tricks, and the other my regular “We Get Letters” “advice” column. With any luck, the Journal’s editorial board will use these articles as grounds for reconsidering their “we’ll publish anything Lucas sends us” policy.

If you find the Letters column amusing, I’ve collected the first three years of that column in Letters to ed(1).

Posted on

upgrading PHP 7.4 to PHP 8 on FreeBSD

What, a technical post? It happens. Rarely. Usually, I’m focused on the tech that goes into a book, but sometimes the real world intervenes.

Like PHP. PHP is very much the real world. My site has been running PHP 7.4 for a while, which goes end of life on 28 November. I put this off as long as possible, but it’s time to update.

I run my e-bookstore on Woocommerce, which is built on WordPress, which is built on PHP. What started as a silly experiment has become the center of my business. I need to minimize downtime, which means I must check everything before upgrading. It’s PHP, which means it’s a maze of twisty little modules that all look alike. PHP has this annoying habit of adding, removing, splitting, and changing modules. Running PHP applications on FreeBSD is all about finding the module your application needs, so I want to identify all possible problems before changing.

First, let’s see what packages need upgrading.

# pkg info -x php
mod_php74-7.4.32_1
php74-7.4.32
php74-ctype-7.4.32
php74-curl-7.4.32
php74-dom-7.4.32
php74-exif-7.4.32
php74-fileinfo-7.4.32
php74-filter-7.4.32
php74-gd-7.4.32
php74-iconv-7.4.32
php74-intl-7.4.32
php74-json-7.4.32
php74-mbstring-7.4.32
php74-mysqli-7.4.32
php74-openssl-7.4.32
php74-pcntl-7.4.32
php74-pdo-7.4.32
php74-pdo_mysql-7.4.32
php74-pecl-imagick-im7-3.5.1_1
php74-phar-7.4.32
php74-posix-7.4.32
php74-session-7.4.32
php74-simplexml-7.4.32
php74-soap-7.4.32
php74-tokenizer-7.4.32
php74-xml-7.4.32
php74-xmlreader-7.4.32
php74-xmlrpc-7.4.32
php74-xmlwriter-7.4.32
php74-zip-7.4.32_1
php74-zlib-7.4.32

31 packages. Software like Tiny Tiny RSS and WordPress depend on PHP, but if the underlying PHP software has all the necessary libraries then they should just work. Should. But PHP modules sometimes disappear, get replaced, or get renamed. I want a list of all the modules I need before running any commands. So, what would the PHP 8.0 version of these packages be named? I have to iterate through sed a couple times to trim out excess version information and wind up with this.

# pkg info -x php | sed s/74/80/g | sed s/-7.4.32//g | sed s/_1//g

mod_php80
php80
php80-ctype
php80-curl
php80-dom
php80-exif
php80-fileinfo
php80-filter
...

Those look sensible. Now check to see if the packages exist.

I could automate this by checking the exit code of each command, but the list is short enough that I can process it by hand. I run one package search at a time, letting xargs prompt me for each one so I can eyeball the results.

# pkg info -x php | sed s/74/80/g | sed s/-7.4.32//g | sed s/_1//g | xargs -L1 -p pkg search
pkg search mod_php80?…y
mod_php80-8.0.25 PHP Scripting Language
pkg search php80?…y

This particular search will spew a couple hundred lines of output, but I’m confident the base PHP 8.0 package is in there.

...
php80-intl-8.0.25 The intl shared extension for php
pkg search php80-json?...y
pkg search php80-mbstring?...

Ooops! Pay attention here. There is no package for PHP 8.0’s JSON module! Make a note of that.

At the end, I have problems with three packages: php80-json, php80-openssl, and php80-xmlrpc. Freshports tells me that the JSON and OpenSSL modules were added into the default PHP 8.0 package, so I can cross those off my list.

The XML-RPC module is another tale. PHP 8.0 no longer has an XML module. Fortunately, that same bug lists a replacement pecl-xmlrpc. There’s a related php80-pecl-xmlrpc module.

I have a list of modules to install. For a last check, I’ll look for anything that depends on PHP 7.4.

# pkg info -dx php74
The list looks different, but contains the same modules. I’m as prepared as I can be.

One last check. Make a list of the packages to install. Eyeball it to make sure it looks right.

# pkg info -x php | sed s/74/80/g | sed s/-7.4.32//g | sed s/_1//g > php8.pkg

Create a boot environment, and do a dry run. If I remove all packages with PHP in their name, what will get pulled? Using -n tells me what the command would do, but doesn’t actually change anything.

# bectl create 12.3-p7-lastbeforePHP
# pkg remove -nx php74

That list looks sensible. Now remove the packages, and install everything on our list.

# pkg remove -x php74
# cat php8.pkg | xargs -L1 -p pkg install -y

The -p argument to xargs prompts me for confirmation, so I can use -y on the pkg command. The install fails on the nonexistent JSON, OpenSSL, and XMLRPC modules, but that’s expected.

At the end, I manually install php80-pecl-xmlrpc.

Reboot.

Test, test, test. Run a test purchase. It works.

Everything looks okay? I guess I can turn it over to the Crowdsourced Monitoring System, aka “y’all,” and go make some paying words.

Posted on

“OpenBSD Mastery: Filesystems” draft done!

After far too long, I have finished a first draft of OpenBSD Mastery: Filesystems. Sponsorships are now closed.

I’m asking tech reviewers to get any comments to me by 15 October 2022. That’s four weeks. It might seem tight, but experience shows that people either get their comments to me immediately, or wait until the last possible weekend. I’m not complaining–I do exactly the same thing. Please return any comments either a) in plain text, with enough context that I can find them when page numbers change, or b) as annotations directly on the PDF.

My tech reviewers are now in their third decade of winning the prize for “most likely to use many different PDF readers.” A file that works for one won’t work for another. I work around this by distributing three PDFs of the manuscript, each identical in contact but prepared differently. Everyone should be able to find one that works for them.

If you’re interested in doing a tech review, please drop me an email (mwl at mwl dot io) saying who you are, why you would make you a good reviewer, and that you won’t share the manuscript. (Piracy is bad, but having my name on an unreviewed and thus certainly incorrect document is horrifying). I’ll ignore responses that can’t follow those instructions, because whenever I don’t I get difficult-to-decipher feedback. (I have previously received PostScript diffs, and… no. Just no.)

I’ll be turning my attention to the Prohibition Orcs copyedits next. Then it’s back to the Epic Giant Fiction Project, and another tech book, title TBA.

Posted on

“OpenBSD Mastery: Filesystems” Status Report

I just finished the ‘non-native filesystems’ part of “OpenBSD Mastery: Filesystems.” I wouldn’t say I’ve finished the hard part, but I have finished the “intertwined to an unholy degree” part.

In the beginning, Berkely released Unix. This made a lot of vendors very angry and has widely been regarded as a bad move.

Why have I spent months on five chapters? Because everything in the core storage system of any Unix is intertwined to a nearly unholy degree. To understand filesystems you must understand partitioning, but to understand why Unix uses partitions as it does you need to understand filesystems. I have to meticulously disentangle facts so that I can start explanations at the bottom of the storage stack, but add in enough higher-level details exactly when you need them so you can make sense of why the bottom layers work as they do.

Otherwise, you’d look at computers and think “Wow, this whole thing is stupid.” Don’t get me wrong, the whole thing IS stupid, but it’s your job to understand the stupidity and I don’t need to be rubbing your nose in it.

Have I written on these before?

Yes, many times.

Does that make them easier to write?

BWAHAHAHAHA. No.

Can I use the earlier edition of Absolute OpenBSD to guide me?

Sure, except that the book is ten years old and every detail within is suspect and must be triple-checked against the current state of the software and oh by the way that book doesn’t even mention GPT or FUSE so burn it all down. AO2e is a checklist of things that will annoy me.

The good news is, the sections that remain are fairly tidy. They’re not standalone, but they are less incestuously intertwined with other topics.

  • NFS
  • iSCSI
  • softraid
  • encrypted storage

    The first two are mostly standalone, and are thus easier to write. Also, as an author I am highly grateful that OpenBSD does not support NFSv4.

    I’m going to push hard to get this done in the next few weeks. Which brings me to:

    Once that happens, sponsorships will close. If you want your name in the book, act now.

    • Posted on

    Auction Against Human Trafficking

    My wife works with Soroptimist International of Grosse Pointe to support their anti-human-trafficking efforts. They usually hold a variety of fundraisers, including a 5K race in September.

    2020 and 2021 have been less than optimal fundraising years. Running a 5K on your own is much less interesting than trying to outrace a whole bunch of folks.

    This year SIGP is raising money for the Joseph Project, a non-for-profit organization who connect human trafficking survivors with skilled pro bono legal counsel. One of the worst parts of human trafficking is that people must break the law to survive, let alone escape. Survival usually means a criminal record. I’ve written about human trafficking before, and completely support the efforts to give these folks a new chance.

    Meanwhile, here I am with a variety of stuff that’s explicitly designed to raise money in the most daft way possible, and a whole mass of readers who couldn’t be unemployed if they tried. To try to cover the gap, I’m auctioning off something that was never before available online, and never will be again.

    The Bail Bond Denied edition of FreeBSD Mastery: Jails.

    Only five copies of this edition exist. One is in my archives. One was auctioned off at BSDCan 2019, the other at Penguicon 2019. The fourth will be reserved for the next charity auction I attend in person.

    The fifth is being auctioned off right here. The members of the Grosse Pointe Soroptimists Club are signing it to express their gratitude for your generosity. I’ll sign it myself, thereby destroying their added value.


    This is your only opportunity to get this book without being present at an event I’m at. Yes, I’m opening this up to seven billion people.

    The auction rules are simple:

    Comment on this post to bid. I advise bidders to click the “notify me of new comments by email” button, so they can see when they’re outbid.

    The auction runs from now until 5 PM EDT 25 September. Most of the bidding happens on the first and last days, but I want to leave time for the Internet’s slow publicity machine to catch on. If bidding is frenzied in those last moments, I’ll leave the auction running to squeeze the most cash out of you get the best result for the Soroptimists.

    Once the auction closes, send your donation to SIGP Paypal, by following the link on grossepointesoroptimist.net and send me your receipt. I will mail you the book at my expense.

    Note that you don’t need to win a silly book to donate. You can donate out of your own generosity. I promise they’ll appreciate it. If you need someone to blame to keep your hard-boiled reputation, tell them I made you do it.

    Posted on

    BSDCan 2020 Charity Auction

    Every BSDCan concludes with a charity auction for the Ottawa Mission. It’s a highlight of the conference, and we’ve raised tens of thousands of bucks for this worthy organization. BDSCan is virtualized this year, because plague. Every year, I try to come up with two unique items to contribute to the auction. One of them is valuable, or at least has a small amount of inherent worth. The other is the stupidest thing I can think of, just to see if I can get people to bid on it. I consider last year my absolute peak success in both regards.

    Today, the coronavirus has messed many people up. The United States is undergoing a long-overdue racism reckoning. If you have a tech job, many (not all) of you are moderately insulated from the worst effects of both of these. I would strongly encourage every one of you to donate to bail funds.

    The Ottawa Mission still needs help, though, and I am unwilling to let the BSDCan tradition lapse. So this year, I’m offering something unique and stupid. It also includes just a soupçon of humiliation for yours truly.

    In October 2000 I attended my very first BSD conference, out in Monterey, California. This was before “Absolute BSD.” This was before my “Big Scary Daemons” column at O’Reilly Network. The conference sponsors flew me out on the strength of three or four FreeBSD articles I published in Sys Admin Magazine. This seemed ludicrous–but then, so did everything else in the dot-com boom that I’d so assiduously avoided. (This was the first of several times that Charles Mackay has saved my family pain.) Apparently the sponsors knew what they were doing, as meeting all these BSD folks changed my life. One day, I might forgive them.

    I signed up for Kirk McKusick’s BSD Kernel Internals course. It included actual printed-on-paper slides, in a three-ring binder. When cleaning the basement, I found this notebook.



    It has my notes. My bad notes. Because I didn’t understand anything, even after our esteemed doctor explained it in words of two or fewer syllables.

    Not all the pages have notes. I thought I understood those slides and his lecture well enough that I didn’t need to take notes. I was wrong. The notes would have been factually incorrect, though, so it’s all the same in the end.

    As part of Virtual BSDCan, I am auctioning off this historical flotsam to support the Ottawa Mission. Right here, on this web page. Comment with your bid. Comments not containing bids might be deleted.

    The mission is in Canada, so all bids are in Canadian dollars. (If you screw up and pay the same number of US dollars or Euros instead, that’s okay.)

    With all that’s going on it’ll to take some time for word to spread, so this auction remains open until 5 PM EDT, Monday 15 June 2020. If last-minute bids are coming in hot and heavy, I will let the auction run until they play out. I have no problem doing a virtual “going… going…. gone!”

    The winner sends me a copy of their donation receipt. I mail the tutorial materials, including the Minimum Viable Three-Ring Binder that’s probably the most valuable component.

    If you lose the auction, the Ottawa Mission would welcome even modest donations. So would those bail bond funds.

    Posted on

    The MWL 2020 Asia Tour

    Yep, I’m a big star now, touring Asia and everything! Sort of. Two countries. Two cities. The world’s most minimal tour. I’m a big star, in a really really tiny universe.

    19-22 March 2020, I’ll be at AsiaBSDCon. I’m presenting a four hour tutorial on FreeBSD jails, as well as attending the conference.

    The fine folks at HasGeek are sponsoring me on an accompanying trip to Bangalore, India, for three events. (Cool fact of the day: they’re not conferences in India, they’re events, because a “conference” apparently involves the Indian government and this isn’t a government thing.)

    25 March, I’m offering a public lecture on Where is the Sysadmin Today at Juspay’s offices. I have rants thoughts. Oh, do I have rants thoughts.

    27 March, I’m attending Netconf. This is an Unconference (Unevent?), so the program won’t be set until it starts. I’ll be proposing my new SNMP talk. I could also give any talk I’ve given before. If you’re attending and want me to give a specific talk, please comment or use the contact form to ask me to submit it.

    28 March, I’m doing a reading of git commit murder at Champaca Bookstore, as well as a Q&A with Swapneel Patneka and anyone else who opens their mouth.

    Why do this trip, when I loathe travel? Over the last twenty years, I’ve promised several folks that I would one day attend AsiaBSDCon. I keep my promises. I’m looking forward to being there, but not to getting there. The Bangalore trip is serendipitous. Presenting technology is how I built my career. Bangalore is a technology center and obviously a place I should present in. HasGeek asked if I would be interested, I said “if you could put an event by AsiaBSDCon,” and those folks actually went and did it. I’m simultaneously amazed and honored that they’ve gone to such trouble.

    Plus, HasGeek opened discussions by promising gelato. They did their research.

    I’ll have a couple free days in each place, yes, and I’ll take advantage of them. I’d rather like to attend a few classes at the Hombu Dojo, but… Fly across the world, teach crowds of strangers to whom English is a second language, talk to folks about areas I’m an expert in? Sure. Set foot on Ueshiba’s tatami? I’ve only practiced martial arts for eighteen years, there’s absolutely no way I’m worthy.

    And India’s history is thousands of years deep, plus there’s elephants and tigers and… and… everything. I can’t decide what to see.

    I’ve mentioned before that I’m cutting down my traveling. This trip will cost me at least a week of writing time before the trip, and probably two weeks of writing time afterwards as I recover. It’s at least a month of proper writing, all told, and probably more. I can’t authoritatively say that this is my final trip to Asia, no matter what. I can say that I’m not planning to travel so far again. If you’re on that side of the world and want to meet me, this is your best opportunity.

    I will do Penguicon and BSDCan in 2020, but otherwise, I’ll be home making words.

    Posted on

    The Six Prequels to “FreeBSD Mastery: Jails”

    I’ve said a few times that I needed to write six books before I could write FreeBSD Mastery: Jails. Some were for the reader, because I didn’t want to take a break from the jails content to explain a seemingly unrelated topic. Some were for me, because I didn’t know everything I needed about a topic to effectively cover jails.

    I thought which six books those were was obvious. I have heard from more than one person that it’s not. I chose to not put a title-by-title course of study in the front of the jails book. Seems I was wrong about that as well.

    So: without further ado, here are the six prequels to FreeBSD Mastery: Jails.

    • Networking for Systems Administrators

      People want to bridge their jails, or VNET them, or NAT them, or otherwise play tricks with their network. You can’t set up a virtual switch if you don’t understand what a switch is. You can’t network your jails if you don’t understand netmasks. Every time your first virtual network grows, you have to troubleshoot everything.

    • FreeBSD Mastery: Storage Essentials

      Jails are all about storage. You can implement one or two jails without knowing what you’re doing, but eventually they’ll ruin your day.

    • FreeBSD Mastery: ZFS
      FreeBSD Mastery: Advanced ZFS

      ZFS is incredibly jail-friendly. It doesn’t suit all deployments, but if you want to implement jails at scale you’re almost certainly exploiting ZFS.

    • FreeBSD Mastery: Specialty Filesystems

      Any non-trivial jail implementation requires understanding devfs, nullfs, and memory filesystems. Many use iSCSI, NFS, and/or autofs. By the time I put all that in a book, I might as well add in namespace filesystems and HAST and completely cover special-purpose filesystems.

    • Absolute FreeBSD, 3rd Edition

      By the time I wrote all of the above, FreeBSD had changed enough that the second edition wouldn’t suffice.

    Yes, I planned this. Every book I write is ordered internally in much the same way. I look at the material for each chapter and say “What must the reader understand before reading this?” I often revisit my chapters as needed, or even split them. Chapters 17 and 19 of AF3e were originally part of early chapters, but I had to split those chapters and put parts of them later because the reader would lack the context to understand the material.

    Mind you, this is only what you need to get jails working. Managing jails is the pinnacle of systems administration practice, so I’d certainly recommend you learn about SSH, PAM, and sudo. Really, though, I’d suggest get a job at the gelato shop. You’ll be happier.