Dealing with Negative Reviews

Pop quiz: what do you do when someone releases an excoriating online review about something you’ve sweated blood over?

a) Leave a vitriolic comment questioning the reviewer’s integrity and decency.
b) Leave a bitter comment calling the reviewer an idiot.
c) Leave a vengeful comment declaring war.
d) A, B, and C, with lots and lots and lots and lots of comments
e) Give up. Never do anything again.
f) Close your browser and get on with your life.

I thought most authors understood the best answer, but then I came across a review where the author not only responded, she melted down. Spectacularly. So I thought it might be worthwhile to say something about dealing with negative reviews of your work.

Remember, reviews are not for the author. They are for prospective readers. The reviewer is not claiming you are a bad person. He is stating that your work did not meet his needs, expectations, or standards. Most negative reviews of my work amount to “I don’t like the tone and attitude of this book.” That’s OK. I deliberately use that tone and attitude, in the belief that the number of people who will appreciate it exceeds the number of people it will annoy.

Sometimes, negative criticism is actually constructive. If I don’t learn from my mistakes, who will?

The correct response to a negative review is: silence.

An even better response would be to send the reviewer a two-sentence note: Thank you for taking the time to review my book. Sorry you didn’t like it, but I appreciate your effort. Sincerely, Your-Name-Here. But I’m not that good of a human being.

Nonfiction Tuckerization Auctions?

For centuries, authors have traded mentions in a book for cold hard cash. Today, this is most often done for charity, as a Tuckerization auction. As a BSD author, though, I think that there’s a way to put this to use to raise development money for various BSD projects. BSD always needs money.

When Absolute FreeBSD came out, the FreeBSD Foundation auctioned off the first copy off the press. It raised $600. I suspect that getting your name in the book, or being able to name something in the book, might raise more.

I’m considering hold an auction to, say, let a reader name something in a tech book: a server, a sample user, whatever. I’d mention their winner by name in the acknowledgments. The money would go to the project covered in the book, and I would ask someone from the project to run the auction. (I don’t want to go near that money, as I’d probably spend it foolishly, for food or shelter or soap.)

I’d need some basic rules — the desired name would have to get past my publisher, for example, so obscenities are out. You couldn’t blatantly insult people — while I’m fine with naming my example server LucasDroolz, I’m not comfortable using someone else’s words to abuse other people. When I insult someone, I want to do it personally.

Would this be a publicity stunt? It would be publicity, yes. But the real goal is to extract money from you and give it to a developer.

Of course, setting up such an auction would be time and expense. I’d risk my own time and expense on such an idea, but this would mean asking other people to do so as well. Therefore, my question for readers is:

Would you bid in such an auction? Given the cause, how much would you bid? Do you think it’s a stupid idea? I’m also open to suggestions on where to run such an auction. eBay has a charity option, but they still take a cut. I suspect there’s a better choice.

Please reply in the comments, not email.

Roundcube/pgsql on FreeBSD

My employer’s current webmail solution is pretty tightly tied to the mail server. We need a webmail solution, but I want to be able to move, change, and upgrade webmail independently of the mail server. I’m testing Roundcube, running on FreeBSD-current/amd64 diskless on KVM.

First install Roundcube. Go to /usr/ports/mail/roundcube and run make config-recursive. You can use MySQL, PostgreSQL, or SQLite. Personally, I’m a Postgres bigot, mainly because Postgres has inflicted less pain than MySQL over the years. (Remember, punishing someone less feels like a reward!) I selected postgres, disabled mysql, then chose other environment-specific options such as LDAP. After configuring all of roundcube’s dependencies, run make all install clean.

No matter the database you prefer, in FreeBSD the client is packaged and built separately from the database server. I built /usr/ports/databases/postgresql84-server. Normally I would enable PAM, to enable database authentication via the operating system authentication, but that’s apparently broken as of this writing.

I want to modify httpd.conf as little as possible, so I do most of my configuration via files in /usr/local/etc/apache/Includes. I create the following files:

vhosts.conf
NameVirtualHost *:80
NameVirtualHost *:443

And of course, we need to acces webmail over SSL. Here’s ssl.conf:

Listen 443
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
SSLPassPhraseDialog builtin
SSLSessionCache "shmcb:/var/run/ssl_scache(512000)"
SSLSessionCacheTimeout 300
AcceptMutex posixsem
SSLMutex "file:/var/run/www/ssl_mutex"

I use RCS for minor stuff like this, so I need to block web site visitors from viewing my RCS files. Here’s blockRCS.conf


Order allow,deny
Deny from all
Satisfy All

Finally, here’s the configuration for the webmail virtual server, webmail.conf. The most interesting thing here is that I automatically redirect HTTP connections to HTTPS. (Perhaps interesting is the wrong word. “Least uninteresting?” Yeah, that’s better.)

AddType application/x-httpd-php .php

ServerName webmail.domain.com
Redirect permanent / https://webmail.domain.com/
ServerAdmin webmail@domain.com
ErrorLog "|/usr/local/sbin/rotatelogs /var/log/httpd/webmail_error_log.%Y-%m-%d-%H_%M_%S 86400 -300"
CustomLog "|/usr/local/sbin/rotatelogs /var/log/httpd/webmail_access_log.%Y-%m-%d-%H_%M_%S 86400 -300" combined
DocumentRoot /usr/local/www/webmail/

Options None
AllowOverride All
Allow from all



ServerAdmin webmail@domain.com
ServerName webmail.domain.com
SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

SSLOptions +StdEnvVars

BrowserMatch ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
SSLCertificateFile etc/apache22/certs/webmail.domain.com.crt
SSLCertificateKeyFile etc/apache22/certs/webmail.domain.com.key
ErrorLog "|/usr/local/sbin/rotatelogs /var/log/httpd/webmail_ssl_error_log.%Y-%m-%d-%H_%M_%S 86400 -300"
CustomLog "|/usr/local/sbin/rotatelogs /var/log/httpd/webmail_ssl_access_log.%Y-%m-%d-%H_%M_%S 86400 -300" combined
DocumentRoot /usr/local/www/roundcube/

Options Indexes FollowSymLinks
AllowOverride All
Allow from all

Initialize your postgresql database. I got Postgres help from here.

# /usr/local/etc/rc.d/postgresql initdb

Edit your Postgres config file, /usr/local/pgsql/data/postgresql.conf. Be sure that your database is only listening on the loopback address.

listen_addresses = 'localhost'

If you’re a Postgres guru, make any other changes you like. Then configure loggin in /etc/syslog.conf:

!postgres
*.* /var/log/pg.log

The log files must exist before syslogd will write to them.

# touch /var/log/pg.log
# chown pgsql:pgsql /var/log/pg.log

Now run /usr/local/etc/rc.d/postgresql start and check for errors. It should start without trouble, unless you mucked with your configuration too much.

Normally I would edit pg_hba.conf to tie my account password to PAM, and through PAM to LDAP, but as that’s broken right now, I’ll create a local user.

# su pgsql
$ createuser -sdrP mwlucas
Enter password for new role:
Enter it again:

Now create the Roundcube database, as per the instructions.

$ createuser roundcube
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) n
Shall the new role be allowed to create more new roles? (y/n) n
$ createdb -O roundcube -E UNICODE roundcubemail
$ psql roundcubemail
psql (8.4.7)
Type "help" for help.

roundcubemail=# alter user roundcube with password 'WeHatesWebmail';
ALTER ROLE
roundcubemail=# \c - roundcube
psql (8.4.7)
You are now connected to database "roundcubemail" as user "roundcube".
roundcubemail=> \i SQL/postgres.initial.sql

After a bunch of SQL spammage, we have a database. Log out with:

roundcubemail=> \q

Roundcube needs two configuration files, db.inc.php and main.inc.php. Go to /usr/local/www/roundcube/config and copy the .dist versions of these files.

Tell Roundcube where to find its database by setting a DSN in db.inc.php.

$rcmail_config['db_dsnw'] = 'pgsql://roundcube:WeHatesWebmail@localhost/roundcubemail';

In main.inc.php, set your mail server. (If you don’t set a mail server, Roundcube will let you connect to any mail server you like. This would confuse my users. Confusion leads to phone calls, something I avidly avoid.)

$rcmail_config['default_host'] = 'mail.domain.com';

Verify that pgsql and Apache are running, and browse to your webmail site. You have a webmail server!

Overall, the setup was pretty straightforward. I’m not saying I’ll keep Roundcube, but my test users are basically content, so it’s a strong candidate.

experiments in publishing

Readers who don’t care about my non-technical writing should skip this post.

My friends include authors who are traditionally published, and some who are self-published. Both groups make excellent cases for their choices, and I’m not going to argue against anyone else’s decisions. Whatever works for you, live and let live, and so on.

But both sets make excellent arguments for what I should do with my career. Both sets tell me my non-technical work is good enough to make it. The traditional authors tell me to keep knocking at the doors of the Big Six publishers, and eventually I will get in. The self-publishers tell me that I have more “platform” than many traditionally published novelists, and I should leverage that to bootstrap my non-technical writing career.

It’s an emotional topic. People are very vested in their opinions. I can argue either side, and my opinion tends to vary depending on whoever I’ve spoken to most recently. The way to answer questions is with an experiment. Experiments start with a falsifiable statement. Here’s mine. “I have sufficient platform to profitably self-publish fiction on ereader platforms.”

Now, some conditions. What, exactly, is “profitable”? Here are some rough numbers that would probably make an accountant cry, but they’re adequate for this experiment.

Big Name authors sell short stories for thousands of dollars. Medium Name authors sell short stories for about $500. No-name authors sell short stories for prices from $10 to free. My first sale was for $100, but all others were for $10. I’m a no-name. So, let’s classify my writing labor as a $10 expense.

Ebooks need a cover. My graphic design skills are roughly equivalent to a badger’s. I can get an adequate cover for $25, largely due to the curiosity of my artist friend Brad McDevitt about the results.

Software to transform a document into an eBook is free. I started with Mobipocket Creator to transform a PDF into Amazon’s Kindle format, and then used Calibre to transform it into the Barnes & Noble ereader format. To reach the Apple audience, as well as a whole bunch of smaller retailers, I need to use Smashwords’ .doc file upload. I’ve experimented with each of these, and I believe that once I have a hang of them I’ll be able to upload to all three sites in about an hour. How much is my authorial time worth? Judging by the $10 price of a completed short story: not much. How about the time of a skilled software operator? That’s a little better. I’ll call it $20/hour.

I could add in a share of the computer, office space, and so on, but that’s all stuff I need for my tech writing business. I’m not going to count that. Similarly, I own a Kindle, for my own purposes, but I’ll use it to test my early ebook builds. I won’t count that, either. Should this be successful, all these would become legitimate expenses.

The expenses to publish a short story are, roughly speaking, $55.

The going price for ereader short stories is $0.99. Amazon gives authors a 35% royalty on $0.99 items, or $0.35 per sale. I must sell 157 copies of a story to break even.

One big question is, “is the work good enough to sell?” I’ll control for that by starting with stories I have previously sold, but whose rights have reverted back to me. If I have success with them, I’ll try a couple pieces that have never sold.

Time is another factor — if I sell 157 copies over 10 years, is that a win? No. How about one year? Maybe.

The first story is up on Amazon. I still need to do B&N and Smashwords.

A month or so after I have the first one up on all three sites, I’ll report my initial results.

Buy My Books for Japanese Disaster Relief

Tuesday, 22 March, O’Reilly, No Starch Press, and Tidbits will donate all revenue, less author royalties, from Deal of the Day sales to the Japanese Red Cross. This includes all of my books available in electronic format: Absolute FreeBSD, Cisco Routers for the Desperate, and Network Flow Analysis.

If you’ve been waiting to pick up any of my books in electronic format — or any books by other NSP/O’Reilly/Tidbits authors — this is the day. Today only. You want them all. I know you do.

If you already own everything I’ve written, in both paper and electronic form, why not try something from another author? Everybody needs to know how to build Badass Lego Guns. Or you can just give directly to the Japanese Red Cross.

I’ll tweet this multiple times through the day. The O’Reilly site will be announcing totals throughout the day.

short story on Kindle

My short story Opening the Eye is now on Amazon as a $0.99 ebook. This story first appeared in Horror Library vol 2.

This story is in that hazy ground between urban fantasy and horror. If you enjoy that sort of thing, please check it out. If you like it, please leave a review. It won’t show up in most Amazon searches unless there’s a certain number of reviews.

If you don’t enjoy that sort of story, don’t read it. If you’re related to me, you don’t want to read this one. Trust me.

I’m not going to turn this blog into a plea for people to read these things, but I will briefly mention when they’re available in new places, such as the Nook or the Apple bookstore.

This is an experiment. (I have experimental conditions and everything!) But more on that later this week.

Wherein I learn about initrd

Post summary: will someone PLEASE port a recent KVM to any BSD? There’s beer in it for you.

I’ve been attempting to upgrade my diskless virtualization cluster to Ubuntu 10.10. Diskless boot worked fine in the ESXi test area, but real hardware would not boot. This same hardware booted fine with Ubuntu 10.04 and 9.whatever. When I looked at the console, I saw:

ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure
/init: .: line 3: can't open '/tmp/net-*.conf'
[ 2.300079] Kernel panic - not syncing: Attempted to kill init!
[ 2.306052] Pid: 1, comm: init Not tainted 2.6.35-27-server #48-Ubuntu
[ 2.312653] Call Trace:
[ 2.315161] [] panic+0x90/0x113
[ 2.320025] [] forget_original_parent+0x33d/0x350
[ 2.326433] [] ? put_files_struct+0xc4/0xf0
[ 2.332339] [] exit_notify+0x1b/0x190
[ 2.337699] [] do_exit+0x1d5/0x400
[ 2.342817] [] ? do_page_fault+0x159/0x350
[ 2.348609] [] do_group_exit+0x55/0xd0
[ 2.354076] [] sys_exit_group+0x17/0x20
[ 2.359617] [] system_call_fastpath+0x16/0x1b

The useful messages are obviously further up, but the scrollback buffer is fubar. (Apparently when an Ubuntu box dies, it dies really really hard.) A serial console let me scroll back through the boot messages.

...
[ 2.004954] Uniform CD-ROM driver Revision: 3.20
[ 2.009944] sr 0:0:1:0: Attached scsi generic sg0 type 5
[ 2.015651] Freeing unused kernel memory: 836k freed
[ 2.021283] Write protecting the kernel read-only data: 10240k
[ 2.027551] Freeing unused kernel memory: 320k freed
[ 2.033118] Freeing unused kernel memory: 1620k freed
Loading, please wait...
[ 2.063067] udev[81]: starting version 163
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/nfs-top ... done.
FATAL: Could not load /lib/modules/2.6.35-27-server/modules.dep: No such file or directory
FATAL: Could not load /lib/modules/2.6.35-27-server/modules.dep: No such file or directory
ipconfig: no devices to configure
ipconfig: no devices to configure
ipconfig: no devices to configure

The machine cannot find its modules directory? Odd. A packet sniffer found that the diskless client didn’t send an NFS request. It was just giving up after running initrd. I carefully reviewed the serial console output and compared it to the test Ubuntu systems, and found that the initial ramdisk wasn’t attaching a device driver to the Ethernet interface.

Initrd is an “initial ramdisk.” It loads a kernel and device drivers, for the purpose of finding the root file system and loading the real kernel and actual device drivers. If you install a machine in one environment, the initial ramdisk includes only the device drivers for that environment.

Checking /etc/udev/rules.d/70-persistent-net.rules of the older system revealed:

# PCI device 0x14e4:0x1659 (tg3)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:17:31:d8:42:52", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

The Ethernet cards on my physical servers use Linux’s tg3 driver.

To add this driver to initrd, I went to the Ubuntu 10.10 server on my ESXi test box and created the file /usr/share/initramfs-tools/modules.d/tg3. That file contained a single word, tg3. I then created the new initrd with:

# update-initramfs -u -k all
# mkinitramfs -o /home/mwlucas/initrd.img-2.6.35-27-server-pxe

Copy that image to my TFTP server, reboot the hardware, and everything boots.

pxelinux.cfg/* versus RCS

I’m a fan of version control in systems administration. If you don’t have a central VCS for your server configuration files, you can always use RCS. I habitually add #$Id$ at the top of configuration files, so I can easily see who touched this file last and when.

On an unrelated note, I’m upgrading my virtualization cluster to Ubuntu 10.10. The worker nodes run diskless. Each diskless node reads a configuration file over TFTP. Mine looked like the following:

#$Id$

LABEL linux
KERNEL vmlinuz-2.6.35-27-server
APPEND root=/dev/nfs initrd=initrd.img-2.6.35-27-server-pxe nfsroot=192.0.2.2:/data1/imagine,noacl ip=dhcp rw
TIMEOUT 0

This has worked fine for a year or so now, with me changing the kernel and initrd versions as I upgraded. With the Ubuntu 10.10 update, however, some pieces of hardware wouldn’t reboot. Most booted fine, but a few didn’t come back up again.

This is notably annoying because the hardware is in a remote datacenter. Driving out to view the console messages burns an hour and, more annoyingly, requires that I stir my lazy carcass out of my house. I have a serial console on one of the machines, but not on the affected one. Fortunately, I do have remote power, and I can make changes on the diskless filesystem.

Packet sniffing revealed that the machine successfully made a TFTP request, then just… stopped. This exact same configuration and filesystem worked on other machines, however. Except that the affected machines all had #$Id$ on the first line of their pxelinux.cfg file, and machines that booted successfully didn’t.

That shouldn’t matter. Really, it shouldn’t. pxelinux.cfg files accept comments. But I removed the tag, making the first line the LABEL statement, and power cycled the machine. And it came up perfectly.

Apparently this particular rev of Linux PXE is incompatible with version control ID tags. Oh joy, oh rapture!

blather versus undeadly.org

So how does the traffic I get here compare to an established Web site, like the OpenBSD aggregator undeadly.org? Undeadly linked to my OpenBSD story

incoming!

Can you guess when?

No, they weren’t the only ones. But 6 of my top 10 referring URLs were in undeadly.org. The lesson is, do not feed the puffer fish. They will swarm and eat you like the tender tasty morsel you are. They even crashed my helpless little server. (Admittedly, I’d done terrible things to the server configuration, including twaddling knobs labeled DO NOT TOUCH, but that’s not the point.)

This is not the first spike I’ve gotten; my BSD/wikileaks article dang near went viral. So, another lesson I might learn is: if you write something that’s honestly interesting, people will find you. You really don’t have to break your back promoting it. Lots of writers babble about self-promotion, but most of it is an example of “solving the wrong problem.” Rather than pimping what you’ve written, make your work more interesting.

But that’s too positive for me. I think I’d rather just fear the Puffy.

diskless ubuntu serial console

I’m using Ubuntu servers with qemu-kvm as a virtualization solution. The software included in 10.04LTS includes a variety of annoyances, such as broken PXE, odd bridge behavior, and “general weirdness.” Although 10.10 is not supported in the long term, I decided to try it.

The good news is, the 10.10 virtualization stack works much better. The bad news is, 10.10 didn’t want to run on my diskless hardware. Boot attempts all died with many lines of:

ipconfig: no devices to configure

and a message about killing init. The server was quite explicit that it was dead, and how it was dying, but didn’t leave any clues as to what had killed it. I’m sure that the console showed useful error messages, but they had scrolled off the top of the screen.

The manual says that if you hit shift-PageUp, Ubuntu should page up through the console messages. That should be amended to read “unless init is dead and your keyboard LEDs are blinking slowly but steadily.”

The only way to resolve this problem is to see the error messages that say why the machine crashed. So, a serial console. I want PXE messages, initrd messages, and kernel boot messages sent to serial console. These are all controlled by the /tftpboot/pxelinux.cfg/machine file. The actual file name is the MAC address of the booting NIC.

If you want to get messages from the pxe and initrd boot stages, the pxelinux.cfg file’s first line must include the SERIAL statement. If you want to get console messages from the booting kernel and/or log into the running system over the serial console, you must append a serial statement to the kernel boot command. The end result for a serial console looks like this:

SERIAL 0 115200
LABEL linux
KERNEL vmlinuz-2.6.35-27-server
APPEND root=/dev/nfs initrd=initrd.img-2.6.35-27-server-pxe nfsroot=192.0.2.1:/nfsroot ip=dhcp rw console=tty0 console=ttyS0,115200n8
TIMEOUT 0

The Web site will probably wrap the APPEND statement around, but that line and everything beneath it down to TIMEOUT is a single line.

If you want a serial login in multiuser mode, you need to create a script to activate the terminal. Here’s the Ubuntu default terminal script:
/etc/init/ttyS0.conf

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -L 115200 ttyS0 vt102

The next time you reboot your diskless box, you should have a full serial console.

Some time soon, more on the actual error and how I fixed it.