ifup-local on bridge members on CentOS

I run a bunch of CentOS 6 physical servers as QEMU virtualization devices. These hosts have two NICs, one for management and one for virtual machine bridges.

When you use Linux for virtualization, it’s important to increase the amount of memory for network transmit and receive buffers. You also need to disable GSO and TSO, to improve performance and to avoid gigabytes of kernel error messages every day. You can do this with ethtool(8). First, let’s check the existing ring sizes.

# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX: 16384
RX Mini: 0
RX Jumbo: 0
TX: 16384
Current hardware settings:
RX: 512
RX Mini: 0
RX Jumbo: 0
TX: 512

Similarly, use ethtool -k eth0 to check GSO and TSO settings.

The card is using much less memory than it can. When you have a bunch of virtual machines pouring data through the card, you want the card to work as efficiently as possible. Fixing this on a running system is easy enough:

# ethtool -G eth0 tx 16384 rx 16384
# ethtool -K eth0 gso off tso off

Repeat the process for eth1.

How do you make this happen automatically at boot? Adding the commands to /etc/rc.local isn’t reliable. By the time the system gets that much stuff running, the ethtool command might fail with a “Cannot allocate memory” error. If you try again it’ll probably work, but it’s not deterministic. And I’m against running a single command four times in rc.local in the hopes that one of them will work.

Enter /sbin/ifup-local. CentOS runs this script after bringing up an interface, with the interface name as an argument. The problem is, it doesn’t run this script on bridge member interfaces. We can adjust eth0 and br0 at boot just fine, but eth1 (the physical interface underlying br0) doesn’t get run.

You can’t run ethtool -G eth0 tx 16384 rx 16384 on br0. Interface br0 doesn’t have any transmit or receive rings. It’s a logical interface. You can disable TSO and GSO on br0, but that won’t disable it on eth1. You can’t wait to reconfigure eth1 in rc.local until the system is running, because increasing the memory doesn’t always work once the system is running full-out multiuser. And Red Hat says this is by design. Apparently network bridges on CentOS/Red Hat are supposed to perform poorly. That’s good to know.

So, what to do?

I adjust the eth1 ring size in ifup-local when bringing up br0, but before any processes send any traffic over the bridge. My /sbin/ifup-local looks like this:

#!/bin/bash

esac
exit 0

This appears to work consistently. Of course, the values for the NIC need to be set on a per-machine basis. I have Ansible do that work for me.

Hopefully, this will save someone else the pain I’ve been through trying to make this work…