[olug] 2.4 kernel/ SMP and APIC

[VincentR]

This is mostly an FYI thing for thought...
There's a little used feature in the 2.4 kernel which enables you to bind an
IRQ to a specific CPU.  Why would you do this?  In my case it was because
UDP packets in a stream were getting out of sequence due to the APIC
allowing multiple CPU's to service interrupts on the same ethernet device.
You could just turn off APIC with boot loader options (append="noapic"), but
that would disable most of the advantages of an SMP system.  You could also
increase the number of packets serviced within an interrupt if your driver
allows that configuration in modules.conf (options 3c59x
max_interrupt_work=60).  That helps, but it doesn't fix the problem.
Here's an init script to set the smp_affinity for each network device (word
wrap will probably screw it up):

[root at --- /root]# cat /etc/init.d/eth_affinity
#!/bin/bash
#
# eth_affinity
#
# chkconfig: 345 15 85
# description: Set IRQ affinity for eth devices.

# Define which networks get which CPU.
# You must use the first three octets.
CPU0="10.100.116 10.5.7 10.5.9 10.129.7 10.0.9"
CPU1="10.10.10 10.10.11 10.10.0"

# Where is /proc?
PROC=/proc
IPV4=/proc/sys/net/ipv4

# Got root?
if [ `id -u` -gt "0" ]; then
        echo "You are not root!"
        exit 1
fi
# Check kernel version.
if [ ! `uname --release | cut -d. -f1-2` = "2.4" ]; then
        echo "You are not running a 2.4 kernel"
        echo "Version detected: "`uname --release`
        exit 1
fi
# Is this an SMP kernel?
if [ ! -f /proc/1/cpu ]; then
        echo "Get an SMP system to do this on!"
        exit 1
fi
# Check for the required files and directories.
if [ ! -d $PROC/irq ] || [ ! -d $IPV4/conf ]; then
        echo "I couldn't find a required file or directory; Exiting..."
        file $PROC/irq $IPV4/conf
        exit 1
fi

#DEBUG SECTION##################################################
# Set DEBUG=1 to test this in /tmp.
DEBUG=0
if [ "$DEBUG" = "1" ]; then
        PROC=/tmp
        IPV4=/tmp/ipv4
fi
# Copy files to /tmp if needed.
if [ ! -d $PROC/irq ] || [ ! -d $IPV4/conf ]; then
        cp -a /proc/irq /tmp
        cp -a /proc/sys/net/ipv4 /tmp > /dev/null 2>&1
fi
#DEBUG SECTION##################################################

# Here we go...
case "$1" in
  start)
        # Find each eth's irq and network; set the affinity.
        for ETH in `/bin/ls $IPV4/conf | egrep -v all\|default\|lo`; do
                IRQ=`ifconfig $ETH | grep Interrupt | awk -F: '{print $2}' |
awk '{print $1}'`
                NET=`ifconfig $ETH | grep "inet addr" | awk -F: '{print $2}'
| cut -d. -f1-3`
                for net in `echo $CPU0`; do
                        if [ "$NET" = "$net" ]; then
                                echo 1 > $PROC/irq/$IRQ/smp_affinity
                        fi
                done
                for net in `echo $CPU1`; do
                        if [ "$NET" = "$net" ]; then
                                echo 2 > $PROC/irq/$IRQ/smp_affinity
                        fi
                done
        done
        touch /var/lock/subsys/eth_affinity
        $0 status
        ;;
  stop)
        # Find each eth's irq; unset the affinity.
        for ETH in `/bin/ls $IPV4/conf | egrep -v all\|default\|lo`; do
                IRQ=`ifconfig $ETH | grep Interrupt | awk -F: '{print $2}' |
awk '{print $1}'`
                echo ffffffff > $PROC/irq/$IRQ/smp_affinity
        done
        rm -f /var/lock/subsys/eth_affinity
        $0 status
        ;;
  status)
        # Find each eth's irq and network; display the affinity.
        for ETH in `/bin/ls $IPV4/conf | egrep -v all\|default\|lo`; do
                IRQ=`ifconfig $ETH | grep Interrupt | awk -F: '{print $2}' |
awk '{print $1}'`
                NET=`ifconfig $ETH | grep "inet addr" | awk -F: '{print $2}'
| cut -d. -f1-3`
                AFFINITY=`cat $PROC/irq/$IRQ/smp_affinity`
                if [ "$AFFINITY" = "00000001" ]; then
                        echo "$ETH IRQ $IRQ has affinity with CPU0 on
network $NET."
                fi
                if [ "$AFFINITY" = "00000002" ]; then
                        echo "$ETH IRQ $IRQ has affinity with CPU1 on
network $NET."
                fi
                if [ "$AFFINITY" = "ffffffff" ]; then
                        echo "$ETH IRQ $IRQ has affinity with ALL on network
$NET."
                fi
                if [ ! "$AFFINITY" = "00000001" ] && [ ! "$AFFINITY" =
"00000002" ] && [ ! "$AFFINITY" = "ffffffff" ]; then
                        echo -e "$ETH IRQ $IRQ network $NET has an affinity
of $AFFINITY."
                        echo "I have no idea what this means!"
                fi
        done
        ;;
  *)
        echo "Usage: eth_affinity {start|stop|status}"
        exit 1
esac
#EOF