Philipp's Computing Blog

Success is about speed and efficiency

Intel Gigabit ET Dual Port Server Adapter

Details about how a recent Linux sees the card: Intel Gigabit ET Dual Port Server Adapter on Arch Linux (Kernel v3.16)

NIC Bonding on Arch Linux

Simple bonding setup as described in the Arch Wiki:

sudo pacman -Sy ifenslave
cp /etc/netctl/examples/bonding /etc/netctl/bonding
vim /etc/netctl/bonding
# -------- content: ----------
Description="Bonded interface of PRO/1000 ET Server Adapter"
Interface=bond0
Connection=bond
BindsToInterfaces=(enp4s0f0 enp4s0f1)
#IP=static
#Address="10.0.0.1/8"
IP=dhcp
IP6=stateless
# ----------------------------
netctl switch-to bonding
# but somehow an IP on a single NIC was left over. So I did:
netctl stop bonding
systemctl stop dhcpcd@enp4s0f0.service
ip link set dev enp4s0f0 down
netctl switch-to bonding
cat /proc/net/bonding/bond0

A more elaborate setup using LACP (802.3ad) mode:

sudo pacman -Sy ifenslave
sudo modprobe bonding mode=802.3ad miimon=100
# ↑ This will let the network interface bond0 appear
sudo ip link set dev bond0 addr 00:80:c8:e7:ab:5c
## Actually it makes sense to set the IP statically here:
#sudo ip addr add 192.168.100.33/24 brd + dev bond0
sudo ip link set dev bond0 up
sudo ifenslave bond0 enp4s0f0 enp4s0f1
cat /proc/net/bonding/bond0
ip addr
## If no IP was set before, we can get one via DHCP (and DHCPv6):
sudo dhcpcd bond0
ip addr
# Just for fun, disable our other (onboard) NIC:
sudo ip link set dev eno1 down
ping google.de
# re-enable the onboard NIC but without setting a
# default gateway and with a high metric:
sudo dhcpcd --nogateway --metric 250 eno1

or with SYSFS:

sudo modprobe bonding
# ↑ This should let the network interface bond0 appear
# automatically so we don't need the following line:
#echo "+bond0" >  /sys/class/net/bonding_masters
echo 802.3ad > /sys/class/net/bond0/bonding/mode
echo 100 > /sys/class/net/bond0/bonding/miimon
echo "+enp1s0f0" > /sys/class/net/bond0/bonding/slaves
echo "+enp1s0f1" > /sys/class/net/bond0/bonding/slaves
cat /proc/net/bonding/bond0
## We can get an IP via DHCP (and DHCPv6):
sudo dhcpcd bond0
ip addr
# Just for fun, disable our other (onboard) NIC:
sudo ip link set dev eno1 down
ping google.de
# re-enable the onboard NIC but without setting a
# default gateway and with a high metric:
sudo dhcpcd --nogateway --metric 250 eno1

Status output of /proc/net/bonding/bond0

To tear it down:

sudo ifenslave bond0 -d enp1s0f0 enp1s0f1
sudo ip link set bond0 down
sudo modprobe -r bonding

or with SYSFS:

sudo ip link set bond0 down
echo "-enp1s0f0" > /sys/class/net/bond0/bonding/slaves
echo "-enp1s0f1" > /sys/class/net/bond0/bonding/slaves
echo "-bond0" >  /sys/class/net/bonding_masters

To set this up permanently (not finished yet):

# content of /etc/modprobe.d/bonding.conf :
options bonding miimon=100
options bonding mode=802.3ad
#options bonding mode=active-backup
#options bonding primary=enp4s0f0

More resources:

More ideas (for the 'active-backup' mode: Wired to wireless failover, or Howto: active-backup bonding, or ArchLinuxでネットワーク設定 (ブリッジ、bonding) )

SR-IOV

SR-IOV is a specification that allows a PCIe device to appear to be multiple separate physical PCIe devices. Intel NICs for the server market – for example – advertise their SR-IOV capabilities to improve networking speed in virtualized guest systems.

Intel NICs supporting SR-IOV include those based on the chipset Intel 82576 Gigabit Ethernet Controller used on the Intel Gigabit ET Dual Port Server Adapter. The network driver has to support SR-IOV and you can enable the capabilities by setting a parameter when loading the module using the modprobe command. For the driver igb, this would be max_vfs=7. Here are the steps necessary:

First, we chack if support for SR-IOV is present:

At first, there was no output here for me and when trying to use the modprobe command below, I got SR-IOV: bus number out of range. Adding pci=assign-busses to the kernel boot line and rebooting helped.
The next step is to load the kernel module for the NIC again:

modprobe -r igb
modprobe igb max_vfs=8
# Newer kernels will say
# "Enabling SR-IOV VFs using the module parameter is
#  deprecated - please use the pci sysfs interface."
# So, instead you can do:
echo 7 > /sys/bus/pci/devices/0000\:04\:00.0/sriov_numvfs
echo 7 > /sys/bus/pci/devices/0000\:04\:00.1/sriov_numvfs
# and check with:
cat /sys/bus/pci/devices/0000\:04\:00.0/sriov_numvfs
cat /sys/bus/pci/devices/0000\:04\:00.1/sriov_numvfs

The output of dmesg following those commands can be seen here.

Resources for SR-IOV:

Resources