From f533bcdb2ea36aafaf46835616a476e37dc6f6cf Mon Sep 17 00:00:00 2001 From: Florian Forster Date: Tue, 4 Oct 2016 10:19:48 +0200 Subject: [PATCH] docs/BUILD.dpdkstat.md: Move dpdkstat build instructions out of README. --- README | 137 +----------------------- docs/BUILD.dpdkstat.md | 230 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 231 insertions(+), 136 deletions(-) create mode 100644 docs/BUILD.dpdkstat.md diff --git a/README b/README index bb72ac2c..a1668fab 100644 --- a/README +++ b/README @@ -98,6 +98,7 @@ Features - dpdk Collect DPDK interface statistics. + See docs/BUILD.dpdkstat.md for detailed build instructions. - drbd Collect individual drbd resource statistics. @@ -1020,142 +1021,6 @@ Crosscompiling * `endianflip' (12345678 -> 87654321) * `intswap' (12345678 -> 56781234) -Configuring with DPDK ---------------------- - -Note: DPDK 16.04 is the minimum version and currently supported version of DPDK -required for the dpdkstat plugin. This is to allow the plugin to take advantage -of functions added to detect if the DPDK primary process is alive. - -Note: For Ubuntu, GCC 4.9 is the minimum version required to build collectd -with DPDK. Ubuntu 14.04, for example, has GCC 4.8 by default and will require -an upgrade: - $ sudo add-apt-repository ppa:ubuntu-toolchain-r/test - $ sudo apt-get update - $ sudo apt-get install gcc-4.9 -Alternatively, if you know that the platform that you wish to run collectd -on supports the SSSE3 instruction set, you can run make as follows: - $ make -j CFLAGS+='-mssse3' - -Build DPDK for use with collectd: - To compile DPDK for use with collectd dpdkstat start by: - - Clone DPDK: - $ git clone git://dpdk.org/dpdk - - Checkout the system requirements at - http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html and make sure you have - the required tools and hugepage setup as specified there. - NOTE: It's recommended to use the 1GB hugepage setup for best performance, - please follow the instruction for "Reserving Hugepages for DPDK Use" - in the link above. - However if you plan on configuring 2MB hugepages on the fly please ensure - to add appropriate commands to reserve hugepages in a system startup script - if collectd is booted at system startup time. These commands include: - mkdir -p /mnt/huge - mount -t hugetlbfs nodev /mnt/huge - echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages - - To configure the DPDK build for the combined shared library modify - config/common_base in your DPDK as follows - # - # Compile to share library - # - -CONFIG_RTE_BUILD_SHARED_LIB=n - +CONFIG_RTE_BUILD_SHARED_LIB=y - - Prepare the configuration for the appropriate target as specified at: - http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html. - For example: - $ make config T=x86_64-native-linuxapp-gcc - - Build the target: - $ make - - Install DPDK to /usr - $ sudo make install prefix=/usr - NOTE 1: You must run make install as the configuration of collectd with - DPDK expects DPDK to be installed somewhere. - NOTE 2: If you don't specify a prefix then DPDK will be installed in /usr/local/ - NOTE 3: If you are not root then use sudo to make install DPDK to the appropriate - location. - - Check that the DPDK library has been installed in /usr/lib or /lib - $ ls /usr/lib | grep dpdk - - Bind the interfaces to use with dpdkstat to DPDK: - DPDK devices can be setup with either the VFIO (for DPDK 1.7+) or UIO modules. - Note: UIO requires inserting an out of tree driver igb_uio.ko that is available - in DPDK. - UIO Setup: - - Insert uio.ko: - $ sudo modprobe uio - - Insert igb_uio.ko: - $ sudo insmod $DPDK_BUILD/kmod/igb_uio.ko - - Bind network device to igb_uio: - $ sudo $DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio eth1 - VFIO Setup: - - VFIO needs to be supported in the kernel and the BIOS. More information can be found - @ http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html. - - Insert the `vfio-pci.ko' module: modprobe vfio-pci - - Set the correct permissions for the vfio device: - $ sudo /usr/bin/chmod a+x /dev/vfio - $ sudo /usr/bin/chmod 0666 /dev/vfio/* - - Bind the network device to vfio-pci: - $ sudo $DPDK_DIR/tools/dpdk_nic_bind.py --bind=vfio-pci eth1 - NOTE: Please ensure to add appropriate commands to bind the network - interfaces to DPDK in a system startup script if collectd is - booted at system startup time. - - Run ldconfig to update the shared library cache. - - Build static DPDK library for use with collectd: - - To configure DPDK to build the combined static library libdpdk.a - ensure that CONFIG_RTE_BUILD_SHARED_LIB is set to n in - config/common_base in your DPDK as follows: - # - # Compile to share library - # - CONFIG_RTE_BUILD_SHARED_LIB=n - - Prepare the configuration for the appropriate target as specified at: - http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html. - For example: - $ make config T=x86_64-native-linuxapp-gcc - - Build the target using -fPIC: - $ make EXTRA_CFLAGS=-fPIC -j - - Install DPDK to /usr - $ sudo make install prefix=/usr - -Configure collectd with DPDK: -NOTE: The Address-Space Layout Randomization (ASLR) security feature in Linux should - be disabled, in order for the same hugepage memory mappings to be present in all - DPDK multi-process applications. Note that this has security implications. - To disable ASLR: - $ echo 0 > /proc/sys/kernel/randomize_va_space - To fully enable ASLR: - $ echo 2 > /proc/sys/kernel/randomize_va_space - See http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html - - - Generate the build script as specified below. (i.e. run `build.sh'). - - Configure collectd with the DPDK shared library: - ./configure --with-libdpdk=/usr - NOTE: To configure collectd with the DPDK static library: - ./configure --with-libdpdk=/usr CFLAGS=" -lpthread -Wl,--whole-archive - -Wl,-ldpdk -Wl,-lm -Wl,-lrt -Wl,-lpcap -Wl,-ldl -Wl,--no-whole-archive" - - Libraries: - ... - libdpdk . . . . . . . . yes - - Modules: - ... - dpdkstat . . . . . . .yes - - - - Make sure that dpdk and dpdkstat are enabled in the configuration log: - - - Build collectd: - $ make -j && make -j install. - NOTE: As mentioned above, if you are building on Ubuntu 14.04 with GCC <= 4.8.X, - you need to use: - $ make -j CFLAGS+='-mssse3' && make -j install - -Usage of dpdkstat: - - The same PCI device configuration should be passed to the primary process - as the secondary process uses the same port indexes as the primary. - NOTE: A blacklist/whitelist of NICs isn't supported yet. - Contact ------- diff --git a/docs/BUILD.dpdkstat.md b/docs/BUILD.dpdkstat.md new file mode 100644 index 00000000..b502edd2 --- /dev/null +++ b/docs/BUILD.dpdkstat.md @@ -0,0 +1,230 @@ +# The dpdkstat plugin + +**Data Plane Development Kit** (DPDK) is a set of drivers and libraries for fast +packet processing. + +## Summary + +The *dpdkstat plugin* has the following requirements: + + * DPDK 16.04 or later + * GCC 4.9 or later + +You can also build with GCC 4.8 (e.g. Ubuntu 14.04) if you specify the SSSE3 +instruction set manually: + + make -j CFLAGS+='-mssse3' + +## Building DPDK + + * Setup the build environment: + + Ensure that you have GCC 4.9 or later. Ubuntu 14.04, for example, has GCC + 4.8 by default and requires an upgrade: + + add-apt-repository ppa:ubuntu-toolchain-r/test + apt-get update + apt-get install gcc-4.9 + + If you know that the platform that you wish to run collectd on supports the + SSSE3 instruction set, GCC 4.8 also works if you enable SSSE3 manually: + + make -j CFLAGS+='-mssse3' + + * Clone DPDK: + + git clone git://dpdk.org/dpdk + + * Checkout the [DPDK system + requirements](http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html) and make + sure you have the required tools and hugepage setup as specified there. + + **Note:** It's recommended to use the 1GB hugepage setup for best + performance, please follow the instruction for "Reserving Hugepages for DPDK + Use" in the link above. + + However if you plan on configuring 2MB hugepages on the fly please ensure to + add appropriate commands to reserve hugepages in a system startup script if + collectd is booted at system startup time. These commands include: + + mkdir -p /mnt/huge + mount -t hugetlbfs nodev /mnt/huge + echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages + + * To configure the DPDK build for the combined shared library modify + `config/common_base` in your DPDK as follows + + # + # Compile to share library + # + -CONFIG_RTE_BUILD_SHARED_LIB=n + +CONFIG_RTE_BUILD_SHARED_LIB=y + + * Prepare the configuration for the appropriate target as specified at: + http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html. + + For example: + + make config T=x86_64-native-linuxapp-gcc + + * Build the target: + + make + + * Install DPDK to `/usr` + + sudo make install prefix=/usr + + **Note 1:** You must run make install as the configuration of collectd with + DPDK expects DPDK to be installed somewhere. + + **Note 2:** If you don't specify a prefix then DPDK will be installed in + `/usr/local/`. + + **Note 3:** If you are not root then use sudo to make install DPDK to the + appropriate location. + + * Check that the DPDK library has been installed in `/usr/lib` or `/lib`: + + ls /usr/lib | grep dpdk + + * Bind the interfaces to use with dpdkstat to DPDK: + + DPDK devices can be setup with either the VFIO (for DPDK 1.7+) or UIO + modules. + + **Note:** UIO requires inserting an out of tree driver `igb_uio.ko` that is + available in DPDK. + + **UIO Setup:** + + * Insert `uio.ko`: + + sudo modprobe uio + + * Insert `igb_uio.ko`: + + sudo insmod $DPDK_BUILD/kmod/igb_uio.ko + + * Bind network device to `igb_uio`: + + sudo $DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio eth1 + + **VFIO Setup:** + + * VFIO needs to be supported in the kernel and the BIOS. More information + can be found at: http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html. + * Insert the `vfio-pci.ko` module: + + modprobe vfio-pci + + * Set the correct permissions for the VFIO device: + + sudo /usr/bin/chmod a+x /dev/vfio + sudo /usr/bin/chmod 0666 /dev/vfio/* + + * Bind the network device to `vfio-pci`: + + sudo $DPDK_DIR/tools/dpdk_nic_bind.py --bind=vfio-pci eth1 + + **Note:** Please ensure to add appropriate commands to bind the network + interfaces to DPDK in a system startup script if collectd is booted at + system startup time. + + * Run `ldconfig` to update the shared library cache. + +### Static library + +To build static DPDK library for use with collectd: + + * To configure DPDK to build the combined static library `libdpdk.a` ensure + that `CONFIG_RTE_BUILD_SHARED_LIB` is set to “n” in `config/common_base` in + your DPDK as follows: + + # + # Compile to share library + # + CONFIG_RTE_BUILD_SHARED_LIB=n + + * Prepare the configuration for the appropriate target as specified at: + http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html. + + For example: + + make config T=x86_64-native-linuxapp-gcc + + * Build the target using `-fPIC`: + + make EXTRA_CFLAGS=-fPIC -j + + * Install DPDK to `/usr`: + + sudo make install prefix=/usr + +## Build collectd with DPDK + +**Note:** DPDK 16.04 is the minimum version and currently supported version of +DPDK required for the dpdkstat plugin. This is to allow the plugin to take +advantage of functions added to detect if the DPDK primary process is alive. + + +**Note:** The *Address-Space Layout Randomization* (ASLR) security feature in +Linux should be disabled, in order for the same hugepage memory mappings to be +present in all DPDK multi-process applications. Note that this has security +implications. + + * To disable ASLR: + + echo 0 > /proc/sys/kernel/randomize_va_space + + * To fully enable ASLR: + + echo 2 > /proc/sys/kernel/randomize_va_space + +See also: http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html + + * Generate the build script as specified below. (i.e. run `build.sh`). + * Configure collectd with the DPDK shared library: + + ./configure --with-libdpdk=/usr + +### Build with the static DPDK library + +To configure collectd with the DPDK static library: + + * Run *configure* with the following CFLAGS: + + ./configure --with-libdpdk=/usr CFLAGS=" -lpthread -Wl,--whole-archive -Wl,-ldpdk -Wl,-lm -Wl,-lrt -Wl,-lpcap -Wl,-ldl -Wl,--no-whole-archive" + + * Make sure that dpdk and dpdkstat are enabled in the *configure* output. + + Expected output: + + Libraries: + ... + libdpdk . . . . . . . . yes + + Modules: + ... + dpdkstat . . . . . . .yes + + * Build collectd: + + make -j && make -j install. + + **Note:** As mentioned above, if you are building on Ubuntu 14.04 with + GCC <= 4.8.X, you need to use: + + make -j CFLAGS+='-mssse3' && make -j install + +## Caveats + + * The same PCI device configuration should be passed to the primary process as + the secondary process uses the same port indexes as the primary. + * A blacklist / whitelist of NICs isn't supported yet. + +## License + +The *dpdkstat plugin* is copyright (c) 2016 *Intel Corporation* and licensed +under the *MIT license*. Full licensing terms can be found in the file +`COPYING`. -- 2.30.2