1 # The dpdkstat plugin
3 **Data Plane Development Kit** (DPDK) is a set of drivers and libraries for fast
4 packet processing.
6 ## Summary
8 The *dpdkstat plugin* has the following requirements:
10 * DPDK 16.04 or later
11 * GCC 4.9 or later
13 You can also build with GCC 4.8 (e.g. Ubuntu 14.04) if you specify the SSSE3
14 instruction set manually:
16 make -j CFLAGS+='-mssse3'
18 ## Building DPDK
20 * Setup the build environment:
22 Ensure that you have GCC 4.9 or later. Ubuntu 14.04, for example, has GCC
23 4.8 by default and requires an upgrade:
25 add-apt-repository ppa:ubuntu-toolchain-r/test
26 apt-get update
27 apt-get install gcc-4.9
29 If you know that the platform that you wish to run collectd on supports the
30 SSSE3 instruction set, GCC 4.8 also works if you enable SSSE3 manually:
32 make -j CFLAGS+='-mssse3'
34 * Clone DPDK:
36 git clone git://dpdk.org/dpdk
38 * Checkout the [DPDK system
39 requirements](http://dpdk.org/doc/guides/linux_gsg/sys_reqs.html) and make
40 sure you have the required tools and hugepage setup as specified there.
42 **Note:** It's recommended to use the 1GB hugepage setup for best
43 performance, please follow the instruction for "Reserving Hugepages for DPDK
44 Use" in the link above.
46 However if you plan on configuring 2MB hugepages on the fly please ensure to
47 add appropriate commands to reserve hugepages in a system startup script if
48 collectd is booted at system startup time. These commands include:
50 mkdir -p /mnt/huge
51 mount -t hugetlbfs nodev /mnt/huge
52 echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
54 * To configure the DPDK build for the combined shared library and enable autoload
55 of pmd drivers modify `config/common_base` in your DPDK as follows
57 #
58 # Compile to share library
59 #
60 -CONFIG_RTE_BUILD_SHARED_LIB=n
61 +CONFIG_RTE_BUILD_SHARED_LIB=y
63 # Default driver path (or "" to disable)
64 -CONFIG_RTE_EAL_PMD_PATH=""
65 +CONFIG_RTE_EAL_PMD_PATH="/usr/lib/dpdk-pmd/"
67 * Prepare the configuration for the appropriate target as specified at:
68 http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html.
70 For example:
72 make config T=x86_64-native-linuxapp-gcc
74 * Build the target:
76 make
78 * Install DPDK to `/usr`
80 sudo make install prefix=/usr
82 * Create dpdk-pmd folder
84 mkdir -p /usr/lib/dpdk-pmd
86 * Create symlinks to pmd drivers
88 find /usr/lib -type f -name 'librte_pmd*' | while read path ; do ln -s $path /usr/lib/dpdk-pmd/`echo $path | grep -o 'librte_.*so'` ; done
90 **Note 1:** You must run make install as the configuration of collectd with
91 DPDK expects DPDK to be installed somewhere.
93 **Note 2:** If you don't specify a prefix then DPDK will be installed in
94 `/usr/local/`.
96 **Note 3:** If you are not root then use sudo to make install DPDK to the
97 appropriate location.
99 **Note 4:** You **MUST** create symlink to a NIC driver lib. This way collectd
100 will be able to work with device bound to dpdk.
102 * Check that the DPDK library has been installed in `/usr/lib` or `/lib`:
104 ls /usr/lib | grep dpdk
106 * Bind the interfaces to use with dpdkstat to DPDK:
108 DPDK devices can be setup with either the VFIO (for DPDK 1.7+) or UIO
109 modules.
111 **Note:** UIO requires inserting an out of tree driver `igb_uio.ko` that is
112 available in DPDK.
114 **UIO Setup:**
116 * Insert `uio.ko`:
118 sudo modprobe uio
120 * Insert `igb_uio.ko`:
122 sudo insmod $DPDK_BUILD/kmod/igb_uio.ko
124 * Bind network device to `igb_uio`:
126 sudo $DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio eth1
128 **VFIO Setup:**
130 * VFIO needs to be supported in the kernel and the BIOS. More information
131 can be found at: http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html.
132 * Insert the `vfio-pci.ko` module:
134 modprobe vfio-pci
136 * Set the correct permissions for the VFIO device:
138 sudo /usr/bin/chmod a+x /dev/vfio
139 sudo /usr/bin/chmod 0666 /dev/vfio/*
141 * Bind the network device to `vfio-pci`:
143 sudo $DPDK_DIR/tools/dpdk_nic_bind.py --bind=vfio-pci eth1
145 **Note:** Please ensure to add appropriate commands to bind the network
146 interfaces to DPDK in a system startup script if collectd is booted at
147 system startup time.
149 * Run `ldconfig` to update the shared library cache.
151 ### Static library
153 To build static DPDK library for use with collectd:
155 * To configure DPDK to build the combined static library `libdpdk.a` ensure
156 that `CONFIG_RTE_BUILD_SHARED_LIB` is set to ānā in `config/common_base` in
157 your DPDK as follows:
159 #
160 # Compile to share library
161 #
162 CONFIG_RTE_BUILD_SHARED_LIB=n
164 * Prepare the configuration for the appropriate target as specified at:
165 http://dpdk.org/doc/guides/linux_gsg/build_dpdk.html.
167 For example:
169 make config T=x86_64-native-linuxapp-gcc
171 * Build the target using `-fPIC`:
173 make EXTRA_CFLAGS=-fPIC -j
175 * Install DPDK to `/usr`:
177 sudo make install prefix=/usr
179 ## Build collectd with DPDK
181 **Note:** DPDK 16.04 is the minimum version and currently supported version of
182 DPDK required for the dpdkstat plugin. This is to allow the plugin to take
183 advantage of functions added to detect if the DPDK primary process is alive.
186 **Note:** The *Address-Space Layout Randomization* (ASLR) security feature in
187 Linux should be disabled, in order for the same hugepage memory mappings to be
188 present in all DPDK multi-process applications. Note that this has security
189 implications.
191 * To disable ASLR:
193 echo 0 > /proc/sys/kernel/randomize_va_space
195 * To fully enable ASLR:
197 echo 2 > /proc/sys/kernel/randomize_va_space
199 See also: http://dpdk.org/doc/guides/prog_guide/multi_proc_support.html
201 * Generate the build script as specified below. (i.e. run `build.sh`).
202 * Configure collectd with the DPDK shared library:
204 ./configure --with-libdpdk=/usr
206 ### Build with the static DPDK library
208 To configure collectd with the DPDK static library:
210 * Run *configure* with the following CFLAGS:
212 ./configure --with-libdpdk=/usr CFLAGS=" -lpthread -Wl,--whole-archive -Wl,-ldpdk -Wl,-lm -Wl,-lrt -Wl,-lpcap -Wl,-ldl -Wl,--no-whole-archive"
214 * Make sure that dpdk and dpdkstat are enabled in the *configure* output.
216 Expected output:
218 Libraries:
219 ...
220 libdpdk . . . . . . . . yes
222 Modules:
223 ...
224 dpdkstat . . . . . . .yes
226 * Build collectd:
228 make -j && make -j install.
230 **Note:** As mentioned above, if you are building on Ubuntu 14.04 with
231 GCC <= 4.8.X, you need to use:
233 make -j CFLAGS+='-mssse3' && make -j install
235 ## Caveats
237 * The same PCI device configuration should be passed to the primary process as
238 the secondary process uses the same port indexes as the primary.
239 * A blacklist / whitelist of NICs isn't supported yet.
241 ## License
243 The *dpdkstat plugin* is copyright (c) 2016 *Intel Corporation* and licensed
244 under the *MIT license*. Full licensing terms can be found in the file
245 `COPYING`.