Profiling and debugging MiNiFi C++

[#1]

This article will cover debugging MiNiFi C++. This will be the first of a series of debugging, memory leak detection, and profiling MiNiFi C++. In my development of MiNiFi C++ I spent the bulk of my time using common GNU tools for debugging and ensuring we don't have memory leaks. Profiling allows us to baseline. This guide will focus on debugging using the GNU Debugger, GDB

Debugging

MiNiFi C++ uses CMAKE to generate builds. A precondition for debugging C++ tools is including debug symbols into the executable. To do this you will need to specify a build type of DEBUG or RELWITHDEBINFO, both of which instruct CMAKE to include those symbols.

$ cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..$ cmake -DCMAKE_BUILD_TYPE=DEBUG ..

When you do this, CMAKE will generate compilation code to include debug symbols (-g). When you build the project debug symbols will be included, and allow you to run GDB. GDB is the GNU debugger. When you've installed MiNiFi you can debug minifi in place by running:

gdb ./minifi
GNU gdb (GDB 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.

This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
Reading symbols from ./minifi...done.
(gdb) run

This will begin the debug session. To set a breakpoint on a piece of code, such as Site2SiteClientProtocol.cpp, you can use the following command to set the breakpoint on your desired line. In this example I'm setting a breakpoint on the function named initiateResourceNegoation of Site2SiteClientProtocol.

(gdb) break Site2SiteClientProtocol.cpp:87
   Breakpoint 1 at 0x5614db: file /opt/marc/code/nifi-minifi-cpp/libminifi/Site2SiteClientProtocol.cpp, line 87
(gdb) run
[Switching to Thread 0x7ffff83fff700 (LWP 7127)]
Thread 42 "minifi" hit Breakpoint 1 org::apache::nifi::minifi::Site2SiteClientProtocol::initiateResourceNegotiation at /opt/marc/code/nifi-minifi-cpp/libminifi/Site2SiteClientProtocol.cpp:87  
       int ret = peer_->write(_currentVersion); 

If at this point you wish to print the value of _currentVersion you can use the print command. You can see from the example output, below, that we have an unsigned integer _currentVersion whose value is 5.

(gdb) print _currentVersion$1 = 5(gdb) ptype _currentVersion
type = unsigned int

You can inspect the stack with commands such as bt ( or backtrace) to print the backtrace of active functions in the execution stack. or frame <number> if you want to print the stack frame. In many cases, if a signal occurs the interrupts execution you may be able to use backtrace to get the current location whilst inspecting variables that may help lead to the problem.

Typical problems that occur are segmentation faults, bus errors, or stack corruption. Segmentation faults are commonly caused by accessing addresses outside of the assigned address space. An example might be null pointers in which your variables are attempting to access 0x00 ( nullptr/NULL ). In this case, printing variables or the stack frame may be helpful in finding which variable is null.

In the case of bus errors, this is typically a misaligned instruction pointer or invalid pointer access. This will likely occur because of pointer mismanagement. In these cases you may need to use GDB to print variables. Pointer corruption is better avoided by using smart pointers since we are using C++11; however, in the event that raw pointers are needed, GDB can be very helpful in locating memory issues.

In many cases, stack corruption can manifest as a general segfault. One common cause is calling a bogus function pointer, as your program counter ( PC ) will have a bogus value. Since we're using C++ you may see issues as a result of bogus virtual function calls. In these cases, Valgrind may be a better tool to debug the situation. To do this simply run the following command.

valgrind ./minifi

When the error is reached Valgrind will provide the offending lines along with the address of the function pointer. Verify that this is expected and/or trace the output with gdb to identify if the value is expected. Using our example above with function initiateResourceNegotation you can locate the address of a function. The info command (help info for provide more information about this command) will allow you to get systematic information about symbols.

(gdb) info address org::apache::nifi::minifi::Site2SiteClientProtocol::initateResourceNegotiation

Symbol "org::apache::nifi::minifi::Site2SiteClientProtocol::initateResourceNegotiation" is a function at address 0x55a950. 

Info address will provide the address of where the function exists in memory. In this case you can correlate the function symbol address with your function pointer found in Valgrind. With this information you should be able to track down the invalid function pointer and alleviate your source of stack corruption.

Conclusion

This quick how to discusses some common commands to debug the three most commons issues I've come across. No guide can be an exhaustive tutorial, please comment with requests and I will attempt to demonstrate by example using MiNiFi C++. I will attempt to discuss common usage of Valgrind in the next how-to. While Valgrind is often used for memory leak detection it can also be used to compliment GDB to identify memory access errors. I like to use both tools since Valgrind strong suit is providing contextual errors in memory access in addition to memory leak detection.