Measuring code coverage with gcov

Last edited on

Introduction

It is possible to use the GNU code coverage tool, gcov, with Squish. (ℹ️ Note that this is only really relevant to gcc users.)

The way Squish instruments an application for testing does not interfere with how gcov operates. The two tools might well be used side by side to gather information about which parts of your code are exercised during test execution.

Squish currently doesn't have an any special support for gcov, yet gcov can still easily be used with Squish as described below. We hope that a future Squish release will include add-ons to support gcov and possibly some other profiling tools, to make their use with Squish even easier.

Here's a short overview. Those who have used gcov before will already be familiar with the concept.

Preparation

First you must recompile your application with the additional compiler flags needed to prepare the application for the coverage analysis. The relevant flags are -fprofile-arcs and -ftest-coverage. Add them to the CXXFLAGS variable in your Makefile or wherever the compiler flags are defined in your build system. (⚠️ And remember to remove them again if you do a release build since they will bloat the binary and slow down execution.)

For the sake of example we will use the Qt 4 addressbook example that is supplied with Squish.

First add {{"-fprofile-arcs -ftest-coverage"}} to the {{config["CXXFLAGS"]}} statement in the {{Build.conf}} file. Then recompile the addressbook example with the new flags

$ ./build -l -C examples/addressbook clean
$ ./build -l -C examples/addressbook

For this example we will take a look at the state of code coverage in the application's main() function before actually running the application

$ cd examples/addressbook
$ gcov main.cpp

This will produce some output that we can safely ignore — except for the line immediately before the "Creating main.cpp.gcov." message. This line says "0.00% of 10 source lines executed ...", which is expected. When we open up the main.cpp.gcov file we'll see that each line of code is prefixed by "######" characters which means that the respective line has not been executed yet.

int main( int argc, char ** argv )
      ######    {
      ######        QApplication a( argc, argv );

      ######        ABMainWindow *mw = new ABMainWindow();
      ######        mw->setCaption( "Qt Example - Addressbook" );
      ######        a.setMainWidget( mw );
      ######        mw->show();
      ....

Execution

Let us now execute a single test case from the sample addressbook test suite using the command line:

$ squishserver &
$ squishrunner --testsuite ../suite_addressbook --testcase tst_add_address
$ squishserver --stop

Here we start the squishserver, run a single test case in one test suite and then stop the squishserver. We could just as easily have run the test from the IDE, but doing it on the command line shows what has to be done when we put it into a shell script or batch file for automatic processing.

Analysis

Running

$ gcov main.cpp

will now result in "100.00% of 10 source lines executed". Inspection of main.cpp.gcov will confirm that every line of code has been executed exactly once.

int main( int argc, char ** argv )
           1    {
           1        QApplication a( argc, argv );

           1        ABMainWindow *mw = new ABMainWindow();
           1        mw->setCaption( "Qt Example - Addressbook" );
           1        a.setMainWidget( mw );
           1        mw->show();
           ....

Far more interesting than the coverage of main.cpp is the analysis of files like mainwindow.cpp. So let's see what the coverage for that file is:

$ gcov mainwindow.cpp

As the output will show, only some of the code has been executed, which means that the test we ran did not provide 100% code coverage. This isn't surprising because the test is very short and simple. And in fact running the entire test suite still won't produce 100% coverage because between them the tests don't execute every line of code in the application.

However, by using gcov to monitor the code coverage our test suite produces, we can identify lines that haven't been executed and perhaps add new tests or extend existing tests to increase the coverage.

Each use of gcov accumulates coverage data. To start afresh simply delete the *.da files that gcov produces.

Caveats

gcov generally writes out the coverage data when the program under test exits. Depending on how much coverage data needs to be written, this can take some time. In some situations, this may collide with safety mechanisms built into Squish which cause Squish to terminate the AUT if it appears to be stuck. In such cases, Squish may end up terminating the AUT process by force while it is in the middle of writing out coverage data. The symptom of this are malformed data files being written out and/or the AUT process crashing.

A simple approach for mitigating this risk is to keep Squish from starting and stopping the application itself. Instead, launch the application manually and then attach and detach to it from the test script. See Attaching to running Qt applications for more on this topic.

Conclusion

The combination of Squish and gcov allows us to monitor which lines of code are exercised by our test suites.

This requires the use of two extra gcc switches to prepare the AUT for coverage analysis. After using Squish to execute one or multiple test cases we can view the detailed generated coverage reports by means of the gcov tool.

Even if we achieve 100% code coverage this does not necessarily mean that our testing effort is complete! This is because even if every line of code is executed, we are only testing with one set of data — we may need to add additional tests or test with additional data so that every part of the application gets tested with a proper range of representative data.