in Code, Cool Tools, HowTo

Code Coverage Of Unit Tests Using Qt 5 On macOS

Qt Library

Qt Framework

I was inspired while watching a talk by Kevin Ottens about refactoring OpenGL code in Qt to take a look at gcov & lcov. gcov is used to analyze code coverage – which lines of code have actually been executed while running an application. lcov is a tool that can produce HTML reports from the gcov output.

If you have a suite of unit tests that you run on your code, you can use these tools to see which the lines of code are covered by your tests and which are not.

I couldn’t find a decent primer on how to set this up properly for my Qt projects on macOS so I could run it from Qt Creator, so I thought I’d write up how I did it.

The example I’m using here is available on GitHub.

Versions

Before I get started, here are the versions of various tools I used:

  • Qt – 5.8 dev branch from git
  • Qt Creator IDE – 4.2.0
  • clang/gcov – Apple LLVM 7.0.2 (clang-700.1.81) (built-in)
  • lcov – 1.13 (installed using homebrew)

The Example

First let’s take a look at the simple example we’re going to work with.

This is just a simple class with one method which takes a QVector of int, adds them, and returns the result. Note that there are a couple of special cases – one for an empty vector and one for a specific sum.

Test Harness

Next, let’s take a look at the testing code.

The asmTestSuite class is used as a convenience. All it does is add the derived class to a static list of tests to run when the derived class is instantiated. Each of our test suites will derive from this class.

Then in main.cpp, we simply run each test suites, keep track of how many failed, and return that number:

Unit Tests

Finally we have a test suite for our asmExampleClass.

For each test we want to run, we would add a new method named test_something. The Qt test library will automatically call all signals that are named like this. For details about writing tests for Qt, please see the Qt Test Overview.

Running The Tests

Now that we have a complete system for setting up and running unit tests, let’s take a look at the output when we run it.

Great! Our test passed. Ship it!

What does it look like if it fails? Here is what happens if I change the QCOMPARE to check for 42 instead of 128:

It gives us some useful information about which test failed, what the actual vs. expected results were, and where to find the test. This makes it easy to track down what’s going wrong.

Code Coverage

Now that we can run the unit tests, let’s look at a way to improve our coverage.

If we look at the asmExampleClass::addSomeStuff() method that’s being tested in this example, we can see that our unit tests don’t cover all paths – for example if you pass in an empty QVector. This is obvious in our simple example, but if you have a large, complex code base this can be difficult to spot. That’s where the code coverage tools come into play. It will help us by identifying which lines are not being executed in the tests.

Code Coverage Setup

Set up is relatively straightforward (once you know how!). It involves three things:

  1. changing our compile & link flags
  2. adding a script to run our code coverage tools
  3. changing Qt Creator’s run command to run the script

Compiler & Linker flags

With clang, the only flag we need to add is --coverage. The key is that it needs to be added for both the compiler and the linker. If you fail to add it as a linker flag you will run into link errors that look something like this:

We add the coverage flag in our qmake .pro file like this:

Coverage Shell Script

I wrote a little shell script to process the coverage information and open the results in a browser.

There are a couple of things to point out in this script that you might need to change for your own project. The first is the path to the commands (lines 5-6) may need to be added if they are not in your PATH environment variable. You might also want to change browsers – I have it set up for Chrome.

The other thing that you might want to change is the list of files which are ignored. Line 18 has a list of patterns of files to ignore – I’ve excluded the Qt frameworks, some Xcode stuff, the test harness code, and all the moc files.

Qt Creator Run Command

The final piece of the puzzle is to execute the coverage shell script whenever we run the test in Qt Creator. For this we need to modify the run command in our project settings.

Using the Projects tab on the left, we select the Run section from the kit we are using.

Qt Creator Run Settings

Qt Creator Run Settings

In the Run section, we set the Command Line Arguments:

> output.log && (%{sourceDir}/test/scripts/runCoverage.sh ./)

What this does is redirect the output of the tests to a log file (in the build directory) and run our runCoverage.sh script if there weren’t any errors. If you’ve put the script in a different place, you will have to modify the path here.

Because running the code coverage tools can take a while, you may want to set up two run commands – one for running the regular tests without coverage and one for running them with coverage. You can use the Add dropdown to create a new run configuration.

Final Results

Now that all of that is in place, when we run the testing program, it will run the tests, process the coverage results, create an HTML report, and open it in your browser.

lcov Code Coverage Results

lcov Code Coverage Results

This shows how many lines of code and functions there are in the sample, and how many were executed.

Drilling down into the actual code file, we can see the lines that were not executed:

lcov Code Coverage - File

lcov Code Coverage – File

We forgot to test a couple of cases. If we go back to asmExampleClass_test.cpp and add a new test to check for the empty QVector:

and re-run our code coverage, we can see that we’ve now tested that code path.

lcov Code Coverage - File #2

lcov Code Coverage – File #2

Repeat until you’ve found, tested, and corrected your edge cases and you’re happy with your coverage.

Go forth and test!

Test All The Things

Write a Comment

Comment

  1. In my case, my .h and .cpp are in separate folders and maybe this is the reason why the coverage report is showing only the headers. Do you know if there’s a different setup for this situation?

    • Unfortunately I don’t have time to experiment with this right now, but I would look at the options to lconv.

      There are several related to directories – maybe you need to pass in other paths somehow so it can find the files? –base-directory?

  2. Thanks for this. I sometimes find inexplicable GUI windows crashes in Qt difficult to trace down. Your code has made that process a much easier and a far less odious task. Thanks again!!