Guidelines and HOWTOs/UnitTests

Author: Brad Hards, Sigma Bravo Pty Limited

Abstract

This article provides guidance on writing unittests for Qt4 and KDE4, based on the QtTestLib framework provided with Qt4.1. It provides an introduction to the ideas behind unit testing, tutorial material on the QtTestLib framework, and suggestions for getting the most value for your effort. This document is matched to Qt 4.1 and KDE 4.

About Unit Testing

A unit test is a test that checks the functionality, behaviour and correctness of a single software component. In Qt4 code (including KDE4 code) unit tests are almost always used to test a single C++ class (although testing a macro or C function is also possible).

Unit tests are a key part of Test Driven Development, however they are useful for all software development processes. It is not essential that all of the code is covered by unit tests (although that is obviously very desirable!). Even a single test is a useful step to improving code quality.

Even if they don't call them "unit tests", most programmers have written some "throwaway" code that they use to check an implementation. If that code was cleaned up a little, and built into the development system, then it could be used over and over to check that the implementation is still OK. To make that work a little easier, we can use test frameworks.

Note that it is sometimes tempting to treat the unit test as a pure verification tool. While it is true that unit tests do help to ensure correct functionality and behaviour, they also assist with other aspects of code quality. Writing a unit test requires a slightly different approach to coding up a class, and thinking about what inputs need to be tested can help to identify logic flaws in the code (even before the tests get run). In addition, the need to make the code testable is a very useful driver to ensure that classes do not suffer from close coupling.

Anyway, enough of the conceptual stuff - lets talk about a specific tool that can reduce some of the effort and let us get on with the job.

About QtTestLib

QtTestlib is a lightweight testing library developed by Trolltech and released under the GPL (a commercial version is also available, for those who need alternative licensing). It is written in C++, and is cross-platform. It is provided as part of the tools included in Qt 4.1, and earlier versions were provided separately. Note that earlier versions had a different API, and the examples will not work with that earlier version.

In addition to normal unit testing capabilities, QtTestLib also offers basic GUI testing, based on sending QEvents. This allows you to test GUI widgets, but is not generally suitable for testing full applications.

Each testcase is a standalone test application. Unlike CppUnit or JUnit, there is no Runner type class. Instead, each testcase is an executable which is simply run.

Tutorial 1: A simple test of a date class

In this tutorial, we will build a simple test for a class that represents a date, using QtTestLib as the test framework. To avoid too much detail on how the date class works, we'll just use the QDate. class that comes with Qt. In a normal unittest, you would more likely be testing code that you've written yourself.

The code below is the entire testcase.

Example 1. QDate test code

#include <QtTest>
#include <QtCore>

class testDate: public QObject
{
    Q_OBJECT
private slots:
    void testValidity();
    void testMonth();
};

void testDate::testValidity()
{
    // 11 March 1967
    QDate date( 1967, 3, 11 );
    QVERIFY( date.isValid() );
}

void testDate::testMonth()
{
    // 11 March 1967
    QDate date;
    date.setYMD( 1967, 3, 11 );
    QCOMPARE( date.month(), 3 );
    QCOMPARE( QDate::longMonthName(date.month()),
              QString("March") );
}


QTEST_MAIN(testDate)
#include "tutorial1.moc"

Stepping through the code, the first line imports the header files for the QtTest namespace. The second line imports the headers for the QtCore namespace (not strictly necessary, since QtTest also imports it, but it is robust and safe). Lines 3 to 9 give us the test class, testData. Note that testDate inherits from QObject and has the Q_OBJECT macro - QtTestLib requires specific Qt functionality that is present in QObject.

Lines 10 to 15 provide our first test, which checks that a date is valid. 3 the use of the QVERIFY macro, which checks that the condition is true. So if date.isValid() returns true, then the test will pass, otherwise the test will fail. QVERIFY is similar to ASSERT in other test suites.

Similarly, lines 16 to 24 provide another test, which checks a setter, and a couple of accessor routines. In this case, we are using QCOMPARE, which checks that the conditions are equal. So if date.month() returns 3, then that part of that test will pass, otherwise the test will fail.

Warning

Warning: As soon as a QVERIFY evaluates to false or a QCOMPARE does not have two equal values, the rest of that test will be skipped over. So in the example above, if the check at line 21 fails, then the check at lines 22 and 23 will not be run.

In a later tutorial we will see how to work around problems that this behaviour can cause.

Line 25 uses the QTEST_MAIN which creates an entry point routine for us, with appropriate calls to invoke the testDate unit test class.

Line 26 includes the Meta-Object compiler output, so we can make use of our QObject functionality.

The qmake project file that corresponds to that code is shown below. You would then use qmake to turn this into a Makefile and then compile it with make.

Example 2. QDate unit test project

CONFIG += qtestlib
TEMPLATE = app
TARGET +=
DEPENDPATH += .
INCLUDEPATH += .

# Input
SOURCES += tutorial1.cpp

This is a fairly normal project file, except for the addition of the CONFIG += qtestlib. This adds the right header and library setup to the Makefile.

This will produce an application that can then be run on the command line. The output looks like the following:

Example 3. QDate unit test output

$ ./tutorial1
********* Start testing of testDate *********
Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051003
PASS   : testDate::initTestCase()
PASS   : testDate::testValidity()
PASS   : testDate::testMonth()
PASS   : testDate::cleanupTestCase()
Totals: 4 passed, 0 failed, 0 skipped
********* Finished testing of testDate *********

Looking at the output above, you can see that the output includes the version of the test library and Qt itself, and then the status of each test that is run. In addition to the testValidity and testMonth tests that we defined, there is also a setup routine (initTestCase) and a teardown routine (cleanupTestCase) that can be used to do additional configuration if required.

Failing tests

If we had made an error in either the production code or the unit test code, then the results would show an error. An example is shown below:

Example 4. QDate unit test output showing failure

$ ./tutorial1
********* Start testing of testDate *********
Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051003
PASS   : testDate::initTestCase()
PASS   : testDate::testValidity()
FAIL!  : testDate::testMonth() Compared values are not the same
   Actual (date.month()): 4
   Expected (3): 3
    Loc: [tutorial1.cpp(25)]
PASS   : testDate::cleanupTestCase()
Totals: 3 passed, 1 failed, 0 skipped
********* Finished testing of testDate *********

Running selected tests

When the number of test functions increases, and some of the functions take a long time to run, it can be useful to only run a selected function. For example, if you only want to run the testMonth function, then you just specify that on the command line, as shown below:

Example 5. QDate unit test output - selected function

$ ./tutorial1 testValidity
********* Start testing of testDate *********
Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051003
PASS   : testDate::initTestCase()
PASS   : testDate::testValidity()
PASS   : testDate::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped
********* Finished testing of testDate *********

Note that the initTestCase and cleanupTestCase routines are always run, so that any necessary setup and cleanup will still be done.

You can get a list of the available functions by passing the -functions option, as shown below:

Example 6. QDate unit test output - listing functions

$ ./tutorial1 -functions
testValidity()
testMonth()

Verbose output options

You can get more verbose output by using the -v1, -v2 and -vs options. -v1 produces a message on entering each test function. I found this is useful when it looks like a test is hanging. This is shown below:

Example 7. QDate unit test output - verbose output

$ ./tutorial1 -v1
********* Start testing of testDate *********
Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051003
INFO   : testDate::initTestCase() entering
PASS   : testDate::initTestCase()
INFO   : testDate::testValidity() entering
PASS   : testDate::testValidity()
INFO   : testDate::testMonth() entering
PASS   : testDate::testMonth()
INFO   : testDate::cleanupTestCase() entering
PASS   : testDate::cleanupTestCase()
Totals: 4 passed, 0 failed, 0 skipped
********* Finished testing of testDate *********

The -v2 option shows each QVERIFY, QCOMPARE and QTEST, as well as the message on entering each test function. I found this useful for verifying that a particular step is being run. This is shown below:

'Example 8. QDate unit test output - more verbose output

$ ./tutorial1 -v2
********* Start testing of testDate *********
Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051003
INFO   : testDate::initTestCase() entering
PASS   : testDate::initTestCase()
INFO   : testDate::testValidity() entering
INFO   : testDate::testValidity() QVERIFY(date.isValid())
    Loc: [tutorial1.cpp(17)]
PASS   : testDate::testValidity()
INFO   : testDate::testMonth() entering
INFO   : testDate::testMonth() COMPARE()
    Loc: [tutorial1.cpp(25)]
INFO   : testDate::testMonth() COMPARE()
    Loc: [tutorial1.cpp(27)]
PASS   : testDate::testMonth()
INFO   : testDate::cleanupTestCase() entering
PASS   : testDate::cleanupTestCase()
Totals: 4 passed, 0 failed, 0 skipped
********* Finished testing of testDate *********

The -vs option shows each signal that is emitted. In our example, there are no signals, so -vs has no effect. Getting a list of signals is useful for debugging failing tests, especially GUI tests which we will see in the third tutorial.

Output to a file

If you want to output the results of your testing to a file, you can use the -o filename, where you replace filename with the name of the file you want to save output to.

Tutorial 2: Data driven testing of a date class

In the previous example, we looked at how we can test a date class. If we decided that we really needed to test a lot more dates, then we'd be cutting and pasting a lot of code. If we subsequently changed the name of a function, then it has to be changed in a lot of places. As an alternative to introducing these types of maintenance problems into our tests, QtTestLib offers support for data driven testing.

The easiest way to understand data driven testing is by an example, as shown below:

Example 9. QDate test code, data driven version

#include <QtTest>
#include <QtCore>


class testDate: public QObject
{
    Q_OBJECT
private slots:
    void testValidity();
    void testMonth_data();
    void testMonth();
};

void testDate::testValidity()
{
    // 12 March 1967
    QDate date( 1967, 3, 12 );
    QVERIFY( date.isValid() );
}

void testDate::testMonth_data()
{
    QTest::addColumn<int>("year");  // the year we are testing
    QTest::addColumn<int>("month"); // the month we are testing
    QTest::addColumn<int>("day");   // the day we are testing
    QTest::addColumn<QString>("monthName");   // the name of the month

    QTest::newRow("1967/3/11") << 1967 << 3 << 11 << QString("March");
    QTest::newRow("1966/1/10") << 1966 << 1 << 10 << QString("January");
    QTest::newRow("1999/9/19") << 1999 << 9 << 19 << QString("September");
    // more rows of dates can go in here...
}

void testDate::testMonth()
{
    QFETCH(int, year);
    QFETCH(int, month);
    QFETCH(int, day);
    QFETCH(QString, monthName);

    QDate date;
    date.setYMD( year, month, day);
    QCOMPARE( date.month(), month );
    QCOMPARE( QDate::longMonthName(date.month()), monthName );
}


QTEST_MAIN(testDate)
#include "tutorial2.moc"

As you can see, we've introduced a new method - testMonth_data, and moved the specific test date out of testMonth. We've had to add some more code (which will be explained soon), but the result is a separation of the data we are testing, and the code we are using to test it. The names of the functions are important - you must use the _data suffix for the data setup routine, and the first part of the data setup routine must match the name of the driver routine.

It is useful to visualise the data as being a table, where the columns are the various data values required for a single run through the driver, and the rows are different runs. In our example, there are four columns (three integers, one for each part of the date; and one QString ), added in lines 19 through 22. The addColumn template obviously requires the type of variable to be added, and also requires a variable name argument. We then add as many rows as required using the newRow function, as shown in lines 23 through 26. The string argument to newRow is a label, which is handy for determining what is going on with failing tests, but doesn't have any effect on the test itself.

To use the data, we simply use QFETCH to obtain the appropriate data from each row. The arguments to QFETCH are the type of the variable to fetch, and the name of the column (which is also the local name of the variable it gets fetched into). You can then use this data in a QCOMPARE or QVERIFY check. The code is run for each row, which you can see below:

Example 10. Results of data driven testing, showing QFETCH

$ ./tutorial2 -v2
********* Start testing of testDate *********
Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051020
INFO   : testDate::initTestCase() entering
PASS   : testDate::initTestCase()
INFO   : testDate::testValidity() entering
INFO   : testDate::testValidity() QVERIFY(date.isValid())
   Loc: [tutorial2.cpp(19)]
PASS   : testDate::testValidity()
INFO   : testDate::testMonth() entering
INFO   : testDate::testMonth(1967/3/11) COMPARE()
   Loc: [tutorial2.cpp(44)]
INFO   : testDate::testMonth(1967/3/11) COMPARE()
   Loc: [tutorial2.cpp(45)]
INFO   : testDate::testMonth(1966/1/10) COMPARE()
   Loc: [tutorial2.cpp(44)]
INFO   : testDate::testMonth(1966/1/10) COMPARE()
   Loc: [tutorial2.cpp(45)]
INFO   : testDate::testMonth(1999/9/19) COMPARE()
   Loc: [tutorial2.cpp(44)]
INFO   : testDate::testMonth(1999/9/19) COMPARE()
   Loc: [tutorial2.cpp(45)]
PASS   : testDate::testMonth()
INFO   : testDate::cleanupTestCase() entering
PASS   : testDate::cleanupTestCase()
Totals: 4 passed, 0 failed, 0 skipped
********* Finished testing of testDate *********

The QTEST macro

As an alternative to using QFETCH and QCOMPARE, you may be able to use the QTEST macro instead. QTEST takes two arguments, and if one is a string, it looks up that string as an argument in the current row. You can see how this can be used below, which is equivalent to the testMonth() code in the previous example.

Example 11. QDate test code, data driven version using QTEST

void testDate::testMonth()
{
    QFETCH(int, year);
    QFETCH(int, month);
    QFETCH(int, day);

    QDate date;
    date.setYMD( year, month, day);
    QCOMPARE( date.month(), month );
    QTEST( QDate::longMonthName(date.month()), "monthName" );
}

In the example above, note that monthname is enclosed in quotes, and we no longer have a QFETCH call for monthname.

The other QCOMPARE could also have been converted to use QTEST, however this would be less efficient, because we already needed to use QFETCH to get month for the setYMD in the line above.

Running selected tests with selected data

In the previous tutorial, we saw how to run a specific test by specifying the name of the test as a command line argument. In data driven testing, you can select which data you want the test run with, by adding a colon and the label for the data row. For example, if we just want to run the testMonth test for the first row, we would use ./tutorial2 -v2 testMonth:1967/3/11. The result of this is shown below.

Example 12. QDate unit test output - selected function and data

$ ./tutorial2 -v2 testMonth:1967/3/11
********* Start testing of testDate *********
Config: Using QTest library 4.1.0, Qt 4.1.0-snapshot-20051020
INFO   : testDate::initTestCase() entering
PASS   : testDate::initTestCase()
INFO   : testDate::testMonth() entering
INFO   : testDate::testMonth(1967/3/11) COMPARE()
   Loc: [tutorial2.cpp(44)]
INFO   : testDate::testMonth(1967/3/11) COMPARE()
   Loc: [tutorial2.cpp(45)]
PASS   : testDate::testMonth()
INFO   : testDate::cleanupTestCase() entering
PASS   : testDate::cleanupTestCase()
Totals: 3 passed, 0 failed, 0 skipped
********* Finished testing of testDate *********

Continued

This tutorial has not been fully migrated yet. Please continue porting from http://developer.kde.org/documentation/tutorials/writingunittests/tute3.html Template:KDE4