Get Involved/Quality/Tutorial: Difference between revisions

From KDE Community Wiki

Latest revision as of 01:43, 16 February 2015

This tutorial was held during AdaLovelaceDay 2012 on October 16 in #kde-testing on irc.freenode.net. For more information about Ada Lovelace and the KDE tutorials please visit http://community.kde.org/AdaLovelaceDay/2012

The tutorial consists of the following chapters:

1. What is software testing?
2. What do we test?
3. How do we test?
4. Can't this be automated?
5. Where do I start?
6. Tools and resources for testing
7. Useful links

Disclaimer: this course is based on personal experience and to a large extend on information gathered online. The links to these sources will be given in section 6. of the course.

What is software testing?

Testing software is something every developer does on a daily basis, be it only to check if the code does what it is supposed to do and of course to make sure that it actually compiles. The main reasons to test software are the following (quote from http://en.wikipedia.org/wiki/Software_testing):

'Software testing can be stated as the process of validating and verifying that a computer program/application/product:

1. meets the requirements that guided its design and development;
2. works as expected;
3. can be implemented with the same characteristics.
4. satisfies the needs of stakeholders.'

In other words software testing is a Quality verification process and part of an overall Software Quality Assurance.

What do we test?

There are mainly 3 areas or levels where we test:

a) the finished software
b) the beta release and release candidates
c) the development version (aka alpha testing)

The finished software

With finished we mean the released version that is shipped to the distributions. This version is easily available for everybody, so it is also often the entry point where people start testing.

We have all done that: we learn about a software that sounds interesting, download and install it and check if the functionality is useful for our needs. This way of testing is not very organized and also often leads to problems when we encounter a bug: how did this happen and how can we reproduce this problem?

This is the typical situation an end user encounters and very often, especially if the encountered bug is not blocking the work, it is often shrugged upon and not reported.

Of course if we actually decide to test it is recommended to do so in clear detailed steps that are traceable so if a bug is found it can be reproduced.

In an ideal situation the end user should not be confronted with too many bugs as it is annoying the user and doesn't shed a good light on the project. So the testing and search for bugs needs to be done before the software is released, and that is where the Beta Testing comes in:

Testing the beta releases and release candidates

Pretty much every project does testing releases that are in a so called "beta stage": a software preview is released to a limited audience, the beta testers, so it can be tested and the feedback gathered is used to improve the final product.

In KDE we have a long experience with beta releases, which are often done in several levels: Beta 1, beta 2 and RC are the usual 3 planed steps before a final release. Sometimes a second RC called RC 2 is release, especially when last minute bug reports come in that trigger the need for an additional testing release.

It also sometimes happen that these testing releases lead to a delay in the final release, something that is quite common.

In Free Software we do have quite strict release schedules compared to what is usually the case in proprietary software development. At least I do not recall a single Windows version that was not delayed considerably compared to the initially announced release schedule. If we delay Free Software sometimes by a week or two, there have been proprietary software releases delayed by a complete year.

While the developers of course do test their development branches it is impossible to test everything, simply because it is almost impossible to cover all use cases of a software project, especially if the project is rather complex.

Sometimes they simply lack all the hardware the product will run on so it is important to have testers that can cover these particular use cases.

Of course a project that is already in it's beta stage is well advanced and usable, but every developer dreams of the situation where the software is tested way before it is at that point: testing the development branch is quite important!

The development branch, alpha testing

This is the stage before the software is released, and in company driven development this part is done internally only. Unlike a company we rarely have the resources to do much alpha testing, so it needs to be done by the developers to a large extend.

For a non-developer alpha testing can be quite challenging when the project is young and doesn't have many functionalities yet and a lot of people are reluctant to install software in such an early stage on a computer that is often also used for work.

But it is of course possible to install software to run in a limited environment that diminishes the impact it has on the system. This can be either done as a local installation that is only accessible by one user, or in a virtualized environment that doesn't impact on the regular installation.

Some distributions also propose so called "Nightly Builds"that are created in regular intervals (usually daily) that can be installed alongside the working environment. The user simply logs in to either his usual desktop or launches the "Nightly Build". Since this installation is totally separate from his work environment it can't break his regular installation.

Development testing is very important as it allows to detect problems early in the development cycle and helps to improve software during its development.

Distributions providing Nightly Builds:

Project Neon: https://launchpad.net/project-neon
Gentoo with the layman overlay manager: http://www.gentoo.org/proj/en/overlays/userguide.xml

How do we test?

Ideally every project has a list of requirements of their software available for the testers. If this doesn't exist (because nobody had time to write it yet or because the development team is very small) and also because requirements evolve, the tester should document their procedure to allow others to step in easily.

A small project with a small set of functionalities can do this quite easily, in other cases this is scattered in very different places and should be gathered on the project wiki.

A very good read on what to test can be found here: http://www.thebraidytester.com/downloads/YouAreNotDoneYet.pdf

For tests to be useful these should be documented and reported on the product wiki as well. Most projects use http://community.kde.org for that.

http://websvn.kde.org/trunk/KDE/kdesdk/umbrello/test/ATP.txt?revision=1244714&view=markup has an Application Testing Procedure form that can be used and modified if needed.

Functional testing

This follows the requirements: there is a precise list of what the software should be capable of and it makes the testing quite easy as there are clear steps on how to test.

A subset of function tests are regression tests to check if a functionality that was working previously has stopped working after a code change. A bug is then submitted with the keyword "regression".

These tests should be done and reported directly to the project, as well as documented in the respective wiki.

A sub-group of the functional testing is the bugs testing. This is usually done when a new version of a product is released or about to be released, either shortly before beta or after the release.

And of course the non-confirmed bugs for the current version need to be tested as well, this should ideally happen as soon as the bug is reported.

Non-Functional testing

Testing the performance of a product is also required:

How fast does it start?
How long do certain steps take?
Does it scale?
Are there security issues?
Is there risk of loss of data?
When and why does it crash?

The non-functional testing is more likely to decide on whether a product can be released or not. a lack of functions is secondary to that decision. It is typically done during development and to some extend in the beta release phase.

Non-functional bugs that are reported and block the correct use of the product typically will have the keyword "release_blocker". Non-functional testing is also done in direct collaboration with the project.

Translation and Localization

Since KDE is translated in many different languages it is important to also test the quality of the translation.

Are all translatable strings translated or are there parts still in English?
Are all translatable strings really translatable or are there hard-coded strings that need to be changed?
Is the translation accurate?
Is the character encoding correct?
Are the dialog boxes, tooltips and notifications correct and not misleading?

The KDE Localization Team (http://techbase.kde.org/Localization) is the one taking care of the translations and is also the primary contact for translation test results, with the exception of non-translatable strings that need to be addressed to the respective product.

Please see the web page for links to the individual language teams: http://l10n.kde.org/

Usability

How easily can the project be used?
Are all the elements where they should be?
How difficult is it to find an element?
Are the dialog boxes, tooltips and notifications correct and informative?
How difficult and complex is it to configure the product?

KDE software development usually follows the Human Interface Guidelines HIG: http://techbase.kde.org/Projects/Usability/HIG

The KDE Usability Team (http://techbase.kde.org/Projects/Usability) is always keen on getting new contributors, please talk to them on https://mail.kde.org/mailman/listinfo/kde-usability

Documentation

Is there a documentation?
Is it accurate?
Is it complete?
Is it up-to-date and reflects the current state of the product?

Typically documentation is delivered with the product, but since distributions tend to be very space aware it might not be installed by default.

KDE Documentation can be found here: http://docs.kde.org and many products also have a Manual on http://userbase.kde.org.

If you don't find documentation in either of these places please talk to the product team, they might have it in their development branch.

Can't testing be automated?

Of course, but not everything. What we currently use for KDE:

CI (Continuous Integration Testing)

A build server (Jenkins) builds the various KDE projects on a regular basis. If a commit causes a build problem an alert is sent out automatically.The Jenkins build server is building either in defined intervals on when a commit happens.

Unit Tests

Jenkins can also run a series of test that are known as Unit Tests that check the functionality of a certain portion of the code.

Since KDE uses an object-oriented development system this is a good method to test the different elements of the code and ensure they work independently of each other.

Of course these tests need to be written by the developers, and they often also write several tests for the same piece of code to catch different situations where a code can go wrong.

Code (syntax) tests

A static code analyzing tool is provided by the EnglishBreakfastNetwork: http://englishbreakfastnetwork.org/
Another static code analyzer is cppcheck which can be integrated with Jenkins.
More information can also be found here: http://techbase.kde.org/Development/Tutorials/Code_Checking

Debugging

KDE has an extensive wiki for debugging: http://techbase.kde.org/Development/Tutorials/Debugging

This is particularly important for alpha testing where debugging is used systematically, but also for bug reporting with beta releases and final releases.

Where do I start?

That is entirely up to you, but here are a few hints:

Beta testing

I often hear from volunteers that they want to do beta testing. Of course this is nice but the beta testing is done in a very limited time frame when the beta release is available so you need to be available at that moment as well.

While beta testing on Windows is done over a period of several months, in KDE you only have a few weeks so the beta testing is not available at all times. Please see the KDE SC release schedule for more information: http://techbase.kde.org/Schedules

Of course besides the KDE SC beta release there are several other KDE projects that release with a different schedule, but that is something you need to check with these products individually.

What is available all year long is the final product on which you can do functional testing and bug testing as well as confirm existing bugs. It is of course important that you use the most recent release.

Since we usually do not back-port fixes to previous versions it doesn't make much sense to test with older versions, except in the very special case of regression testing where you search the exact commit when a bug was introduced. But that is rarely done on final product versions, it usually is part of the development branch testing.

You have a preferred project

If you have a preferred project you use regularly and you know well it makes sense to start testing on this product.

You just have to decide on what type of test you want to do, whether you want to work with a finished product, or will you test the ongoing development branch and install the git version: If you mostly test on one or a very few subset of products you should consider a local build.

Several projects do have instructions on how to do a local build, like for example Calligra and Amarok:

But in general these same instructions apply for all products with slight variations.

Gentoo users can do that quite easily with this command:

layman -a kde && emerge -avt --autounmask=y --autounmask-write=y =<product>-9999

For more information you should check this page: http://www.gentoo.org/proj/en/overlays/userguide.xml

Be aware that the same user should not use a local build and the distribution package to avoid conflicts, so you should consider creating a test user for your local build.

You are an occasional tester

In that case you best start with bug testing, as these are always available and it is something easy to start with. Filter on a product and search for unconfirmed bugs. I will give more hints on bug database filtering a little later on.

You want to use the bleeding edge and work closely with the developers

In that case you best us a nightly build distribution that provides you with the current development branch called 'trunk'.

Project Neon is a good solution for that: http://techbase.kde.org/Getting_Started/Using_Project_Neon_to_contribute_to_KDE It is also available as snapshots that can be run with kvm: the tarballs can be downloaded here: http://kubuntu.s3.amazonaws.com/list.html Just install kvm, download the tarball, extract it and run the script that is inside.

If you use Gentoo you can also build from the development branch with layman by specifying emerge =kde-base/kde-meta-9999.

In both cases please make sure to enable debugging. For project Neon that would mean installing 'project-neon-foo-dbg', for Gentoo you would ideally enable tests (by having ‘test’ in FEATURES), install debugging symbols (by having ‘splitdebug’ in FEATURES and ‘-ggdb’ in CXXFLAGS) and enable debugging assertions (by having ‘debug’ in USE).

Tools and resources for testing

The advanced search mask of bugzilla: https://bugs.kde.org/query.cgi

When reporting a bug it helps searching if the bug is already reported You can actually save a default query in your preferences so you don't have to change the basic settings every time. So if you are testing on a particular project you would make or use a saved search for this project.

Human resources:

please don't hesitate to ask on irc.freenode.net in #kde-quality if you have any questions arising during your testing.

Wikis:

The community wiki at http://community.kde.org is the ideal place to document testing. Please make sure to add your tests as sub-pages to the respective product and don't forget to link to it. By adding the [[Category:Testing]] tag to the page it will be easily searchable in the wiki.

Some projects use http://techbase.kde.org for this, so please coordinate with the product on where to add the information.

Forums:

The KDE forum is also a very good place to coordinate testing: http://forum.kde.org. We are currently discussing the best place to use for general testing, but of course the product sub-forums are also an ideal place to start with.

Mailig lists:

In any case: if you decide to be a tester, please make yourself known and talk to us. We have a mailing list for that at https://mail.kde.org/mailman/listinfo/kde-testing you should subscribe to.

Useful links

Definitions: