Calligra/Testing: Difference between revisions
| (7 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
== | == Blackbox testing == | ||
This | This section is dedicated to black box testing at the moment. There are two sets of test files. The first one is used to test input filters by comparing the output of the ''calligraconverter'' tool against the expected output. These are feature based files prepared by a testing team sponsored by Nokia and are primary intended to filter developers. The corresponding test tool is a shell script applying XSL templates. The second set is comprised of real world documents and it is used in combination with the ''cstester'' and ''visualimagecompare'' tools as described below. The purpose is to test both the filter and the layout engine and to display differences in document layout in means of visual output. | ||
== Getting the test files == | === Getting the test files === | ||
Checkout test files from the KDE subversion repository: | Checkout test files from the KDE subversion repository: | ||
| Line 16: | Line 16: | ||
</nowiki>}} | </nowiki>}} | ||
== Running tests == | === Running tests === | ||
=== Filter tests === | === Filter tests === | ||
There are two important directories in calligratests at the moment, the ''import'' and the ''interoperability'' | There are two important directories in calligratests at the moment, the ''import'' and the ''interoperability'' directory, which have the following 1st level structure of interest (list of subdirectories containing tests): | ||
{{Output|1=<nowiki> | {{Output|1=<nowiki> | ||
import | import | ||
| Line 34: | Line 34: | ||
|-- wordprocessing</nowiki>}} | |-- wordprocessing</nowiki>}} | ||
Each of the listed directories contains a Makefile. To start tests, make sure the ''import/share/bin'' directory is in your PATH environment variable and run make. | Each of the listed directories contains a Makefile. To start tests, make sure the ''import/share/bin'' directory is in your $PATH environment variable and run make. | ||
=== cstester === | === cstester === | ||
It's more convenient to use the cstrunner tool. Please check the following file present in your working copy of calligra for further instructions and details: calligra/devtools/cstester/README.txt | It's more convenient to use the cstrunner tool. Please check the following file present in your working copy of calligra for further instructions and details: ''calligra/devtools/cstester/README.txt'' | ||
==== cstrunner ==== | ==== cstrunner ==== | ||
A tool to run cstester in multiple processes. It creates a report of files terminated by a signal. This can happen as a result of a bug in calligra or when the allowed amount of time or CPU usage has been exceeded. The md5.txt file providing the ms5sums of generated thumbnails is created in each output directory. The following scripts are used by cstrunner. Make sure calligra/devtools/cstester/scripts is included in your $PATH environment variable. | A tool to run cstester in multiple processes. It creates a report of files terminated by a signal. This can happen as a result of a bug in calligra or when the allowed amount of time or CPU usage has been exceeded. The md5.txt file providing the ms5sums of generated thumbnails is created in each output directory. The following scripts are used by cstrunner. Make sure ''calligra/devtools/cstester/scripts'' is included in your $PATH environment variable. | ||
''cstwrapper.sh'' - Limits the resources cstester is allowed to use and makes sure the correct exit code is returned to cstrunner. | ''cstwrapper.sh'' - Limits the resources cstester is allowed to use and makes sure the correct exit code is returned to cstrunner. | ||
| Line 48: | Line 48: | ||
cstrunner expects the following parameters: | cstrunner expects the following parameters: | ||
{{Input|1=<nowiki> | {{Input|1=<nowiki> | ||
$ cstrunner docDir resultDir concurrentProcesses | $ cstrunner <docDir> <resultDir> <concurrentProcesses> | ||
</nowiki>}} | </nowiki>}} | ||
To run over all documents in the current directory and put the thumbnails in to ../result/sha1 with 4 concurrent processes | To run over all documents in the current directory and put the thumbnails in to ../result/sha1 with 4 concurrent processes: | ||
{{Input|1=<nowiki> | {{Input|1=<nowiki> | ||
$ cstrunner . ../result/sha1 4 | $ cstrunner . ../result/sha1 4 | ||
| Line 72: | Line 72: | ||
</nowiki>}} | </nowiki>}} | ||
=== visualimagecompare === | |||
List of supported commands : | List of supported commands : | ||
| Line 85: | Line 85: | ||
Space Switch between tabs | Space Switch between tabs | ||
</nowiki>}} | </nowiki>}} | ||
== Static Code Analysis == | |||
Our idea is: use more than one tool to maximize scope of tests. | |||
Following tools are available for [http://en.wikipedia.org/wiki/Static_program_analysis Static C/C++ code analysis]: | |||
*Coverity - Boudwijn registered Calligra as open source project and tested once. TODO: Details | |||
*PVS-Studio - free version for certain period of time kindly donated to Calligra, maintainer: Jaroslaw Staniek | |||
*[http://www.parasoft.com/static-analysis Prevent] by Parasoft - status unknown | |||
*[http://www.klocwork.com/products/insight/source-code-analysis/ Klocwork] - status unknown, volunteers needed | |||
*[http://clang-analyzer.llvm.org/ Clang Static Analyzer] - free/open source, status unknown, volunteers needed | |||
*[http://cppcheck.sourceforge.net Cppcheck] - free/open source, status unknown, volunteers needed | |||
If you want to help, contact the mailing list [email protected]. | |||
Latest revision as of 19:59, 5 June 2014
Blackbox testing
This section is dedicated to black box testing at the moment. There are two sets of test files. The first one is used to test input filters by comparing the output of the calligraconverter tool against the expected output. These are feature based files prepared by a testing team sponsored by Nokia and are primary intended to filter developers. The corresponding test tool is a shell script applying XSL templates. The second set is comprised of real world documents and it is used in combination with the cstester and visualimagecompare tools as described below. The purpose is to test both the filter and the layout engine and to display differences in document layout in means of visual output.
Getting the test files
Checkout test files from the KDE subversion repository:
$ svn co svn+ssh://[email protected]/home/kde/trunk/tests/calligratests
Download test files for cstester:
$ wget http://www.valdyas.org/~boud/documents.tar $ wget http://www.valdyas.org/~boud/documents_pdf.tar
Running tests
Filter tests
There are two important directories in calligratests at the moment, the import and the interoperability directory, which have the following 1st level structure of interest (list of subdirectories containing tests):
import |-- excel |-- msword |-- powerpoint
interoperability/ |-- presentations |-- spreadsheets |-- wordprocessing
Each of the listed directories contains a Makefile. To start tests, make sure the import/share/bin directory is in your $PATH environment variable and run make.
cstester
It's more convenient to use the cstrunner tool. Please check the following file present in your working copy of calligra for further instructions and details: calligra/devtools/cstester/README.txt
cstrunner
A tool to run cstester in multiple processes. It creates a report of files terminated by a signal. This can happen as a result of a bug in calligra or when the allowed amount of time or CPU usage has been exceeded. The md5.txt file providing the ms5sums of generated thumbnails is created in each output directory. The following scripts are used by cstrunner. Make sure calligra/devtools/cstester/scripts is included in your $PATH environment variable.
cstwrapper.sh - Limits the resources cstester is allowed to use and makes sure the correct exit code is returned to cstrunner. cstmd5gen.sh - Generates md5sums for a given thumbnail directory and stores them into the md5.txt file.
cstrunner expects the following parameters:
$ cstrunner <docDir> <resultDir> <concurrentProcesses>
To run over all documents in the current directory and put the thumbnails in to ../result/sha1 with 4 concurrent processes:
$ cstrunner . ../result/sha1 4
Apply your changes and start cstrunner again:
$ cstrunner . ../result/sha2 4
To create the input file for visualimagecompare use the cstmd5diff.sh script having the following parameters:
$ cstmd5diff.sh <documents dir> <previous result dir> <current result dir>
To display changes between outputs from the previous two runs of cstrunner:
$ cstmd5diff.sh . ../results/sha1 ../results/sha2 > md5-sha1-sha2.log $ visualimagecompare ../results/sha1 ../results/sha2 md5-sha1-sha2.log
visualimagecompare
List of supported commands :
PageUp, n Move to the next page of the current document PageDown, b Move to the previous page of the current document Up Move to the previous document Down Move to the next document Left Switch to the left tab Right Switch to the right tab Space Switch between tabs
Static Code Analysis
Our idea is: use more than one tool to maximize scope of tests.
Following tools are available for Static C/C++ code analysis:
- Coverity - Boudwijn registered Calligra as open source project and tested once. TODO: Details
- PVS-Studio - free version for certain period of time kindly donated to Calligra, maintainer: Jaroslaw Staniek
- Prevent by Parasoft - status unknown
- Klocwork - status unknown, volunteers needed
- Clang Static Analyzer - free/open source, status unknown, volunteers needed
- Cppcheck - free/open source, status unknown, volunteers needed
If you want to help, contact the mailing list [email protected].