Infrastructure/Continuous Integration System
Warning
- Rollout of Gitlab CI: https://mail.kde.org/pipermail/kde-devel/2021-October/000735.html
- Gitlab CI Dashboards and retirement of build.kde.org: https://mail.kde.org/pipermail/kde-devel/2022-September/001270.html
- Debugging KDE CI runners: https://mail.kde.org/pipermail/kde-devel/2022-October/001401.html
To ensure high-quality KDE software releases which build successfully using a variety of configurations and to track the status of automated tests to prevent regressions, we operate a Continuous Integration system. It initiates builds as soon as changes are pushed to KDE code repositories. This system is powered by Jenkins and consists of two servers to carry out builds and run tests.
Adding your project
Any project which is hosted in a KDE code repository may request to be added to the CI system. Before doing so, please make sure that the dependencies for your project (and any branches you are requesting) have been specified correctly. This will help ensure that the first build is completed successfully.
Once the dependencies are in place, please file a ticket, mentioning the names of the project repositories, and the branches which should be covered. If you have dependencies on non-KDE projects, please make sure to mention these as well, as they may not yet be installed on the build servers.
Dependencies
The system handles dependencies in different ways depending on their location in the platform stack. If a dependency is system level, with no dependency on Qt or KDE, then it is usually installed system-wide on the build servers from distribution packages. To request an update or the installation of a dependency in this category, please file a ticket.
If a dependency is using Qt however, it is necessary for the CI system to build this project itself. This is because Qt cannot be installed on a system-wide basis as we support builds against both Qt 4 and Qt 5. These dependencies are always available to the project being built as long as the version of Qt matches. To request an update or the installation of a dependency in this category, please file a ticket. Please note that many of these are updated weekly after the appropriate version of Qt built successfully.
The final category of dependencies are those hosted in KDE code repositories. These are only made available to a project if it has declared a need for them, which must be done in the dependency-data file. Dependencies listed in this file are fully recursive, so please do not define recursive dependencies as this will cause the build for the affected projects to fail. The dependency-data file can be found in the repo-metadata repository on invent.kde.org, and is synced to the build servers hourly. Please note that if the build servers have not previously built a KDE project you are attempting to depend on, then you need to request it to be added first.
Tests
After the completion of a successful build, the CI system will automatically check to see if a project has any tests available for execution via CTest. If it does, then it will proceed to setup a test environment and execute the tests. To ensure that test results are available quickly and to free up the build servers for other builds, tests are limited to 120 seconds of execution time each. Should this limit be exceeded the test is deemed to have failed and stopped.
The degree to which a full KDE workspace session is made available in the test environment varies with the KDE projects your project specifies as dependencies. All test environments will include an Xvfb server (to provide an X server for tests requiring a GUI) as well as a D-Bus session bus. Any project which specifies a dependency on kdelibs will also have kde-runtime made available in the test environment to ensure the proper operation of various components in kdelibs.
Test results will be made available both in the build log as well as on the website as part of the build results. Information available includes comparative graphs (both overall and for individual tests) as well as test logs. This information is kept for the last 200 builds of a project.
Notifications
The CI system is able to send notifications both via email and as messages on IRC. These can be sent if a project has tests which have failed, or only if a build has failed. However, we will only enable notifications for failing tests if a project has a history of the tests completing successfully, which is necessary to ensure the usefulness of the failure reports.
To request this for a project, please file a ticket stating the desired email address(es) or IRC channels which notifications should be sent to. When requesting this for a mailing list or team IRC channel, please ensure you have the consent of the other participants first.
Workflows
Updating builds on switching the "stable" branch
First update the build metadata & build setups, then do an initial build of the Dependency set and then of the product(s) themselves:
- Update the info in [1]:
- Change the branch name for stable in the the file "dependencies/logical-module-structure"
- Update dependencies listed in the file "dependencies/dependency-data-kf5-qt5" if needed (wildcard rules might keep you covered, explicit listing done only for non-basic needs)
- Trigger build.kde.org to update build setups to latest repo-metadata info
- Start "DSL Job Seed" on build.kde.org
That job will recreate all build setups based on the changed branch info in repo-metadata
If you do not have rights to start build jobs, ask on IRC channel #kde-sysadmin for someone to help or getting that right yourself
- Start "DSL Job Seed" on build.kde.org
- Wait for the DSL Job Seed to have finished
- Trigger the always required initial build of the Dependency set and then of the product(s) themselves
See below for respective guides for the different product types.
Note: The initial builds need to be always done, both for the first stable branch creation as well as for any later change of the stable branch. KDE CI only automatically triggers builds on commit pushes to a branch if an initial build has been manually started for the build setup with a new branch name, as done by the above.
Initial builds for an individually released product (listed in "Extragear"):
- Start the matching jobs "Dependency Build Extragear stable-kf5-qt5 *" on build.kde.org
This ensures a refreshed build with any new dependencies - Wait for the Dependency Builds to have finished
- Start the matching jobs "Extragear <product> *" on build.kde.org
Initial builds for a product bundle (KDE Frameworks, Plasma, KDE Applications):
- Start the matching jobs "Global Rebuild *" on build.kde.org
Those global rebuild jobs will first run the matching "Dependency Build" for a refreshed build with any new dependencies. If that finished successfully, it will then run the builds of all product belonging to that build group.
A few product builds may fail due to server connection issues, so once the mass rebuild is done manually check those builds that failed if it was due to server issues and, if so, restart a build job for them manually.