Infrastructure/Continuous Integration System: Difference between revisions

From KDE Community Wiki
(Add warning about outdated content: build.kde.org is shutdown)
(Remove all Jenkins and add first draft about Gitlab CI)
Line 1: Line 1:
{{Warning|The content on this page is mostly outdated since the Jenkins CI on https://build.kde.org/ has been replaces by GitLab CI. For the time being you can find some information about KDE's GitLab CI in the kde-devel mailinglist archive:
{{Warning|The content on this page is in the process of being updated since the Jenkins CI on https://build.kde.org/ has been replaced by GitLab CI. For the time being you can find some information about KDE's GitLab CI in the kde-devel mailinglist archive:


* Rollout of Gitlab CI: https://mail.kde.org/pipermail/kde-devel/2021-October/000735.html  
* Rollout of Gitlab CI: https://mail.kde.org/pipermail/kde-devel/2021-October/000735.html  
Line 5: Line 5:
* Debugging KDE CI runners: https://mail.kde.org/pipermail/kde-devel/2022-October/001401.html
* Debugging KDE CI runners: https://mail.kde.org/pipermail/kde-devel/2022-October/001401.html


Please help to update this page!
}}
}}


To ensure high-quality KDE software releases which build successfully using a variety of configurations and to track the status of automated tests to prevent regressions, we operate a [https://build.kde.org/ Continuous Integration system]. It initiates builds as soon as changes are pushed to KDE code repositories. This system is powered by [https://jenkins.io/ Jenkins] and consists of two servers to carry out builds and run tests.
Continuous Integration (CI) is a process that allows developers to ensure their code is properly tested, follows quality standards and compiles correctly, while Continuous Delivery (CD) is a process that allows to automate the generation of binaries or packages for end users.


== Adding your project ==
KDE ensures the quality of its code with the help of [https://docs.gitlab.com/ee/ci/ Gitlab CI/CD]. The Gitlab CI/CD system consists mostly of compiling software projects inside [https://docs.docker.com/get-started/overview/ Docker containers] or virtual machines (called ''CI images'') that have their environment prepared for builds and testing.


Any project which is hosted in a KDE code repository may request to be added to the CI system. Before doing so, please make sure that the dependencies for your project (and any branches you are requesting) have been specified correctly. This will help ensure that the first build is completed successfully.
== Runners and Jobs ==


Once the dependencies are in place, please file a [https://go.kde.org/u/systickets ticket], mentioning the names of the project repositories, and the branches which should be covered. If you have dependencies on non-KDE projects, please make sure to mention these as well, as they may not yet be installed on the build servers.
We call a ''job'' when software is run inside one of these containers or virtual machines, and in standard Gitlab CI/CD we describe jobs using a [https://docs.gitlab.com/ee/ci/yaml/ .gitlab-ci.yml] file. Jobs can run only under certain circumstances by using [https://docs.gitlab.com/ee/ci/jobs/job_control.html rules], such as when making a Merge Request, when a given amount of time has passed, when a certain file exists in the repository, or whenever new code is merged into the main branch.


== Dependencies ==
Additionally, KDE has some custom integrated tooling used to modify jobs for building KDE projects, and it resides in a <code>.kde-ci.yml</code> file, whose example configuration can be found in [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml sysadmin/ci-utilities/config-template.yml]. A custom [https://invent.kde.org/nicolasfella/dependency-generator dependency-generator] is run on these files to update dependency information stored in [https://invent.kde.org/sysadmin/repo-metadata/-/tree/master/dependencies sysadmin/repo-metadata/dependencies], which is then used for [https://community.kde.org/Get_Involved/development/Set_up_a_development_environment kdesrc-build], among other things.


The system handles dependencies in different ways depending on their location in the platform stack. If a dependency is system level, with no dependency on Qt or KDE, then it is usually installed system-wide on the build servers from distribution packages. To request an update or the installation of a dependency in this category, please file a [https://go.kde.org/u/systickets ticket].
A ''runner'' on the other hand is a process that runs jobs. It does not run those jobs in Gitlab itself; it requires additional software to be running in addition to Gitlab. By default, this software is [https://docs.gitlab.com/runner/ Gitlab Runner] which as mentioned before uses a <code>.gitlab-ci.yml</code> file written in [https://en.wikipedia.org/wiki/YAML YAML], but you can use [https://docs.gitlab.com/ee/integration/ Gitlab Integrations] to add [https://www.drone.io/ Drone CI] or [https://www.jenkins.io/ Jenkins] for example. These other pieces of software use different file formats and syntax (such as Jsonnet and Groovy) and will not be covered here.


If a dependency is using Qt however, it is necessary for the CI system to build this project itself. This is because Qt cannot be installed on a system-wide basis as we support builds against both Qt 4 and Qt 5. These dependencies are always available to the project being built as long as the version of Qt matches. To request an update or the installation of a dependency in this category, please file a [https://go.kde.org/u/systickets ticket]. Please note that many of these are updated weekly after the appropriate version of Qt built successfully.
System administrators and project maintainers are allowed to create runners, while users without these roles cannot. When runners are created from the system administration interface, they are called ''shared runners'', which all KDE projects can use. These are general use and should meet the needs of most KDE projects. When runners are created from a project's settings, they are called ''project runners'', and they should only be used when a project has special needs that cannot be met with shared runners.


The final category of dependencies are those hosted in KDE code repositories. These are only made available to a project if it has declared a need for them, which must be done in the <code>dependency-data</code> file. Dependencies listed in this file are fully recursive, so please do not define recursive dependencies as this will cause the build for the affected projects to fail. The <code>dependency-data</code> file can be found in the <code>repo-metadata</code> repository on <code>invent.kde.org</code>, and is synced to the build servers hourly. Please note that if the build servers have not previously built a KDE project you are attempting to depend on, then you need to request it to be added first.
System administrators, project maintainers and users without these roles are allowed to run jobs.


== Tests ==
If you are completely new to Gitlab CI/CD, then you may create your own projects on [https://invent.kde.org Invent] or create a fork of an existing project to create your own runners and your own jobs. It is recommended that you use the CI images mentioned below so that your jobs consume less resources from KDE infrastructure.


After the completion of a successful build, the CI system will automatically check to see if a project has any tests available for execution via CTest. If it does, then it will proceed to setup a test environment and execute the tests. To ensure that test results are available quickly and to free up the build servers for other builds, tests are limited to 120 seconds of execution time each. Should this limit be exceeded the test is deemed to have failed and stopped.
== CI images ==


The degree to which a full KDE workspace session is made available in the test environment varies with the KDE projects your project specifies as dependencies. All test environments will include an Xvfb server (to provide an X server for tests requiring a GUI) as well as a D-Bus session bus. Any project which specifies a dependency on kdelibs will also have kde-runtime made available in the test environment to ensure the proper operation of various components in kdelibs.
As mentioned before, jobs are run on top of CI images. You can find a list of CI images in [https://invent.kde.org/sysadmin/ci-images https://invent.kde.org/sysadmin/ci-images]. To use them in a <code>.gitlab-ci.yml</code> file, you can replace <code><nowiki>https://invent.kde.org</nowiki></code> with <code>invent-registry.kde.org</code> and add the container tag at the end if applicable (usually <code>:latest</code>), for example: <code>invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest</code>.


Test results will be made available both in the build log as well as on the website as part of the build results. Information available includes comparative graphs (both overall and for individual tests) as well as test logs. This information is kept for the last 200 builds of a project.
Some of the provided CI images are very large (even up to 6 GB of disk space), but because CI images are cached, their size does not pose a problem. Additionally, since they are already preconfigured for you, they will consume much less resources from KDE infrastructure, as they will most likely only need to compile your project and run specific tasks and nothing more.


== Notifications ==
== Including CI templates ==


The CI system is able to send notifications both via email and as messages on IRC. These can be sent if a project has tests which have failed, or only if a build has failed. However, we will only enable notifications for failing tests if a project has a history of the tests completing successfully, which is necessary to ensure the usefulness of the failure reports.
Instead of writing your own <code>.gitlab-ci.yml</code> files yourself, to make life easier for KDE developers, Gitlab CI templates are provided in [https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates sysadmin/ci-utilities/gitlab-templates].


To request this for a project, please file a [https://go.kde.org/u/systickets ticket] stating the desired email address(es) or IRC channels which notifications should be sent to. When requesting this for a mailing list or team IRC channel, please ensure you have the consent of the other participants first.
Your typical Gitlab CI file for a KDE project will likely just use the [https://docs.gitlab.com/ee/ci/yaml/includes.html include: command] to add these template files whenever you need to enable a new job, for example, for building with Qt6, for checking whether your project follows our REUSE guidelines, or for generating flatpak bundles built from the main branch.


== Workflows ==
To include it, you should use the raw link for the template: go to the template file on Invent, for example [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/linux-qt6.yml linux-qt6], then right-click on "Open Raw" on the right side of the screen, below the commit number and to the right of the Replace/Delete buttons, and click on "Copy Link".


=== Updating builds on switching the "stable" branch ===
Here is an example <code>.gitlab-ci.yml</code> file:


First update the build metadata & build setups, then do an initial build of the Dependency set and then of the product(s) themselves:
{{Input|1=<nowiki>
# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0


# Update the info in [https://invent.kde.org/sysadmin/repo-metadata]:
include:
#* Change the branch name for stable in the the file "dependencies/logical-module-structure"
# - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/reuse-lint.yml
#* Update dependencies listed in the file "dependencies/dependency-data-kf5-qt5" if needed (wildcard rules might keep you covered, explicit listing done only for non-basic needs)
# - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux.yml
# Trigger build.kde.org to update build setups to latest repo-metadata info
# - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd.yml
#* Start [https://build.kde.org/job/Administration/job/DSL%20Job%20Seed/ "DSL Job Seed"] on build.kde.org<br/>That job will recreate all build setups based on the changed branch info in repo-metadata<br/>If you do not have rights to start build jobs, ask on IRC channel #kde-sysadmin for someone to help or getting that right yourself
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows.yml
# Wait for the DSL Job Seed to have finished
# - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux-qt6.yml
# Trigger the '''always required''' initial build of the Dependency set and then of the product(s) themselves<br/>See below for respective guides for the different product types.
# - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd-qt6.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows-qt6.yml
# - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/flatpak.yml
</nowiki>}}


 
These are currently the most common builds enabled via the provided templates. You can simply copy the file and uncomment what job you need for your project.
''Note:'' The initial builds need to be always done, both for the first stable branch creation as well as for any later change of the stable branch. KDE CI only automatically triggers builds on commit pushes to a branch if an initial build has been manually started for the build setup with a new branch name, as done by the above.
 
 
Initial builds for an individually released product (listed in "Extragear"):
# Start the matching jobs [https://build.kde.org/job/Administration/ "Dependency Build Extragear stable-kf5-qt5 *"] on build.kde.org<br/>This ensures a refreshed build with any new dependencies
# Wait for the Dependency Builds to have finished
# Start the matching jobs [https://build.kde.org/job/Extragear/view/Everything%20-%20stable-kf5-qt5/ "Extragear <product> *"] on build.kde.org
 
 
Initial builds for a product bundle (KDE Frameworks, Plasma, KDE Applications):
# Start the matching jobs [https://build.kde.org/job/Administration/ "Global Rebuild *"] on build.kde.org<br/>Those global rebuild jobs will first run the matching "Dependency Build" for a refreshed build with any new dependencies. If that finished successfully, it will then run the builds of all product belonging to that build group.<br/>A few product builds may fail due to server connection issues, so once the mass rebuild is done manually check those builds that failed if it was due to server issues and, if so, restart a build job for them manually.

Revision as of 17:11, 13 October 2023

Warning

The content on this page is in the process of being updated since the Jenkins CI on https://build.kde.org/ has been replaced by GitLab CI. For the time being you can find some information about KDE's GitLab CI in the kde-devel mailinglist archive:


Continuous Integration (CI) is a process that allows developers to ensure their code is properly tested, follows quality standards and compiles correctly, while Continuous Delivery (CD) is a process that allows to automate the generation of binaries or packages for end users.

KDE ensures the quality of its code with the help of Gitlab CI/CD. The Gitlab CI/CD system consists mostly of compiling software projects inside Docker containers or virtual machines (called CI images) that have their environment prepared for builds and testing.

Runners and Jobs

We call a job when software is run inside one of these containers or virtual machines, and in standard Gitlab CI/CD we describe jobs using a .gitlab-ci.yml file. Jobs can run only under certain circumstances by using rules, such as when making a Merge Request, when a given amount of time has passed, when a certain file exists in the repository, or whenever new code is merged into the main branch.

Additionally, KDE has some custom integrated tooling used to modify jobs for building KDE projects, and it resides in a .kde-ci.yml file, whose example configuration can be found in sysadmin/ci-utilities/config-template.yml. A custom dependency-generator is run on these files to update dependency information stored in sysadmin/repo-metadata/dependencies, which is then used for kdesrc-build, among other things.

A runner on the other hand is a process that runs jobs. It does not run those jobs in Gitlab itself; it requires additional software to be running in addition to Gitlab. By default, this software is Gitlab Runner which as mentioned before uses a .gitlab-ci.yml file written in YAML, but you can use Gitlab Integrations to add Drone CI or Jenkins for example. These other pieces of software use different file formats and syntax (such as Jsonnet and Groovy) and will not be covered here.

System administrators and project maintainers are allowed to create runners, while users without these roles cannot. When runners are created from the system administration interface, they are called shared runners, which all KDE projects can use. These are general use and should meet the needs of most KDE projects. When runners are created from a project's settings, they are called project runners, and they should only be used when a project has special needs that cannot be met with shared runners.

System administrators, project maintainers and users without these roles are allowed to run jobs.

If you are completely new to Gitlab CI/CD, then you may create your own projects on Invent or create a fork of an existing project to create your own runners and your own jobs. It is recommended that you use the CI images mentioned below so that your jobs consume less resources from KDE infrastructure.

CI images

As mentioned before, jobs are run on top of CI images. You can find a list of CI images in https://invent.kde.org/sysadmin/ci-images. To use them in a .gitlab-ci.yml file, you can replace https://invent.kde.org with invent-registry.kde.org and add the container tag at the end if applicable (usually :latest), for example: invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest.

Some of the provided CI images are very large (even up to 6 GB of disk space), but because CI images are cached, their size does not pose a problem. Additionally, since they are already preconfigured for you, they will consume much less resources from KDE infrastructure, as they will most likely only need to compile your project and run specific tasks and nothing more.

Including CI templates

Instead of writing your own .gitlab-ci.yml files yourself, to make life easier for KDE developers, Gitlab CI templates are provided in sysadmin/ci-utilities/gitlab-templates.

Your typical Gitlab CI file for a KDE project will likely just use the include: command to add these template files whenever you need to enable a new job, for example, for building with Qt6, for checking whether your project follows our REUSE guidelines, or for generating flatpak bundles built from the main branch.

To include it, you should use the raw link for the template: go to the template file on Invent, for example linux-qt6, then right-click on "Open Raw" on the right side of the screen, below the commit number and to the right of the Replace/Delete buttons, and click on "Copy Link".

Here is an example .gitlab-ci.yml file: 

# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0

include:
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/reuse-lint.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux-qt6.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd-qt6.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows-qt6.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/flatpak.yml

These are currently the most common builds enabled via the provided templates. You can simply copy the file and uncomment what job you need for your project.

Retrieved from "https://community.kde.org/index.php?title=Infrastructure/Continuous_Integration_System&oldid=98617"