Infrastructure/Continuous Integration System
Warning
- Rollout of Gitlab CI: https://mail.kde.org/pipermail/kde-devel/2021-October/000735.html
- Gitlab CI Dashboards and retirement of build.kde.org: https://mail.kde.org/pipermail/kde-devel/2022-September/001270.html
- Debugging KDE CI runners: https://mail.kde.org/pipermail/kde-devel/2022-October/001401.html
Continuous Integration (CI) is a process that allows developers to ensure their code is properly tested, follows quality standards and compiles correctly, while Continuous Delivery (CD) is a process that allows to automate the generation of binaries or packages for end users.
KDE ensures the quality of its code with the help of Gitlab CI/CD. The Gitlab CI/CD system consists mostly of compiling software projects inside Docker containers or virtual machines (called CI images) that have their environment prepared for builds and testing.
Runners and Jobs
We call a job when software is run inside one of these containers or virtual machines, and in standard Gitlab CI/CD we describe jobs using a .gitlab-ci.yml file. Jobs can run only under certain circumstances by using rules, such as when making a Merge Request, when a given amount of time has passed, when a certain file exists in the repository, or whenever new code is merged into the main branch.
Additionally, KDE has some custom integrated tooling used to modify jobs for building KDE projects, and it resides in a .kde-ci.yml file, whose example configuration can be found in sysadmin/ci-utilities/config-template.yml. A custom dependency-generator is run on these files to update dependency information stored in sysadmin/repo-metadata/dependencies, which is then used for kdesrc-build, among other things.
A runner on the other hand is a process that runs jobs. It does not run those jobs in Gitlab itself; it requires additional software to be running in addition to Gitlab. By default, this software is Gitlab Runner which as mentioned before uses a .gitlab-ci.yml file written in YAML, but you can use Gitlab Integrations to add Drone CI or Jenkins for example. These other pieces of software use different file formats and syntax (such as Jsonnet and Groovy) and will not be covered here.
System administrators and project maintainers are allowed to create runners, while users without these roles cannot. When runners are created from the system administration interface, they are called shared runners, which all KDE projects can use. These are general use and should meet the needs of most KDE projects. When runners are created from a project's settings, they are called project runners, and they should only be used when a project has special needs that cannot be met with shared runners.
System administrators, project maintainers and users without these roles are allowed to run jobs.
If you are completely new to Gitlab CI/CD, then you may create your own projects on Invent or create a fork of an existing project to create your own runners and your own jobs. It is recommended that you use the CI images mentioned below so that your jobs consume less resources from KDE infrastructure.
CI images
As mentioned before, jobs are run on top of CI images. You can find a list of CI images in https://invent.kde.org/sysadmin/ci-images. To use them in a .gitlab-ci.yml file, you can replace https://invent.kde.org with invent-registry.kde.org and add the container tag at the end if applicable (usually :latest), for example: invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest.
Some of the provided CI images are very large (even up to 6 GB of disk space), but because CI images are cached, their size does not pose a problem. Additionally, since they are already preconfigured for you, they will consume much less resources from KDE infrastructure, as they will most likely only need to compile your project and run specific tasks and nothing more.
Including CI templates
Instead of writing your own .gitlab-ci.yml files yourself, to make life easier for KDE developers, Gitlab CI templates are provided in sysadmin/ci-utilities/gitlab-templates.
Your typical Gitlab CI file for a KDE project will likely just use the include: command to add these template files whenever you need to enable a new job, for example, for building with Qt6, for checking whether your project follows our REUSE guidelines, or for generating flatpak bundles built from the main branch.
To include it, you should use the raw link for the template: go to the template file on Invent, for example linux-qt6, then right-click on "Open Raw" on the right side of the screen, below the commit number and to the right of the Replace/Delete buttons, and click on "Copy Link".
Here is an example .gitlab-ci.yml file:
# SPDX-FileCopyrightText: None # SPDX-License-Identifier: CC0-1.0 include: # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/reuse-lint.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux-qt6.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd-qt6.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows-qt6.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/flatpak.yml
These are currently the most common builds enabled via the provided templates. You can simply copy the file and uncomment what job you need for your project.