Infrastructure/Continuous Integration System: Difference between revisions

From KDE Community Wiki
m (→‎The .kde-ci.yml file: Use bold for better list item visibility)
m (Replace redirect link)
 
(21 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{Warning|The content on this page is in the process of being updated since the Jenkins CI on https://build.kde.org/ has been replaced by GitLab CI. For the time being you can find some information about KDE's GitLab CI in the kde-devel mailinglist archive:
* Rollout of Gitlab CI: https://mail.kde.org/pipermail/kde-devel/2021-October/000735.html
* Gitlab CI Dashboards and retirement of build.kde.org: https://mail.kde.org/pipermail/kde-devel/2022-September/001270.html
* Debugging KDE CI runners: https://mail.kde.org/pipermail/kde-devel/2022-October/001401.html
}}
Continuous Integration (CI) is a process that allows developers to ensure their code is properly tested, follows quality standards and compiles correctly, while Continuous Delivery (CD) is a process that allows to automate the generation of binaries or packages for end users.
Continuous Integration (CI) is a process that allows developers to ensure their code is properly tested, follows quality standards and compiles correctly, while Continuous Delivery (CD) is a process that allows to automate the generation of binaries or packages for end users.


Line 13: Line 5:
== Runners and Jobs ==
== Runners and Jobs ==


We call a ''job'' when software is run inside one of these containers or virtual machines, and in standard Gitlab CI/CD we describe jobs using a [https://docs.gitlab.com/ee/ci/yaml/ .gitlab-ci.yml] file. Jobs can be configured to run only under certain circumstances by using [https://docs.gitlab.com/ee/ci/jobs/job_control.html rules], so that they can run when making a Merge Request, once a given amount of time has passed, if a certain file exists in the repository, or whenever new code is merged into the main branch.
We call a ''job'' when software is run inside of a CI image, and in standard Gitlab CI/CD we describe jobs using a [https://docs.gitlab.com/ee/ci/yaml/ .gitlab-ci.yml] file. Jobs can be configured to run only under certain circumstances by using [https://docs.gitlab.com/ee/ci/jobs/job_control.html rules], so that they can run when making a Merge Request, once a given amount of time has passed, if a certain file exists in the repository, or whenever new code is merged into the main branch.


Additionally, KDE has some additional tooling used to describe the dependency information required by a project, and includes certain options to modify jobs for building KDE projects. These modifications are described in a <code>.kde-ci.yml</code> file, and an example configuration showing all available options can be found in [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml sysadmin/ci-utilities/config-template.yml]. A custom [https://invent.kde.org/nicolasfella/dependency-generator dependency-generator] is run on these files to update dependency information stored in [https://invent.kde.org/sysadmin/repo-metadata/-/tree/master/dependencies sysadmin/repo-metadata/dependencies], which is then used for [https://community.kde.org/Get_Involved/development/Set_up_a_development_environment kdesrc-build] to determine the order in which to build projects, among other things.
Additionally, KDE has some additional tooling used to describe the dependency information required by a project, and includes certain options to modify jobs for building KDE projects. These modifications are described in a <code>.kde-ci.yml</code> file, and an example configuration showing all available options can be found in [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml sysadmin/ci-utilities/config-template.yml]. A custom [https://invent.kde.org/nicolasfella/dependency-generator dependency-generator] is run on these files to update dependency information stored in [https://invent.kde.org/sysadmin/repo-metadata/-/tree/master/dependencies sysadmin/repo-metadata/dependencies], which is then used for [https://develop.kde.org/docs/getting-started/building/kdesrc-build-setup/ kdesrc-build] to determine the order in which to build projects, among other things.


A ''runner'' on the other hand is a process that runs jobs. It does not run those jobs in Gitlab itself; it requires additional software to be running in addition to Gitlab. By default, this software is [https://docs.gitlab.com/runner/ Gitlab Runner] which as mentioned before uses a <code>.gitlab-ci.yml</code> file written in [https://en.wikipedia.org/wiki/YAML YAML]. While it is possible to [https://docs.gitlab.com/ee/integration/ integrate] other runner software like [https://www.drone.io/ Drone CI] or [https://www.jenkins.io/ Jenkins], these are not used or supported under KDE infrastructure.
A ''runner'' on the other hand is a process that runs jobs. It does not run those jobs in Gitlab itself; it requires additional software to be running in addition to Gitlab. By default, this software is [https://docs.gitlab.com/runner/ Gitlab Runner] which as mentioned before uses a <code>.gitlab-ci.yml</code> file written in [https://en.wikipedia.org/wiki/YAML YAML]. While it is possible to [https://docs.gitlab.com/ee/integration/ integrate] other runner software like [https://www.drone.io/ Drone CI] or [https://www.jenkins.io/ Jenkins], these are not used or supported under KDE infrastructure.
Line 24: Line 16:


If you are completely new to Gitlab CI/CD, then you may create your own projects on [https://invent.kde.org Invent] or create a fork of an existing project to make your own runners and jobs. It is recommended that you use the CI images mentioned below so that your jobs consume less resources from KDE infrastructure.
If you are completely new to Gitlab CI/CD, then you may create your own projects on [https://invent.kde.org Invent] or create a fork of an existing project to make your own runners and jobs. It is recommended that you use the CI images mentioned below so that your jobs consume less resources from KDE infrastructure.
[[File:Project Maintainer Settings.png|500px|thumb|center|The CI/CD project settings can be found under the Settings menu on the sidebar when you are a project maintainer or creator.]]


== CI images ==
== CI images ==


As mentioned before, jobs are run on top of CI images. You can find a list of CI images in [https://invent.kde.org/sysadmin/ci-images https://invent.kde.org/sysadmin/ci-images]. To use them in a <code>.gitlab-ci.yml</code> file, you can replace <code><nowiki>https://invent.kde.org</nowiki></code> with <code>invent-registry.kde.org</code> and add the container tag at the end if applicable (usually <code>:latest</code>), for example: <code>invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest</code>.
As mentioned before, jobs are run on top of CI images. You can find a list of CI images in [https://invent.kde.org/sysadmin/ci-images https://invent.kde.org/sysadmin/ci-images]. Any additional shared dependencies for KDE projects must be added here.
 
Developers are expected not to make their own custom CI files using these images. Instead, use the Gitlab Templates mentioned in the next section.
 
To fetch the container images from the KDE Container Registry, you can simply copy the URL for the image, replace <code><nowiki>https://invent.kde.org</nowiki></code> with <code>invent-registry.kde.org</code> and add the container tag at the end if applicable (usually <code>:latest</code>), for example: <code>invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest</code>.


Some of the provided CI images are very large (even up to 6 GB of disk space), but because CI images are cached, their size does not pose a problem. Additionally, since they already come preconfigured for you, they will have less instructions to run and will consume much less resources from KDE infrastructure.
Some of the provided CI images are very large (even up to 6 GB of disk space), but because the CI images are cached, their size does not pose a problem. Additionally, since they already come preconfigured for you, they will have less instructions to run and will consume much less resources from KDE infrastructure.


== Including CI templates ==
== Including CI templates ==
Line 35: Line 33:
Instead of writing your own custom <code>.gitlab-ci.yml</code> files yourself, to make life easier for KDE developers, Gitlab CI templates are provided in [https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates sysadmin/ci-utilities/gitlab-templates].
Instead of writing your own custom <code>.gitlab-ci.yml</code> files yourself, to make life easier for KDE developers, Gitlab CI templates are provided in [https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates sysadmin/ci-utilities/gitlab-templates].


Your typical Gitlab CI file for a KDE project will likely just use the [https://docs.gitlab.com/ee/ci/yaml/includes.html include: command] to add these template files whenever you need to enable a new job, for example, for building with Qt6, for checking whether your project follows our REUSE guidelines, or for generating flatpak bundles built from the main branch.
Your typical Gitlab CI file for a KDE project will likely just use the [https://docs.gitlab.com/ee/ci/yaml/includes.html include: command] to add these template files whenever you need to enable a new job, for example, for building with Qt6, for checking whether your project follows our [[Guidelines_and_HOWTOs/Licensing|REUSE guidelines]], or for generating flatpak bundles built from the main branch.
 
This is the preferred way to include templates in your CI file:
 
{{bc-hl|lang=yaml|code=
include:
  - project: sysadmin/ci-utilities
    file:
      - /gitlab-templates/reuse-lint.yml
      - /gitlab-templates/linux.yml
      - /gitlab-templates/freebsd.yml
      - /gitlab-templates/windows.yml
      - /gitlab-templates/android.yml
      - /gitlab-templates/linux-qt6.yml
      - /gitlab-templates/freebsd-qt6.yml
      - /gitlab-templates/windows-qt6.yml
      - /gitlab-templates/android-qt6.yml
      - /gitlab-templates/flatpak.yml
}}
 
These are currently the most common builds among the provided templates.
 
Note: The old way to include templates by using raw links to the template files will fail for some of the templates because they include shared base templates with local includes. You'll get an error like
<blockquote>
Unable to create pipeline
* Local file `gitlab-templates/craft-android-base.yml` does not have project!
</blockquote>
 
=== Job artifacts ===
 


To include it, you should use the raw link for the template: go to the template file on Invent, for example [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/linux-qt6.yml linux-qt6], then right-click on "Open Raw" on the right side of the screen, below the commit number and to the right of the Replace/Delete buttons, and click on "Copy Link".
Most CI images and templates for KDE software are primarily designed for continuous integration instead of continuous delivery, but they still generate [https://docs.gitlab.com/ee/ci/jobs/job_artifacts.html job artifacts], files or compressed files that can be accessed outside a job, and that can be transferred from one job to the other. You can view a project's artifacts on https://invent.kde.org from its navigation menu; to view the artifacts produced by CI for a particular Merge Request, view Pipeline > Jobs and click [Browse] in the right-hand sidebar (click [<<] to show a collapsed sidebar), or click the [↓] Download artifacts button for a build.


Here is an example <code>.gitlab-ci.yml</code> file:
Many of the final job artifacts generated by CI pipelines can be seen in [https://invent.kde.org/groups/teams/ci-artifacts/-/packages teams/ci-artifacts/packages].


{{Input|1=<nowiki>
A few templates, such as [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/craft-appimage.yml craft-appimage], [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/flatpak.yml flatpak] and [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/website-hugo.yml website-hugo], are designed for continuous delivery instead of continuous integration. The first two generate <code>.appimage</code> and <code>.flatpak</code> files, which are job artifacts. The last one is used to publish websites. You should set up an app project to [https://develop.kde.org/docs/packaging/flatpak/publishing/#publishing-to-kdes-nightly-repositories publish its flatpak] to a [https://cdn.kde.org/flatpak/ "nightly" repository] so testers can try out and reproduce bugs with a flatpak built from the latest code.
# SPDX-FileCopyrightText: None
 
# SPDX-License-Identifier: CC0-1.0
<!-- note similar text also appears at https://community.kde.org/Windows#Windows_binaries -->
CI produces Windows binaries of several KDE apps; these are collected in the Windows Qt 5.15 project, under [sidebar] > Deploy > Package registry, at https://invent.kde.org/teams/ci-artifacts/windows-qt5.15/-/packages .
 
{{Note|Flatpak and website jobs used to be listed under a Jenkins instance https://binary-factory.kde.org|Old infrastructure}}


include:
=== Special cases ===
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/reuse-lint.yml
Other templates, such as [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/reuse-lint.yml reuse-lint] and [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/json-validation.yml json-validation], consist of small tools used to perform specific tasks on the repository and report the results back if unsuccessful. These do not generate job artifacts.
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd.yml
- https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows.yml
- https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux-qt6.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd-qt6.yml
- https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows-qt6.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/flatpak.yml
</nowiki>}}


These are currently the most common builds among the provided templates. You can simply copy the file and uncomment what job you need for your project.
You can read more about what each template is for in the [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/README.md gitlab-templates readme].


== The .kde-ci.yml file ==
== The .kde-ci.yml file ==
Line 72: Line 94:
Here is an example copied verbatim from the [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml config-template.yml]:
Here is an example copied verbatim from the [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml config-template.yml]:


{{Input|1=<nowiki>
{{bc-hl|lang=yaml|code=
Dependencies:
Dependencies:
- 'on': ['Linux', 'FreeBSD', 'Windows', 'Android']
- 'on': ['Linux', 'FreeBSD', 'Windows', 'Android']
Line 105: Line 127:
   # add files to validate when using json-validation pipeline (for example, to add json files that do not explicitely has a ".json" suffix)
   # add files to validate when using json-validation pipeline (for example, to add json files that do not explicitely has a ".json" suffix)
   json-validate-include: []
   json-validate-include: []
 
}}
</nowiki>}}


=== Dependencies ===
=== Dependencies ===
Line 127: Line 148:
* '''@same''': point to the same branch name as the current project's
* '''@same''': point to the same branch name as the current project's
* '''@stable''' or '''@latest''': point to the branches specified in [https://invent.kde.org/sysadmin/repo-metadata/-/blob/master/branch-rules.yml sysadmin/repo-metadata/branch-rules.yml]
* '''@stable''' or '''@latest''': point to the branches specified in [https://invent.kde.org/sysadmin/repo-metadata/-/blob/master/branch-rules.yml sysadmin/repo-metadata/branch-rules.yml]
* any other version mentioned in branch-rule.yml, such as '''@latest-kf6'''
* any other version mentioned in branch-rules.yml, such as '''@latest-kf6'''


You should only include projects that you explicitly depend on.
You should only include projects that you explicitly depend on.
Line 141: Line 162:
* '''in-source-build''': whether to build the project in its own separate <code>build/</code> folder (out-of-source build) or on the source project root (in-source build)
* '''in-source-build''': whether to build the project in its own separate <code>build/</code> folder (out-of-source build) or on the source project root (in-source build)
* '''cmake-options''': options that can be passed to CMake during build
* '''cmake-options''': options that can be passed to CMake during build
* '''test-before-installing''': whether to run tests before installing the project (using CTest provided by the ECM KDECMakeSettings module)
* '''test-before-installing''': whether to run tests before installing the project (using CTest provided by the [https://api.kde.org/ecm/kde-module/KDECMakeSettings.html ECM KDECMakeSettings module])
* '''run-tests''': whether to run tests (using CTest provided by the [https://api.kde.org/ecm/kde-module/KDECMakeSettings.html ECM KDECMakeSettings module])
* '''run-tests''': whether to run tests (using CTest provided by the ECM KDECMakeSettings module)
* '''tests-load-sensitive''':
* '''tests-load-sensitive''': whether to make tests run only when the load of a CI worker is low, needed for fragile tests that can fail under high load conditions
* '''per-test-timeout''': timeout in seconds per test
* '''per-test-timeout''': timeout in seconds per test
* '''setup-x-environment''': whether to run a graphical session
* '''setup-x-environment''': whether to run a graphical session
* '''setup-dbus-session''': whether to run a D-Bus session
* '''setup-dbus-session''': whether to run a D-Bus session
* '''force-inject-asan''': whether to run the gcc/clang built-in [https://clang.llvm.org/docs/AddressSanitizer.html AddressSanitizer]
* '''force-inject-asan''': whether to use LD_PRELOAD to force inject [https://clang.llvm.org/docs/AddressSanitizer.html AddressSanitizer] in a project (useful for non-KDE libraries that need to load KDE plugins/libraries, sometimes needed in QML applications)
* '''ctest-arguments''': options that can be passed to ctest during tests
* '''ctest-arguments''': options that can be passed to ctest during tests
* '''require-passing-tests-on''': in which platform tests must pass for the job to be successful (same platforms as in Dependencies)
* '''require-passing-tests-on''': in which platform tests must pass for the job to be successful (same platforms as in Dependencies)
Line 155: Line 176:
* '''run-gcovr''': whether to run [https://gcovr.com/en/stable/ gcovr] for checking code coverage
* '''run-gcovr''': whether to run [https://gcovr.com/en/stable/ gcovr] for checking code coverage
* '''gcovr-arguments''': options that can be passed to gcovr
* '''gcovr-arguments''': options that can be passed to gcovr
* '''json-validate-ignore''': which directories/files [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/validate-json-files.py validate-json-files] should ignore
* '''json-validate-ignore''': which directories/files [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/gitlab-templates/json-validation.yml json-validation] should ignore
* '''json-validate-include''': which directories/files validate-json-files should include that do not have the <code>.json</code> file extension
* '''json-validate-include''': which directories/files validate-json-files should include that do not have the <code>.json</code> file extension


=== Examples ===
=== Examples ===
==== All platforms ====


Here is an example copied from Okular, which is built for all platforms available:
Here is an example copied from Okular, which is built for all platforms available:


{{Input|1=<nowiki>
{{bc-hl|lang=yaml|code=
# SPDX-FileCopyrightText: None
# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0
# SPDX-License-Identifier: CC0-1.0
Line 197: Line 220:
   cppcheck-ignore-files:
   cppcheck-ignore-files:
     - autotests
     - autotests
}}


</nowiki>}}
==== KF5 + KF6 ====


And an example copied from Konsole, which at this moment builds against KF5 and KF6:
And an example copied from Konsole, which at this moment builds against KF5 and KF6:


{{Input|1=<nowiki>
{{bc-hl|lang=yaml|code=
# SPDX-FileCopyrightText: None
# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0
# SPDX-License-Identifier: CC0-1.0
Line 267: Line 291:
Options:
Options:
     require-passing-tests-on: [ 'Linux', 'FreeBSD/Qt6' ]
     require-passing-tests-on: [ 'Linux', 'FreeBSD/Qt6' ]
</nowiki>}}
}}
 
==== Warning free code ====
 
{{bc-hl|lang=yaml|code=
Options:
    cmake-options: '-DCMAKE_COMPILE_WARNING_AS_ERROR=ON'
}}
 
== History for archiving reasons ==
 
* Until 2020, KDE used [https://www.jenkins.io/ Jenkins] for its infrastructure, namely over https://build.kde.org and https://binary-factory.kde.org, as described in the [https://community.kde.org/index.php?title=Infrastructure/Continuous_Integration_System&oldid=94998 previous history of this page].
* KDE [https://dot.kde.org/2020/06/30/kdes-gitlab-now-live migrated to Gitlab] in 2020.
* The KDE system administration [https://mail.kde.org/pipermail/kde-frameworks-devel/2021-September/118851.html starts integrating the Gitlab CI workflow with repo-metadata by adding .kde-ci.yml files] to KDE Frameworks in September 2021.
* In October of the same year, it's [https://mail.kde.org/pipermail/kde-frameworks-devel/2021-September/119121.html officially rolled out for all projects], with only Linux builds currently available.
* https://build.kde.org is [https://mail.kde.org/pipermail/kde-devel/2022-September/001270.html retired] in September 2022.
* In October of the same year, there was some [https://mail.kde.org/pipermail/kde-devel/2022-October/001401.html clarification on the roles of CI and CD under KDE infrastructure] in the mailing lists.

Latest revision as of 16:19, 18 April 2024

Continuous Integration (CI) is a process that allows developers to ensure their code is properly tested, follows quality standards and compiles correctly, while Continuous Delivery (CD) is a process that allows to automate the generation of binaries or packages for end users.

KDE ensures the quality of its code with the help of Gitlab CI/CD. The Gitlab CI/CD system consists mostly of compiling software projects inside CI images, usually Docker containers or virtual machines that have their environment prepared for building and testing.

Runners and Jobs

We call a job when software is run inside of a CI image, and in standard Gitlab CI/CD we describe jobs using a .gitlab-ci.yml file. Jobs can be configured to run only under certain circumstances by using rules, so that they can run when making a Merge Request, once a given amount of time has passed, if a certain file exists in the repository, or whenever new code is merged into the main branch.

Additionally, KDE has some additional tooling used to describe the dependency information required by a project, and includes certain options to modify jobs for building KDE projects. These modifications are described in a .kde-ci.yml file, and an example configuration showing all available options can be found in sysadmin/ci-utilities/config-template.yml. A custom dependency-generator is run on these files to update dependency information stored in sysadmin/repo-metadata/dependencies, which is then used for kdesrc-build to determine the order in which to build projects, among other things.

A runner on the other hand is a process that runs jobs. It does not run those jobs in Gitlab itself; it requires additional software to be running in addition to Gitlab. By default, this software is Gitlab Runner which as mentioned before uses a .gitlab-ci.yml file written in YAML. While it is possible to integrate other runner software like Drone CI or Jenkins, these are not used or supported under KDE infrastructure.

System administrators and project maintainers are allowed to create runners, while users without these roles cannot. When runners are created from the system administration interface, they are called shared runners, which all KDE projects can use. These are general use and should meet the needs of most KDE projects. When runners are created from a project's Settings, they are called project runners, and they should only be used when a project has special needs that cannot be met with shared runners.

System administrators, project maintainers and users without these roles are allowed to run jobs.

If you are completely new to Gitlab CI/CD, then you may create your own projects on Invent or create a fork of an existing project to make your own runners and jobs. It is recommended that you use the CI images mentioned below so that your jobs consume less resources from KDE infrastructure.

The CI/CD project settings can be found under the Settings menu on the sidebar when you are a project maintainer or creator.

CI images

As mentioned before, jobs are run on top of CI images. You can find a list of CI images in https://invent.kde.org/sysadmin/ci-images. Any additional shared dependencies for KDE projects must be added here.

Developers are expected not to make their own custom CI files using these images. Instead, use the Gitlab Templates mentioned in the next section.

To fetch the container images from the KDE Container Registry, you can simply copy the URL for the image, replace https://invent.kde.org with invent-registry.kde.org and add the container tag at the end if applicable (usually :latest), for example: invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest.

Some of the provided CI images are very large (even up to 6 GB of disk space), but because the CI images are cached, their size does not pose a problem. Additionally, since they already come preconfigured for you, they will have less instructions to run and will consume much less resources from KDE infrastructure.

Including CI templates

Instead of writing your own custom .gitlab-ci.yml files yourself, to make life easier for KDE developers, Gitlab CI templates are provided in sysadmin/ci-utilities/gitlab-templates.

Your typical Gitlab CI file for a KDE project will likely just use the include: command to add these template files whenever you need to enable a new job, for example, for building with Qt6, for checking whether your project follows our REUSE guidelines, or for generating flatpak bundles built from the main branch.

This is the preferred way to include templates in your CI file:

include:
  - project: sysadmin/ci-utilities
    file:
      - /gitlab-templates/reuse-lint.yml
      - /gitlab-templates/linux.yml
      - /gitlab-templates/freebsd.yml
      - /gitlab-templates/windows.yml
      - /gitlab-templates/android.yml
      - /gitlab-templates/linux-qt6.yml
      - /gitlab-templates/freebsd-qt6.yml
      - /gitlab-templates/windows-qt6.yml
      - /gitlab-templates/android-qt6.yml
      - /gitlab-templates/flatpak.yml

These are currently the most common builds among the provided templates.

Note: The old way to include templates by using raw links to the template files will fail for some of the templates because they include shared base templates with local includes. You'll get an error like

Unable to create pipeline

  • Local file `gitlab-templates/craft-android-base.yml` does not have project!

Job artifacts

Most CI images and templates for KDE software are primarily designed for continuous integration instead of continuous delivery, but they still generate job artifacts, files or compressed files that can be accessed outside a job, and that can be transferred from one job to the other. You can view a project's artifacts on https://invent.kde.org from its navigation menu; to view the artifacts produced by CI for a particular Merge Request, view Pipeline > Jobs and click [Browse] in the right-hand sidebar (click [<<] to show a collapsed sidebar), or click the [↓] Download artifacts button for a build.

Many of the final job artifacts generated by CI pipelines can be seen in teams/ci-artifacts/packages.

A few templates, such as craft-appimage, flatpak and website-hugo, are designed for continuous delivery instead of continuous integration. The first two generate .appimage and .flatpak files, which are job artifacts. The last one is used to publish websites. You should set up an app project to publish its flatpak to a "nightly" repository so testers can try out and reproduce bugs with a flatpak built from the latest code.

CI produces Windows binaries of several KDE apps; these are collected in the Windows Qt 5.15 project, under [sidebar] > Deploy > Package registry, at https://invent.kde.org/teams/ci-artifacts/windows-qt5.15/-/packages .

Old infrastructure

Flatpak and website jobs used to be listed under a Jenkins instance https://binary-factory.kde.org


Special cases

Other templates, such as reuse-lint and json-validation, consist of small tools used to perform specific tasks on the repository and report the results back if unsuccessful. These do not generate job artifacts.

You can read more about what each template is for in the gitlab-templates readme.

The .kde-ci.yml file

As mentioned before, .kde-ci.yml files contain dependency information and instructions that modify jobs to suit the needs of a project.

The purpose of this file is to centralize settings accross multiple jobs while having control over each individual job, all without needing to write your own custom .gitlab-ci.yml files.

It has three main settings:

  • Dependencies: a listing of build dependencies that vary according to the platform and builds.
  • Environment: a listing of variables that can be used inside a job.
  • Options: a listing of additional options that can be passed to jobs to ensure their code quality.

Here is an example copied verbatim from the config-template.yml:

Dependencies:
- 'on': ['Linux', 'FreeBSD', 'Windows', 'Android']
  'require':
    'frameworks/*': '@stable'
    'kde/workspace/kdecoration': '@stable'

Environment:
  Variable: SOMEVALUE

Options:
  in-source-build: False
  cmake-options: ''
  test-before-installing: False
  run-tests: True
  tests-load-sensitive: False
  per-test-timeout: 60
  setup-x-environment: True
  setup-dbus-session: True
  force-inject-asan: False
  ctest-arguments: ''
  # a list of platforms on which unit test failures are considered fatal
  require-passing-tests-on: []
  run-cppcheck: True
  cppcheck-arguments: '--enable=warning,style,performance'
  cppcheck-ignore-files:
    - src/3rdparty/
  run-gcovr: True
  gcovr-arguments: ''
  # add json files to ignore when using json-validation pipeline
  json-validate-ignore: []
  # add files to validate when using json-validation pipeline (for example, to add json files that do not explicitely has a ".json" suffix)
  json-validate-include: []

Dependencies

Dependencies is the most volatile of all settings, because it will need to be updated every time a new build dependency is added to the project.

The on section allows you to select which platforms the project will be built for and can be repeated as many times as needed using different combinations of platforms. The available platforms are:

  • Linux
  • FreeBSD
  • Windows
  • macOS
  • Android
  • @all

The platform names are case sensitive. The @all platform serves as a shorthand for all platforms (instead of typing them all).

The require section allows you to mention the dependency and its version. The dependency name can be found under sysadmin/repo-metadata/dependencies, and it matches the project path on Invent. You can state each dependency version as:

  • @same: point to the same branch name as the current project's
  • @stable or @latest: point to the branches specified in sysadmin/repo-metadata/branch-rules.yml
  • any other version mentioned in branch-rules.yml, such as @latest-kf6

You should only include projects that you explicitly depend on.

Options

Options is less volatile and is rarely changed.

Additional attention should be paid to it: you should NOT copy all options from the template file. Only add options as needed.

The available options are:

  • in-source-build: whether to build the project in its own separate build/ folder (out-of-source build) or on the source project root (in-source build)
  • cmake-options: options that can be passed to CMake during build
  • test-before-installing: whether to run tests before installing the project (using CTest provided by the ECM KDECMakeSettings module)
  • run-tests: whether to run tests (using CTest provided by the ECM KDECMakeSettings module)
  • tests-load-sensitive: whether to make tests run only when the load of a CI worker is low, needed for fragile tests that can fail under high load conditions
  • per-test-timeout: timeout in seconds per test
  • setup-x-environment: whether to run a graphical session
  • setup-dbus-session: whether to run a D-Bus session
  • force-inject-asan: whether to use LD_PRELOAD to force inject AddressSanitizer in a project (useful for non-KDE libraries that need to load KDE plugins/libraries, sometimes needed in QML applications)
  • ctest-arguments: options that can be passed to ctest during tests
  • require-passing-tests-on: in which platform tests must pass for the job to be successful (same platforms as in Dependencies)
  • run-cppcheck: whether to run cppcheck linting for static analysis
  • cppcheck-arguments: options that can be passed to cppcheck
  • cppcheck-ignore-files: which directories/files cppcheck should ignore
  • run-gcovr: whether to run gcovr for checking code coverage
  • gcovr-arguments: options that can be passed to gcovr
  • json-validate-ignore: which directories/files json-validation should ignore
  • json-validate-include: which directories/files validate-json-files should include that do not have the .json file extension

Examples

All platforms

Here is an example copied from Okular, which is built for all platforms available:

# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0

Dependencies:
- 'on': ['@all']
  'require':
    'frameworks/kbookmarks': '@stable'
    'frameworks/threadweaver': '@stable'
    'frameworks/ki18n': '@stable'
    'frameworks/kio': '@stable'
    'frameworks/karchive': '@stable'
    'libraries/phonon': '@stable'
    'graphics/kdegraphics-mobipocket': '@same'

- 'on': ['Linux', 'FreeBSD', 'Windows']
  'require':
    'frameworks/khtml': '@stable'
    'frameworks/purpose': '@stable'
    'frameworks/breeze-icons': '@stable'
    'graphics/libkexiv2': '@same'

- 'on': ['Linux', 'FreeBSD']
  'require':
    'frameworks/kactivities': '@stable'
    'frameworks/kpty': '@stable'

- 'on': ['Android']
  'require':
    'frameworks/kirigami': '@stable'

Options:
  cppcheck-arguments: '--enable=warning,style,performance -DOKULAR_EXPORT_PLUGIN'
  cppcheck-ignore-files:
    - autotests

KF5 + KF6

And an example copied from Konsole, which at this moment builds against KF5 and KF6:

# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0

Dependencies:
- 'on': ['Linux/Qt5', 'FreeBSD/Qt5', 'Windows/Qt5']
  'require':
    'frameworks/extra-cmake-modules': '@stable'
    'frameworks/kconfig': '@stable'
    'frameworks/knotifications': '@stable'
    'frameworks/ki18n': '@stable'
    'frameworks/kcoreaddons': '@stable'
    'frameworks/kdbusaddons': '@stable'
    'frameworks/kbookmarks': '@stable'
    'frameworks/kconfigwidgets': '@stable'
    'frameworks/kcrash': '@stable'
    'frameworks/kguiaddons': '@stable'
    'frameworks/kiconthemes': '@stable'
    'frameworks/kio': '@stable'
    'frameworks/knewstuff': '@stable'
    'frameworks/knotifyconfig': '@stable'
    'frameworks/kparts': '@stable'
    'frameworks/kservice': '@stable'
    'frameworks/ktextwidgets': '@stable'
    'frameworks/kwidgetsaddons': '@stable'
    'frameworks/kwindowsystem': '@stable'
    'frameworks/kxmlgui': '@stable'
    'frameworks/kdoctools': '@stable'

- 'on': ['Linux/Qt6', 'FreeBSD/Qt6', 'Windows/Qt6']
  'require':
    'frameworks/extra-cmake-modules': '@latest-kf6'
    'frameworks/kconfig': '@latest-kf6'
    'frameworks/knotifications': '@latest-kf6'
    'frameworks/ki18n': '@latest-kf6'
    'frameworks/kcoreaddons': '@latest-kf6'
    'frameworks/kdbusaddons': '@latest-kf6'
    'frameworks/kbookmarks': '@latest-kf6'
    'frameworks/kconfigwidgets': '@latest-kf6'
    'frameworks/kcrash': '@latest-kf6'
    'frameworks/kguiaddons': '@latest-kf6'
    'frameworks/kiconthemes': '@latest-kf6'
    'frameworks/kio': '@latest-kf6'
    'frameworks/knewstuff': '@latest-kf6'
    'frameworks/knotifyconfig': '@latest-kf6'
    'frameworks/kparts': '@latest-kf6'
    'frameworks/kservice': '@latest-kf6'
    'frameworks/ktextwidgets': '@latest-kf6'
    'frameworks/kwidgetsaddons': '@latest-kf6'
    'frameworks/kwindowsystem': '@latest-kf6'
    'frameworks/kxmlgui': '@latest-kf6'
    'frameworks/kdoctools': '@latest-kf6'

- 'on': ['Linux/Qt5', 'FreeBSD/Qt5']
  'require':
    'frameworks/kpty': '@stable'
    'frameworks/kglobalaccel': '@stable'

- 'on': ['Linux/Qt6', 'FreeBSD/Qt6']
  'require':
    'frameworks/kpty': '@latest-kf6'
    'frameworks/kglobalaccel': '@latest-kf6'

Options:
    require-passing-tests-on: [ 'Linux', 'FreeBSD/Qt6' ]

Warning free code

Options:
    cmake-options: '-DCMAKE_COMPILE_WARNING_AS_ERROR=ON'

History for archiving reasons