Infrastructure/Continuous Integration System

From KDE Community Wiki
< Infrastructure
Revision as of 01:36, 14 October 2023 by Thiagosueto (talk | contribs) (Add archival history session for further reading, remove warning about being outdated)

Continuous Integration (CI) is a process that allows developers to ensure their code is properly tested, follows quality standards and compiles correctly, while Continuous Delivery (CD) is a process that allows to automate the generation of binaries or packages for end users.

KDE ensures the quality of its code with the help of Gitlab CI/CD. The Gitlab CI/CD system consists mostly of compiling software projects inside CI images, usually Docker containers or virtual machines that have their environment prepared for building and testing.

Runners and Jobs

We call a job when software is run inside one of these containers or virtual machines, and in standard Gitlab CI/CD we describe jobs using a .gitlab-ci.yml file. Jobs can be configured to run only under certain circumstances by using rules, so that they can run when making a Merge Request, once a given amount of time has passed, if a certain file exists in the repository, or whenever new code is merged into the main branch.

Additionally, KDE has some additional tooling used to describe the dependency information required by a project, and includes certain options to modify jobs for building KDE projects. These modifications are described in a .kde-ci.yml file, and an example configuration showing all available options can be found in sysadmin/ci-utilities/config-template.yml. A custom dependency-generator is run on these files to update dependency information stored in sysadmin/repo-metadata/dependencies, which is then used for kdesrc-build to determine the order in which to build projects, among other things.

A runner on the other hand is a process that runs jobs. It does not run those jobs in Gitlab itself; it requires additional software to be running in addition to Gitlab. By default, this software is Gitlab Runner which as mentioned before uses a .gitlab-ci.yml file written in YAML. While it is possible to integrate other runner software like Drone CI or Jenkins, these are not used or supported under KDE infrastructure.

System administrators and project maintainers are allowed to create runners, while users without these roles cannot. When runners are created from the system administration interface, they are called shared runners, which all KDE projects can use. These are general use and should meet the needs of most KDE projects. When runners are created from a project's Settings, they are called project runners, and they should only be used when a project has special needs that cannot be met with shared runners.

System administrators, project maintainers and users without these roles are allowed to run jobs.

If you are completely new to Gitlab CI/CD, then you may create your own projects on Invent or create a fork of an existing project to make your own runners and jobs. It is recommended that you use the CI images mentioned below so that your jobs consume less resources from KDE infrastructure.

CI images

As mentioned before, jobs are run on top of CI images. You can find a list of CI images in https://invent.kde.org/sysadmin/ci-images. To use them in a .gitlab-ci.yml file, you can replace https://invent.kde.org with invent-registry.kde.org and add the container tag at the end if applicable (usually :latest), for example: invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest.

Some of the provided CI images are very large (even up to 6 GB of disk space), but because CI images are cached, their size does not pose a problem. Additionally, since they already come preconfigured for you, they will have less instructions to run and will consume much less resources from KDE infrastructure.

Including CI templates

Instead of writing your own custom .gitlab-ci.yml files yourself, to make life easier for KDE developers, Gitlab CI templates are provided in sysadmin/ci-utilities/gitlab-templates.

Your typical Gitlab CI file for a KDE project will likely just use the include: command to add these template files whenever you need to enable a new job, for example, for building with Qt6, for checking whether your project follows our REUSE guidelines, or for generating flatpak bundles built from the main branch.

To include it, you should use the raw link for the template: go to the template file on Invent, for example linux-qt6, then right-click on "Open Raw" on the right side of the screen, below the commit number and to the right of the Replace/Delete buttons, and click on "Copy Link".

Here is an example .gitlab-ci.yml file: 

# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0

include:
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/reuse-lint.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux-qt6.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd-qt6.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows-qt6.yml
#  - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/flatpak.yml

These are currently the most common builds among the provided templates. You can simply copy the file and uncomment what job you need for your project.

The .kde-ci.yml file

As mentioned before, .kde-ci.yml files contain dependency information and instructions that modify jobs to suit the needs of a project.

The purpose of this file is to centralize settings accross multiple jobs while having control over each individual job, all without needing to write your own custom .gitlab-ci.yml files.

It has three main settings:

  • Dependencies: a listing of build dependencies that vary according to the platform and builds.
  • Environment: a listing of variables that can be used inside a job.
  • Options: a listing of additional options that can be passed to jobs to ensure their code quality.

Here is an example copied verbatim from the config-template.yml: 

Dependencies:
- 'on': ['Linux', 'FreeBSD', 'Windows', 'Android']
  'require':
    'frameworks/*': '@stable'
    'kde/workspace/kdecoration': '@stable'

Environment:
  Variable: SOMEVALUE

Options:
  in-source-build: False
  cmake-options: ''
  test-before-installing: False
  run-tests: True
  tests-load-sensitive: False
  per-test-timeout: 60
  setup-x-environment: True
  setup-dbus-session: True
  force-inject-asan: False
  ctest-arguments: ''
  # a list of platforms on which unit test failures are considered fatal
  require-passing-tests-on: []
  run-cppcheck: True
  cppcheck-arguments: '--enable=warning,style,performance'
  cppcheck-ignore-files:
    - src/3rdparty/
  run-gcovr: True
  gcovr-arguments: ''
  # add json files to ignore when using json-validation pipeline
  json-validate-ignore: []
  # add files to validate when using json-validation pipeline (for example, to add json files that do not explicitely has a ".json" suffix)
  json-validate-include: []

Dependencies

Dependencies is the most volatile of all settings, because it will need to be updated every time a new build dependency is added to the project.

The on section allows you to select which platforms the project will be built for and can be repeated as many times as needed using different combinations of platforms. The available platforms are:

  • Linux
  • FreeBSD
  • Windows
  • macOS
  • Android
  • @all

The platform names are case sensitive. The @all platform serves as a shorthand for all platforms (instead of typing them all).

The require section allows you to mention the dependency and its version. The dependency name can be found under sysadmin/repo-metadata/dependencies, and it matches the project path on Invent. You can state each dependency version as:

  • @same: point to the same branch name as the current project's
  • @stable or @latest: point to the branches specified in sysadmin/repo-metadata/branch-rules.yml
  • any other version mentioned in branch-rule.yml, such as @latest-kf6

You should only include projects that you explicitly depend on.

Options

Options is less volatile and is rarely changed.

Additional attention should be paid to it: you should NOT copy all options from the template file. Only add options as needed.

The available options are:

  • in-source-build: whether to build the project in its own separate build/ folder (out-of-source build) or on the source project root (in-source build)
  • cmake-options: options that can be passed to CMake during build
  • test-before-installing: whether to run tests before installing the project (using CTest provided by the ECM KDECMakeSettings module)
  • run-tests: whether to run tests (using CTest provided by the ECM KDECMakeSettings module)
  • tests-load-sensitive:
  • per-test-timeout: timeout in seconds per test
  • setup-x-environment: whether to run a graphical session
  • setup-dbus-session: whether to run a D-Bus session
  • force-inject-asan: whether to run the gcc/clang built-in AddressSanitizer
  • ctest-arguments: options that can be passed to ctest during tests
  • require-passing-tests-on: in which platform tests must pass for the job to be successful (same platforms as in Dependencies)
  • run-cppcheck: whether to run cppcheck linting for static analysis
  • cppcheck-arguments: options that can be passed to cppcheck
  • cppcheck-ignore-files: which directories/files cppcheck should ignore
  • run-gcovr: whether to run gcovr for checking code coverage
  • gcovr-arguments: options that can be passed to gcovr
  • json-validate-ignore: which directories/files validate-json-files should ignore
  • json-validate-include: which directories/files validate-json-files should include that do not have the .json file extension

Examples

Here is an example copied from Okular, which is built for all platforms available: 

# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0

Dependencies:
- 'on': ['@all']
  'require':
    'frameworks/kbookmarks': '@stable'
    'frameworks/threadweaver': '@stable'
    'frameworks/ki18n': '@stable'
    'frameworks/kio': '@stable'
    'frameworks/karchive': '@stable'
    'libraries/phonon': '@stable'
    'graphics/kdegraphics-mobipocket': '@same'

- 'on': ['Linux', 'FreeBSD', 'Windows']
  'require':
    'frameworks/khtml': '@stable'
    'frameworks/purpose': '@stable'
    'frameworks/breeze-icons': '@stable'
    'graphics/libkexiv2': '@same'

- 'on': ['Linux', 'FreeBSD']
  'require':
    'frameworks/kactivities': '@stable'
    'frameworks/kpty': '@stable'

- 'on': ['Android']
  'require':
    'frameworks/kirigami': '@stable'

Options:
  cppcheck-arguments: '--enable=warning,style,performance -DOKULAR_EXPORT_PLUGIN'
  cppcheck-ignore-files:
    - autotests

And an example copied from Konsole, which at this moment builds against KF5 and KF6: 

# SPDX-FileCopyrightText: None
# SPDX-License-Identifier: CC0-1.0

Dependencies:
- 'on': ['Linux/Qt5', 'FreeBSD/Qt5', 'Windows/Qt5']
  'require':
    'frameworks/extra-cmake-modules': '@stable'
    'frameworks/kconfig': '@stable'
    'frameworks/knotifications': '@stable'
    'frameworks/ki18n': '@stable'
    'frameworks/kcoreaddons': '@stable'
    'frameworks/kdbusaddons': '@stable'
    'frameworks/kbookmarks': '@stable'
    'frameworks/kconfigwidgets': '@stable'
    'frameworks/kcrash': '@stable'
    'frameworks/kguiaddons': '@stable'
    'frameworks/kiconthemes': '@stable'
    'frameworks/kio': '@stable'
    'frameworks/knewstuff': '@stable'
    'frameworks/knotifyconfig': '@stable'
    'frameworks/kparts': '@stable'
    'frameworks/kservice': '@stable'
    'frameworks/ktextwidgets': '@stable'
    'frameworks/kwidgetsaddons': '@stable'
    'frameworks/kwindowsystem': '@stable'
    'frameworks/kxmlgui': '@stable'
    'frameworks/kdoctools': '@stable'

- 'on': ['Linux/Qt6', 'FreeBSD/Qt6', 'Windows/Qt6']
  'require':
    'frameworks/extra-cmake-modules': '@latest-kf6'
    'frameworks/kconfig': '@latest-kf6'
    'frameworks/knotifications': '@latest-kf6'
    'frameworks/ki18n': '@latest-kf6'
    'frameworks/kcoreaddons': '@latest-kf6'
    'frameworks/kdbusaddons': '@latest-kf6'
    'frameworks/kbookmarks': '@latest-kf6'
    'frameworks/kconfigwidgets': '@latest-kf6'
    'frameworks/kcrash': '@latest-kf6'
    'frameworks/kguiaddons': '@latest-kf6'
    'frameworks/kiconthemes': '@latest-kf6'
    'frameworks/kio': '@latest-kf6'
    'frameworks/knewstuff': '@latest-kf6'
    'frameworks/knotifyconfig': '@latest-kf6'
    'frameworks/kparts': '@latest-kf6'
    'frameworks/kservice': '@latest-kf6'
    'frameworks/ktextwidgets': '@latest-kf6'
    'frameworks/kwidgetsaddons': '@latest-kf6'
    'frameworks/kwindowsystem': '@latest-kf6'
    'frameworks/kxmlgui': '@latest-kf6'
    'frameworks/kdoctools': '@latest-kf6'

- 'on': ['Linux/Qt5', 'FreeBSD/Qt5']
  'require':
    'frameworks/kpty': '@stable'
    'frameworks/kglobalaccel': '@stable'

- 'on': ['Linux/Qt6', 'FreeBSD/Qt6']
  'require':
    'frameworks/kpty': '@latest-kf6'
    'frameworks/kglobalaccel': '@latest-kf6'

Options:
    require-passing-tests-on: [ 'Linux', 'FreeBSD/Qt6' ]


History for archiving reasons

Retrieved from "https://community.kde.org/index.php?title=Infrastructure/Continuous_Integration_System&oldid=98624"