Infrastructure/Continuous Integration System: Difference between revisions
Thiagosueto (talk | contribs) (Remove all Jenkins and add first draft about Gitlab CI) |
Thiagosueto (talk | contribs) m (Minor language changes) |
||
| Line 9: | Line 9: | ||
Continuous Integration (CI) is a process that allows developers to ensure their code is properly tested, follows quality standards and compiles correctly, while Continuous Delivery (CD) is a process that allows to automate the generation of binaries or packages for end users. | Continuous Integration (CI) is a process that allows developers to ensure their code is properly tested, follows quality standards and compiles correctly, while Continuous Delivery (CD) is a process that allows to automate the generation of binaries or packages for end users. | ||
KDE ensures the quality of its code with the help of [https://docs.gitlab.com/ee/ci/ Gitlab CI/CD]. The Gitlab CI/CD system consists mostly of compiling software projects inside [https://docs.docker.com/get-started/overview/ Docker containers] or virtual machines | KDE ensures the quality of its code with the help of [https://docs.gitlab.com/ee/ci/ Gitlab CI/CD]. The Gitlab CI/CD system consists mostly of compiling software projects inside ''CI images'', usually [https://docs.docker.com/get-started/overview/ Docker containers] or virtual machines that have their environment prepared for building and testing. | ||
== Runners and Jobs == | == Runners and Jobs == | ||
We call a ''job'' when software is run inside one of these containers or virtual machines, and in standard Gitlab CI/CD we describe jobs using a [https://docs.gitlab.com/ee/ci/yaml/ .gitlab-ci.yml] file. Jobs can run only under certain circumstances by using [https://docs.gitlab.com/ee/ci/jobs/job_control.html rules], | We call a ''job'' when software is run inside one of these containers or virtual machines, and in standard Gitlab CI/CD we describe jobs using a [https://docs.gitlab.com/ee/ci/yaml/ .gitlab-ci.yml] file. Jobs can be configured to run only under certain circumstances by using [https://docs.gitlab.com/ee/ci/jobs/job_control.html rules], so that they can run when making a Merge Request, once a given amount of time has passed, if a certain file exists in the repository, or whenever new code is merged into the main branch. | ||
Additionally, KDE has some custom integrated tooling used to modify jobs for building KDE projects | Additionally, KDE has some custom integrated tooling used to modify jobs for building KDE projects. These modifications are described in a <code>.kde-ci.yml</code> file, and an example configuration showing all available options can be found in [https://invent.kde.org/sysadmin/ci-utilities/-/blob/master/config-template.yml sysadmin/ci-utilities/config-template.yml]. A custom [https://invent.kde.org/nicolasfella/dependency-generator dependency-generator] is run on these files to update dependency information stored in [https://invent.kde.org/sysadmin/repo-metadata/-/tree/master/dependencies sysadmin/repo-metadata/dependencies], which is then used for [https://community.kde.org/Get_Involved/development/Set_up_a_development_environment kdesrc-build] to determine the order in which to build projects, among other things. | ||
A ''runner'' on the other hand is a process that runs jobs. It does not run those jobs in Gitlab itself; it requires additional software to be running in addition to Gitlab. By default, this software is [https://docs.gitlab.com/runner/ Gitlab Runner] which as mentioned before uses a <code>.gitlab-ci.yml</code> file written in [https://en.wikipedia.org/wiki/YAML YAML] | A ''runner'' on the other hand is a process that runs jobs. It does not run those jobs in Gitlab itself; it requires additional software to be running in addition to Gitlab. By default, this software is [https://docs.gitlab.com/runner/ Gitlab Runner] which as mentioned before uses a <code>.gitlab-ci.yml</code> file written in [https://en.wikipedia.org/wiki/YAML YAML]. While it is possible to [https://docs.gitlab.com/ee/integration/ integrate] other runner software like [https://www.drone.io/ Drone CI] or [https://www.jenkins.io/ Jenkins], these are not used or supported under KDE infrastructure. | ||
System administrators and project maintainers are allowed to create runners, while users without these roles cannot. When runners are created from the system administration interface, they are called ''shared runners'', which all KDE projects can use. These are general use and should meet the needs of most KDE projects. When runners are created from a project's | System administrators and project maintainers are allowed to create runners, while users without these roles cannot. When runners are created from the system administration interface, they are called ''shared runners'', which all KDE projects can use. These are general use and should meet the needs of most KDE projects. When runners are created from a project's Settings, they are called ''project runners'', and they should only be used when a project has special needs that cannot be met with shared runners. | ||
System administrators, project maintainers and users without these roles are allowed to run jobs. | System administrators, project maintainers and users without these roles are allowed to run jobs. | ||
If you are completely new to Gitlab CI/CD, then you may create your own projects on [https://invent.kde.org Invent] or create a fork of an existing project to | If you are completely new to Gitlab CI/CD, then you may create your own projects on [https://invent.kde.org Invent] or create a fork of an existing project to make your own runners and jobs. It is recommended that you use the CI images mentioned below so that your jobs consume less resources from KDE infrastructure. | ||
== CI images == | == CI images == | ||
| Line 29: | Line 29: | ||
As mentioned before, jobs are run on top of CI images. You can find a list of CI images in [https://invent.kde.org/sysadmin/ci-images https://invent.kde.org/sysadmin/ci-images]. To use them in a <code>.gitlab-ci.yml</code> file, you can replace <code><nowiki>https://invent.kde.org</nowiki></code> with <code>invent-registry.kde.org</code> and add the container tag at the end if applicable (usually <code>:latest</code>), for example: <code>invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest</code>. | As mentioned before, jobs are run on top of CI images. You can find a list of CI images in [https://invent.kde.org/sysadmin/ci-images https://invent.kde.org/sysadmin/ci-images]. To use them in a <code>.gitlab-ci.yml</code> file, you can replace <code><nowiki>https://invent.kde.org</nowiki></code> with <code>invent-registry.kde.org</code> and add the container tag at the end if applicable (usually <code>:latest</code>), for example: <code>invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest</code>. | ||
Some of the provided CI images are very large (even up to 6 GB of disk space), but because CI images are cached, their size does not pose a problem. Additionally, since they | Some of the provided CI images are very large (even up to 6 GB of disk space), but because CI images are cached, their size does not pose a problem. Additionally, since they already come preconfigured for you, they will have less instructions to run and will consume much less resources from KDE infrastructure. | ||
== Including CI templates == | == Including CI templates == | ||
Instead of writing your own <code>.gitlab-ci.yml</code> files yourself, to make life easier for KDE developers, Gitlab CI templates are provided in [https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates sysadmin/ci-utilities/gitlab-templates]. | Instead of writing your own custom <code>.gitlab-ci.yml</code> files yourself, to make life easier for KDE developers, Gitlab CI templates are provided in [https://invent.kde.org/sysadmin/ci-utilities/-/tree/master/gitlab-templates sysadmin/ci-utilities/gitlab-templates]. | ||
Your typical Gitlab CI file for a KDE project will likely just use the [https://docs.gitlab.com/ee/ci/yaml/includes.html include: command] to add these template files whenever you need to enable a new job, for example, for building with Qt6, for checking whether your project follows our REUSE guidelines, or for generating flatpak bundles built from the main branch. | Your typical Gitlab CI file for a KDE project will likely just use the [https://docs.gitlab.com/ee/ci/yaml/includes.html include: command] to add these template files whenever you need to enable a new job, for example, for building with Qt6, for checking whether your project follows our REUSE guidelines, or for generating flatpak bundles built from the main branch. | ||
| Line 56: | Line 56: | ||
</nowiki>}} | </nowiki>}} | ||
These are currently the most common builds | These are currently the most common builds among the provided templates. You can simply copy the file and uncomment what job you need for your project. | ||
Revision as of 00:01, 14 October 2023
Warning
- Rollout of Gitlab CI: https://mail.kde.org/pipermail/kde-devel/2021-October/000735.html
- Gitlab CI Dashboards and retirement of build.kde.org: https://mail.kde.org/pipermail/kde-devel/2022-September/001270.html
- Debugging KDE CI runners: https://mail.kde.org/pipermail/kde-devel/2022-October/001401.html
Continuous Integration (CI) is a process that allows developers to ensure their code is properly tested, follows quality standards and compiles correctly, while Continuous Delivery (CD) is a process that allows to automate the generation of binaries or packages for end users.
KDE ensures the quality of its code with the help of Gitlab CI/CD. The Gitlab CI/CD system consists mostly of compiling software projects inside CI images, usually Docker containers or virtual machines that have their environment prepared for building and testing.
Runners and Jobs
We call a job when software is run inside one of these containers or virtual machines, and in standard Gitlab CI/CD we describe jobs using a .gitlab-ci.yml file. Jobs can be configured to run only under certain circumstances by using rules, so that they can run when making a Merge Request, once a given amount of time has passed, if a certain file exists in the repository, or whenever new code is merged into the main branch.
Additionally, KDE has some custom integrated tooling used to modify jobs for building KDE projects. These modifications are described in a .kde-ci.yml file, and an example configuration showing all available options can be found in sysadmin/ci-utilities/config-template.yml. A custom dependency-generator is run on these files to update dependency information stored in sysadmin/repo-metadata/dependencies, which is then used for kdesrc-build to determine the order in which to build projects, among other things.
A runner on the other hand is a process that runs jobs. It does not run those jobs in Gitlab itself; it requires additional software to be running in addition to Gitlab. By default, this software is Gitlab Runner which as mentioned before uses a .gitlab-ci.yml file written in YAML. While it is possible to integrate other runner software like Drone CI or Jenkins, these are not used or supported under KDE infrastructure.
System administrators and project maintainers are allowed to create runners, while users without these roles cannot. When runners are created from the system administration interface, they are called shared runners, which all KDE projects can use. These are general use and should meet the needs of most KDE projects. When runners are created from a project's Settings, they are called project runners, and they should only be used when a project has special needs that cannot be met with shared runners.
System administrators, project maintainers and users without these roles are allowed to run jobs.
If you are completely new to Gitlab CI/CD, then you may create your own projects on Invent or create a fork of an existing project to make your own runners and jobs. It is recommended that you use the CI images mentioned below so that your jobs consume less resources from KDE infrastructure.
CI images
As mentioned before, jobs are run on top of CI images. You can find a list of CI images in https://invent.kde.org/sysadmin/ci-images. To use them in a .gitlab-ci.yml file, you can replace https://invent.kde.org with invent-registry.kde.org and add the container tag at the end if applicable (usually :latest), for example: invent-registry.kde.org/sysadmin/ci-images/suse-qt65:latest.
Some of the provided CI images are very large (even up to 6 GB of disk space), but because CI images are cached, their size does not pose a problem. Additionally, since they already come preconfigured for you, they will have less instructions to run and will consume much less resources from KDE infrastructure.
Including CI templates
Instead of writing your own custom .gitlab-ci.yml files yourself, to make life easier for KDE developers, Gitlab CI templates are provided in sysadmin/ci-utilities/gitlab-templates.
Your typical Gitlab CI file for a KDE project will likely just use the include: command to add these template files whenever you need to enable a new job, for example, for building with Qt6, for checking whether your project follows our REUSE guidelines, or for generating flatpak bundles built from the main branch.
To include it, you should use the raw link for the template: go to the template file on Invent, for example linux-qt6, then right-click on "Open Raw" on the right side of the screen, below the commit number and to the right of the Replace/Delete buttons, and click on "Copy Link".
Here is an example .gitlab-ci.yml file:
# SPDX-FileCopyrightText: None # SPDX-License-Identifier: CC0-1.0 include: # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/reuse-lint.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/linux-qt6.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/freebsd-qt6.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/windows-qt6.yml # - https://invent.kde.org/sysadmin/ci-utilities/raw/master/gitlab-templates/flatpak.yml
These are currently the most common builds among the provided templates. You can simply copy the file and uncomment what job you need for your project.