Revision as of 14:47, 27 September 2016

Neon uses a Jenkins continuous integration system to build its packages

The Setup

drax.kde.org is the master server, owned by Blue Systems and administrated by KDE Sysadmins. It runs a Jenkins instance which is a Continuous Integration website at build.neon.kde.org that has many jobs to build the packages and run other functions, either on demand or at pre-scheduled intervals.

The code behind build.neon is from pangea-tooling which also runs the code for DCI Debian CI, KCI Kubuntu CI, ACI Appstream CI, and MCI Mobile neon Plasma CI.

The Jenkins jobs farm off the hard build work to 4 DigitalOcean build servers. It runs most jobs inside a Docker container to give a fresh build environment.

To use the scripts to access Jenkins you will need to set ~/.config/pangea-jenkins.json using access key available inside Jenkins to administrators.

The setup of the machines is maintained in pangea-kitchen which uses Chef to set up the servers with software all configured.

The Jobs

The Jenkins jobs are created by running the pangea-tooling script jenkins_jobs_update_nci.rb. This creates some manual Jobs specified in the script such as the ISO jobs but mostly uses factories to create batches of jobs based on archives.

The YAML files in pangea-conf-projects define what jobs get created.

For each package there is a parent MultiJob which runs some sub jobs.

parent job this is set to checkout the relevant archive from KDE Git as source/ (for Developer Editions), then check out the relevant archive from KDE neon Git as packaging/. It then runs a number of child jobs...

src will create the source package. For User Edition this means running uscan to use the debian/watch file to download the relevant tar, for Dev Editions it uses the source the parent job checked out. It then builds the source package.

bin job will extract the source, install the build dependencies and compile the package. It finishes by checking the output from lintian and fails on any errors, you can override errors with lintian-overrides files in the normal .deb packaging method (see dh_lintian). It also checks for any list-missing files and fails if there are any, override by adding a debian/not-installed files. It also fails if cmake reports build-dependencies it needs, override with debian/meta cmake-ignore.

adt job runs Debian's test framework autopkgtest. See Ubuntu guide for some details. It runs adt-run on the binaries which installs them and runs the relevant test suite as defined in debian/tests/. It doesn't fail if tests fail.

pub job will upload to aptly, see The Archive below.

lintqml job will scan for QML dependencies which have not been satisfied by the package dependencies, it will print a JSON output of any missing QML modules it requests. The packager should add these to the packaging manually and rebuild.

snap job will package it up as a Snappy Snap package. This is experimental, you can see the output at distribute.kde.org.

Other Jobs

watcher

mgmt jobs iso jobs

The Archive

aptly instance forked? repos? ./ci-tooling/ci/repo_console.rb --gateway ssh://drax:9090

The Packaging

how to package ci variable overlapping files

@@ Line 7: / Line 7: @@
 The code behind build.neon is from [https://github.com/blue-systems/pangea-tooling pangea-tooling] which also runs the code for [http://dci.pangea.pub/ DCI Debian CI], [http://kci.pangea.pub/ KCI Kubuntu CI], [http://aci.pangea.pub/ ACI Appstream CI], and [http://mobile.neon.pangea.pub:8080/ MCI Mobile neon Plasma CI].
-The Jenkins jobs farm off the hard build work to 4 DigitalOcean build servers.
+The Jenkins jobs farm off the hard build work to 4 DigitalOcean build servers.  It runs most jobs inside a Docker container to give a fresh build environment.
 To use the scripts to access Jenkins you will need to set <code>~/.config/pangea-jenkins.json</code> using access key available inside Jenkins to administrators.
+The setup of the machines is maintained in [https://github.com/blue-systems/pangea-kitchen/ pangea-kitchen] which uses Chef to set up the servers with software all configured.
 = The Jobs =
@@ Line 25: / Line 27: @@
 <code>bin</code> job will extract the source, install the build dependencies and compile the package.  It finishes by checking the output from lintian and fails on any errors, you can override errors with lintian-overrides files in the normal .deb packaging method (see dh_lintian).  It also checks for any list-missing files and fails if there are any, override by adding a debian/not-installed files.  It also fails if cmake reports build-dependencies it needs, override with debian/meta cmake-ignore.
-<code>adt</code> job  runs Debian's test framework.  TODO describe this...
+<code>adt</code> job  runs Debian's test framework autopkgtest.  See [http://packaging.ubuntu.com/html/auto-pkg-test.html Ubuntu guide] for some details.  It runs adt-run on the binaries which installs them and runs the relevant test suite as defined in debian/tests/.  It doesn't fail if tests fail.
 <code>pub</code> job will upload to aptly, see The Archive below.
@@ Line 32: / Line 34: @@
 <code>snap</code> job will package it up as a Snappy Snap package.  This is experimental, you can see the output at [http://distribute.kde.org/snappy-testing/ distribute.kde.org].
+== Other Jobs ==
+<code>watcher</code>
+mgmt jobs
+iso jobs
@@ Line 38: / Line 47: @@
 forked?
 repos?
+./ci-tooling/ci/repo_console.rb --gateway ssh://drax:9090
 = The Packaging =