Neon/Locally build packages: Difference between revisions

From KDE Community Wiki
(Added known false negatives and tidied up existing content to be more clear.)
(Improve ccache documentation for pbuilder.)
 
(9 intermediate revisions by the same user not shown)
Line 4: Line 4:


* A pre-existing KDE [https://community.kde.org/Get_Involved/development development environment].
* A pre-existing KDE [https://community.kde.org/Get_Involved/development development environment].
* <code>dpkg-dev</code> package will include the standard debian toolchain for building packages.
* Some knowledge of the content in the [https://www.debian.org/doc/manuals/maint-guide/ Debian New Maintainers' Guide].
* <code>dpkg-dev</code> and <code>xbuilder</code> packages will include the standard Debian toolchain for building packages.


= Setting up local files for building package =
= Setting up local files for building package =
Line 12: Line 13:
# Remove symbols to simulate CI removing the symbol files, <code>rm debian/*.symbols</code>
# Remove symbols to simulate CI removing the symbol files, <code>rm debian/*.symbols</code>
# Change debian/source/format to use native instead of quilt, <code>sed -i 's/quilt/native/' debian/source/format</code>
# Change debian/source/format to use native instead of quilt, <code>sed -i 's/quilt/native/' debian/source/format</code>
# Apply patches, <code>QUILT_PATCHES=debian/patches quilt push -a</code>


Or in one command:
Or, in one command:
<code>mkdir -p usr/share/locale && touch usr/share/locale/stub && rm -f debian/*.symbols && sed -i 's/quilt/native/' debian/source/format</code>
 
<code>mkdir -p usr/share/locale && touch usr/share/locale/stub && rm -f debian/*.symbols && sed -i 's/quilt/native/' debian/source/format && QUILT_PATCHES=debian/patches quilt push -a</code>


= Building the package =
= Building the package =
Line 26: Line 29:
* Complaints about dsc and missing key.
* Complaints about dsc and missing key.
* Lintian returns <code>malformed-debian-changelog-version</code>.
* Lintian returns <code>malformed-debian-changelog-version</code>.
= Speeding up compilation =
A compiler cache can help speed things up immensely, particularly when it comes to packages that take a while to compile.
Install <code>ccache</code>. It might be worth also setting these options:
* <code>ccache -o max_size="20G"</code> - Increases the cache size to 20G, roughly kdesrc-build uses about 12G currently and the default is 10G.
* <code>ccache -o compression="true"</code> - Enable compression, reduces cache size by compressing objects.
* <code>ccache -o compression_level="9"</code> - Highest level of compression, helps utilize your space significantly better at a relatively minimal performance cost.
Configure <code>~/.devscripts</code> to use <code>ccache</code> when using <code>debuild</code>:
<code>echo 'DEBUILD_PREPEND_PATH="/usr/lib/ccache"' | tee -a ~/.devscripts</code>
Now you should be able to use <code>debuild</code> with ccache. You may want to validate this by checking its' usage statistics while building, use <code>ccache -s</code>.
= Using pbuilder =
pbuilder is a useful tool when you want to build in a clean environment, and to validate that required development packages are correctly configured in the <code>debian/control</code> file.
pbuilder requires the definition of the Neon repositories when creating the base image:
<code>sudo pbuilder create --distribution focal --othermirror "deb http://archive.neon.kde.org/user focal main|deb-src http://archive.neon.kde.org/user focal main" --keyring /usr/share/keyrings/neon-archive-keyring.gpg</code>
== Using ccache ==
Create a ccache path, such as:
<code>sudo mkdir -p /cache/pbuilder</code>
Now configure pbuilder to use ccache, with similar settings we used above
<code>echo 'CCACHEDIR=/cache/pbuilder
EXTRAPACKAGES=ccache
export CCACHE_MAXSIZE="20G"
export CCACHE_COMPRESS"="true"
export CCACHE_COMPRESSLEVEL="9"' | sudo tee -a /etc/pbuilderrc</code>
=== Deduplicating ccache caches ===
Very likely, you will want to share the same ccache data as to not duplicate effort, to do this will make one central location for cache files and produce mount binds for each user.
Create our generic ccache path:
<code>sudo mkdir -p /cache/ccache && sudo chown 65534:65534 /cache/ccache</code>
We will be storing all files under this as nobody:nogroup.
Create our user ccache path:
<code>sudo mkdir -p /cache/user && sudo chmod 0000 /cache/user</code>
Create our pbuilder ccache path:
<code>sudo mkdir -p /cache/pbuilder && sudo chmod 0000 /cache/pbuilder</code>
Add bindfs mount points (make sure <code>bindfs</code> is installed):
<code>echo 'bindfs#/cache/ccache /cache/user fuse force-user=1000,force-group=1000,perms=u+rwX,create-for-user=65534,create-for-group=65534 0 0
bindfs#/cache/ccache /cache/pbuilder fuse force-user=1234,force-group=1234,perms=u+rwX,create-for-user=65534,create-for-group=65534 0 0' |sudo tee -a /etc/fstab</code>
mount the new binds, <code>sudo mount -a</code>
Set your new user cache directory:
<code>ccache -o cache_dir=/cache/user</code>

Latest revision as of 11:56, 21 December 2020

This page describes some basics in taking a package from https://invent.kde.org/neon/ and building it locally. This can be useful for doing local testing before committing changes.

Prerequisites

Setting up local files for building package

  1. Clone the package repository repository, i.e. https://invent.kde.org/neon/kde/libkdegames
  2. Copy the source tree from your kde sources to where the debian/ path resides.
  3. Create a usr/share/locale to simulate CI locale injection, mkdir -p usr/share/locale && touch usr/share/locale/stub
  4. Remove symbols to simulate CI removing the symbol files, rm debian/*.symbols
  5. Change debian/source/format to use native instead of quilt, sed -i 's/quilt/native/' debian/source/format
  6. Apply patches, QUILT_PATCHES=debian/patches quilt push -a

Or, in one command:

mkdir -p usr/share/locale && touch usr/share/locale/stub && rm -f debian/*.symbols && sed -i 's/quilt/native/' debian/source/format && QUILT_PATCHES=debian/patches quilt push -a

Building the package

Use your favorite tool:

  • dpkg-buildpackage
  • debuild

Known false negatives

  • Complaints about dsc and missing key.
  • Lintian returns malformed-debian-changelog-version.

Speeding up compilation

A compiler cache can help speed things up immensely, particularly when it comes to packages that take a while to compile.

Install ccache. It might be worth also setting these options:

  • ccache -o max_size="20G" - Increases the cache size to 20G, roughly kdesrc-build uses about 12G currently and the default is 10G.
  • ccache -o compression="true" - Enable compression, reduces cache size by compressing objects.
  • ccache -o compression_level="9" - Highest level of compression, helps utilize your space significantly better at a relatively minimal performance cost.

Configure ~/.devscripts to use ccache when using debuild:

echo 'DEBUILD_PREPEND_PATH="/usr/lib/ccache"' | tee -a ~/.devscripts

Now you should be able to use debuild with ccache. You may want to validate this by checking its' usage statistics while building, use ccache -s.

Using pbuilder

pbuilder is a useful tool when you want to build in a clean environment, and to validate that required development packages are correctly configured in the debian/control file.

pbuilder requires the definition of the Neon repositories when creating the base image:

sudo pbuilder create --distribution focal --othermirror "deb http://archive.neon.kde.org/user focal main|deb-src http://archive.neon.kde.org/user focal main" --keyring /usr/share/keyrings/neon-archive-keyring.gpg

Using ccache

Create a ccache path, such as: sudo mkdir -p /cache/pbuilder

Now configure pbuilder to use ccache, with similar settings we used above

echo 'CCACHEDIR=/cache/pbuilder EXTRAPACKAGES=ccache export CCACHE_MAXSIZE="20G" export CCACHE_COMPRESS"="true" export CCACHE_COMPRESSLEVEL="9"' | sudo tee -a /etc/pbuilderrc

Deduplicating ccache caches

Very likely, you will want to share the same ccache data as to not duplicate effort, to do this will make one central location for cache files and produce mount binds for each user.

Create our generic ccache path: sudo mkdir -p /cache/ccache && sudo chown 65534:65534 /cache/ccache

We will be storing all files under this as nobody:nogroup.

Create our user ccache path: sudo mkdir -p /cache/user && sudo chmod 0000 /cache/user

Create our pbuilder ccache path: sudo mkdir -p /cache/pbuilder && sudo chmod 0000 /cache/pbuilder

Add bindfs mount points (make sure bindfs is installed):

echo 'bindfs#/cache/ccache /cache/user fuse force-user=1000,force-group=1000,perms=u+rwX,create-for-user=65534,create-for-group=65534 0 0 bindfs#/cache/ccache /cache/pbuilder fuse force-user=1234,force-group=1234,perms=u+rwX,create-for-user=65534,create-for-group=65534 0 0' |sudo tee -a /etc/fstab

mount the new binds, sudo mount -a

Set your new user cache directory:

ccache -o cache_dir=/cache/user