Neon/InstallableImages: Difference between revisions

From KDE Community Wiki
No edit summary
 
(13 intermediate revisions by one other user not shown)
Line 1: Line 1:
= KDE neon live/installable images =
= KDE neon live/installable images =


*Images are at [http://images.neon.kde.org images.neon.kde.org] on drax  (stores the most recent ones and all from today).
*Images are at [http://files.kde.org/neon files.kde.org] on depot (stores the most recent ones and all from today).
*Old images are at [http://old-images.neon.kde.org.uk/ old-images.neon.kde.org.uk] (stores the most recent 5).


Our images are built with build.neon.kde.org using the scripts in pangea-tooling nci/imager
Our images are built with build.neon.kde.org using the scripts in pangea-tooling nci/imager
imager.rb starts a new containment and runs build.sh and copies the result to the web site dir for images.neon.kde.org
imager.rb starts a new containment and runs build.sh and copies the result depot and images.neon.kde.org


build.sh is run in the containment and installs the packages needed, sets some variables and runs ubuntu-defaults-image
build.sh is run in the containment and installs the packages needed, sets some variables and runs ubuntu-defaults-image
Line 14: Line 13:


live-build uses livecd-rootfs, Ubuntu hooks (/usr/share/livecd-rootfs/live-build/ubuntu-core/) to build the squashfs live filesystem
live-build uses livecd-rootfs, Ubuntu hooks (/usr/share/livecd-rootfs/live-build/ubuntu-core/) to build the squashfs live filesystem
== Seeds ==
Our meta packages and ISOs are made from Seeds
http://weegie.edinburghlinux.co.uk/~jr/neon-seeds/seeds/structure.png
http://weegie.edinburghlinux.co.uk/~jr/neon-seeds/seeds/


== Bits we need to fork and customise ==
== Bits we need to fork and customise ==


*ubuntu-defaults-image is adapted to read our hooks and ignore the kubuntu-desktop meta package, this can probably be done away with
*ubuntu-defaults-image is adapted to read our hooks and ignore the kubuntu-desktop meta package, this can probably be done away with
*config-hooks/20_package_list.sh adds and removes some packages compared to kubuntu meta packages and tasks, this can probably be done away with
*https://invent.kde.org/neon/infrastructure/pangea-data/imager/ contains the neon specific customisations for the neon iso's
*config-hooks/apt.conf.sh adds apt debugging and no language pack settings
  - config-hooks/apt.conf.sh adds apt debugging and no language pack settings
*config-hooks/repo.sh adds neon repo
  - config-hooks/repo.sh adds neon repo
*https://code.launchpad.net/~kdeneon/livecd-rootfs-neon/trunk is a branch of lp:livecd-rootfs/trunk it adds code to use our meta packages when PROJECT is set to neon
*https://invent.kde.org/neon/neon/livecd-rootfs/ is a fork of lp:livecd-rootfs/jammy it adds code to use our meta packages when PROJECT is set to neon.  Using jammy as an example updating the fork is done by:
  - clone [email protected]:neon/neon/livecd-rootfs.git
  - cd livecd-rootfs && git switch Neon/release
  - git remote add lp https://git.launchpad.net/livecd-rootfs
  - git fetch lp
  - git merge lp/ubuntu/jammy
  - then check in debian/changelog  or do a git diff origin/Neon/release  to see what's changed and if it's at the right version compared to the jammy update at https://launchpad.net/ubuntu/+source/livecd-rootfs
  - if all good -- git push
 
*neon:neon/seeds makes the neon-desktop and neon-live metapackages.  Kubuntu uses a kubuntu-live apt task but our archive doesn’t support tasks so we use metapackages instead.
*neon:neon/seeds makes the neon-desktop and neon-live metapackages.  Kubuntu uses a kubuntu-live apt task but our archive doesn’t support tasks so we use metapackages instead.
*https://launchpad.net/ubiquity-neon has our branch of ubiquity (branched from wily tag) with a few changes to remove branding
*https://invent.kde.org/neon/neon/calamares-settings.git contains neon's settings for calamares which has replaced ubiquity as intaller.
*[http://packaging.neon.kde.org/cgit/neon/base-files.git/ base-files] sets important files including lsb-release
*https://invent.kde.org/neon/forks/base-files sets important files including lsb-release
*plymouth theme is now from Plasma breeze-plymouth
*plymouth theme is now from Plasma breeze-plymouth
*grub theme is now from Plasma breeze-grub
*grub theme is now from Plasma breeze-grub
*TODO: get neon-live installed, syslinux theme, set ubuntu archive to a mirror on live cd and installed image, set grub entry to be neon, make source images, sign images, make zsync work, archive old images, make working directory different for different jobs
*syslinux theme is our own theme, TODO document how this works
*TODO: sign images
 
= UEFI Support =


= UEFI Support: =
live-build does not support UEFI.  There's a few patches and addons to do so, see [https://todo.kde.org/?controller=task&action=show&task_id=1611 todo item] for links.


live-build (at least the version we use) does not support UEFI so our images do not support UEFI
The live-build UEFI bug has patches to add it natively.  Netrunner just keeps it from Kubuntu as part of the remaster.  Tanglu adds a efi script to live-build which takes it from debian-installer.


The latest Debian Live images have no UEFI in them so it hasn't been added there (and live-build is largely unmaintained).
Ubuntu images get their livefs built on Launchpad (using a not very well documented and somewhat incomplete coded part of Launchpad which I think it only useable by this team for this purpose) https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/xenial/kubuntu/ which are seem to be built with livefs-rootfs (says launchpad code ./lib/lp/soyuz/templates/livefs-new.pt).


Development versions of Tanglu include UEFI support, investigate how it does this.
Then the ISO images are made by some magic proprietary script which takes the live image and adds its own stuff including UEFI support.
http://people.canonical.com/~ubuntu-archive/cd-build-logs/kubuntu/xenial/


Netrunner Kubuntu version does include UEFI support, it’s just a remaster of Kubuntu so I think copies Kubuntu’s stuff verbatim
== KDE neon UEFI implementation June 2016 ==
https://github.com/NeptuneOS/remaster-kit/blob/master/usr/share/remaster-kit/functions#L200
https://github.com/netrunner/build-scripts


Ubuntu images get their livefs built on Launchpad (using a not very well documented and somewhat incomplete coded part of Launchpad which I think it only useable by this team for this purpose) https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/xenial/kubuntu/ which are seem to be built with livefs-rootfs (says launchpad code ./lib/lp/soyuz/templates/livefs-new.pt).
ISO booting by UEFI support is done by copying the grub-efi files from a Kubuntu ISO, [http://packaging.neon.kde.org/cgit/forks/live-build.git/tree/share/bootloaders/grub-efi putting them in live-build] and in the [http://packaging.neon.kde.org/cgit/forks/live-build.git/tree/scripts/build/lb_binary_efi lb efi script] copying it into the binary/ directory and extracting the efi images (with mcopy) to add the right files on the CD.  The breeze-grub theme is added and boot name changed to Neon in grub.cfg.
 
For UEFI support on installed systems Ubiquity needs a local apt archive on the CD image which contains grub-efi, we again grab a copy of this from Kubuntu images. It also needs a file called .disk/cd_type.


Then the ISO images are made by some magic proprietary script which takes the live image and adds its own stuff including UEFI supportThis is probably a hidden fork of live-build.
Ubiquity installs the EFI files into /boot/EFI/efi/neon on the installed system but the path "efi/ubuntu" is hardcoded in two placesThe fwup efi binary which is used for firmware updates installs to efi/ubuntu and the Grub efi binary is hardcoded to look in efi/ubuntu for grub.cfg so a [http://bazaar.launchpad.net/~jr/ubiquity-neon/trunk/revision/6383 patch in ubiquity] copies our grub.cfg into efi/ubuntu (this patch need to be applied manually in the archive as the package is native not quilt).  This has the downside that any changes in the ubuntu grub.cfg will get removed, similarly a later Ubuntu install will wipe the neon grub.cfg but as the code to set up grub.cfg is the same for both having a custom config which differs is a very niche setup.
http://people.canonical.com/~ubuntu-archive/cd-build-logs/kubuntu/xenial/


== SecureBoot Support ==
== SecureBoot Support ==


Goodness knows, needs stuff signed by a key approved by microsoft
The shim, grub and linux binaries are all copied directly from Kubuntu and signed with that so this just works.  If we want to make changes such as not hard coding efi/ubuntu in the grub binary we'd need to build it ourselves, sign up to Microsoft's approval scheme.  This is left as future work.
 
How this [https://www.fsf.org/campaigns/secure-boot-vs-restricted-boot/whitepaper-web works with GPL 3 terms] is another mystery for future work.
 
= Pinebook Images =
 
We're also building images for the [https://www.pine64.org/?page_id=3707 Pinebook] cheap ARM-based laptop.
 
As of Decemeber 2017 we have built all of dev-unstable packages for ARM64 architecture.
 
<code>[https://build.neon.kde.org/job/img_neon_xenial_devedition-gitunstable_arm64/ img_neon_xenial_devedition-gitunstable_arm64]</code> build job will produce a .img disk image which can be dd'ed to a microSD card and booted up on Pinebook.
 
Code to create image is in [https://github.com/blue-systems/pangea-tooling/tree/master/nci/imager-img pangea-tooling].
 
Unlike the ISO images which use live-build from Ubuntu this uses live-build from Debian.  The images get made in Docker on master (Drax) amd64 server using QEMU to emulate arm, this is slow but necessary because we use process namespacing on our arm64 Docker setup for security.
 
Similarly we use live-config from Debian rather than Casper from Ubuntu as this works in normal user-space while Casper needs faff with initramdisks.
 
There are 3 custom parts:
 
* DRM driver in the Kernel https://github.com/ayufan-pine64/linux-pine64
* Proprietary GLES implementation ( libMali.so ) https://github.com/shadeslayer/pine64-mali-x11
** EULA http://files.pine64.org/doc/MALI/MALI%20EULA.pdf
** More info http://wiki.pine64.org/index.php/Mali_Driver
* X11 driver https://github.com/shadeslayer/xf86-video-armsoc
We install Linux and Mali from Netrunner repos and build armsoc driver ourselves.

Latest revision as of 11:17, 22 January 2024

KDE neon live/installable images

  • Images are at files.kde.org on depot (stores the most recent ones and all from today).

Our images are built with build.neon.kde.org using the scripts in pangea-tooling nci/imager imager.rb starts a new containment and runs build.sh and copies the result depot and images.neon.kde.org

build.sh is run in the containment and installs the packages needed, sets some variables and runs ubuntu-defaults-image

ubuntu-defaults-image is a script copied from ubuntu-defaults-builder, an Ubuntu script intended to make locale specific ubuntu derived images (we ignore the template for meta package and locale stuff).

It uses Ubuntu’s fork of Debian’s live-build, it’s quite an old fork (3.0~ from August 2012) with lots of Ubuntu specific patches

live-build uses livecd-rootfs, Ubuntu hooks (/usr/share/livecd-rootfs/live-build/ubuntu-core/) to build the squashfs live filesystem

Seeds

Our meta packages and ISOs are made from Seeds

structure.png http://weegie.edinburghlinux.co.uk/~jr/neon-seeds/seeds/

Bits we need to fork and customise

 - config-hooks/apt.conf.sh adds apt debugging and no language pack settings
 - config-hooks/repo.sh adds neon repo
 - clone [email protected]:neon/neon/livecd-rootfs.git
 - cd livecd-rootfs && git switch Neon/release
 - git remote add lp https://git.launchpad.net/livecd-rootfs
 - git fetch lp
 - git merge lp/ubuntu/jammy
 - then check in debian/changelog  or do a git diff origin/Neon/release  to see what's changed and if it's at the right version compared to the jammy update at https://launchpad.net/ubuntu/+source/livecd-rootfs
 - if all good -- git push
  • neon:neon/seeds makes the neon-desktop and neon-live metapackages. Kubuntu uses a kubuntu-live apt task but our archive doesn’t support tasks so we use metapackages instead.
  • https://invent.kde.org/neon/neon/calamares-settings.git contains neon's settings for calamares which has replaced ubiquity as intaller.
  • https://invent.kde.org/neon/forks/base-files sets important files including lsb-release
  • plymouth theme is now from Plasma breeze-plymouth
  • grub theme is now from Plasma breeze-grub
  • syslinux theme is our own theme, TODO document how this works
  • TODO: sign images

UEFI Support

live-build does not support UEFI. There's a few patches and addons to do so, see todo item for links.

The live-build UEFI bug has patches to add it natively. Netrunner just keeps it from Kubuntu as part of the remaster. Tanglu adds a efi script to live-build which takes it from debian-installer.

Ubuntu images get their livefs built on Launchpad (using a not very well documented and somewhat incomplete coded part of Launchpad which I think it only useable by this team for this purpose) https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/xenial/kubuntu/ which are seem to be built with livefs-rootfs (says launchpad code ./lib/lp/soyuz/templates/livefs-new.pt).

Then the ISO images are made by some magic proprietary script which takes the live image and adds its own stuff including UEFI support. http://people.canonical.com/~ubuntu-archive/cd-build-logs/kubuntu/xenial/

KDE neon UEFI implementation June 2016

ISO booting by UEFI support is done by copying the grub-efi files from a Kubuntu ISO, putting them in live-build and in the lb efi script copying it into the binary/ directory and extracting the efi images (with mcopy) to add the right files on the CD. The breeze-grub theme is added and boot name changed to Neon in grub.cfg.

For UEFI support on installed systems Ubiquity needs a local apt archive on the CD image which contains grub-efi, we again grab a copy of this from Kubuntu images. It also needs a file called .disk/cd_type.

Ubiquity installs the EFI files into /boot/EFI/efi/neon on the installed system but the path "efi/ubuntu" is hardcoded in two places. The fwup efi binary which is used for firmware updates installs to efi/ubuntu and the Grub efi binary is hardcoded to look in efi/ubuntu for grub.cfg so a patch in ubiquity copies our grub.cfg into efi/ubuntu (this patch need to be applied manually in the archive as the package is native not quilt). This has the downside that any changes in the ubuntu grub.cfg will get removed, similarly a later Ubuntu install will wipe the neon grub.cfg but as the code to set up grub.cfg is the same for both having a custom config which differs is a very niche setup.

SecureBoot Support

The shim, grub and linux binaries are all copied directly from Kubuntu and signed with that so this just works. If we want to make changes such as not hard coding efi/ubuntu in the grub binary we'd need to build it ourselves, sign up to Microsoft's approval scheme. This is left as future work.

How this works with GPL 3 terms is another mystery for future work.

Pinebook Images

We're also building images for the Pinebook cheap ARM-based laptop.

As of Decemeber 2017 we have built all of dev-unstable packages for ARM64 architecture.

img_neon_xenial_devedition-gitunstable_arm64 build job will produce a .img disk image which can be dd'ed to a microSD card and booted up on Pinebook.

Code to create image is in pangea-tooling.

Unlike the ISO images which use live-build from Ubuntu this uses live-build from Debian. The images get made in Docker on master (Drax) amd64 server using QEMU to emulate arm, this is slow but necessary because we use process namespacing on our arm64 Docker setup for security.

Similarly we use live-config from Debian rather than Casper from Ubuntu as this works in normal user-space while Casper needs faff with initramdisks.

There are 3 custom parts:

We install Linux and Mali from Netrunner repos and build armsoc driver ourselves.