Neon/Appstream

Generation

Appstream data is generated through a bunch of steps

• package gets built and published to repository
• every so often mgmt_appstream-generator_* jobs run
  • they pull in neon-appstream-extra-metainfo
  • run appstream-generator (aka asgen)
  • asgen downloads all packages published in the repo at archive.neon.kde.org
  • asgen aggregates all appstream data
  • asgen generates dep11 data for publishing (e.g. https://archive.neon.kde.org/user/dists/bionic/main/dep11/)
  • additional tooling pushes the asgen data onto the aptly server (iff appstream actually found changes)
• every so often aptly publishes a repository (for user editions only at snapshots, otherwise for every build)
  • aptly finds the data pushed onto the server by asgen and includes it in the repo

Simply put: appstream data is always at least one aptly publishing step behind (possibly more if asgen finds no delta to pick up)!

Duplicated Components

https://build.neon.kde.org/view/mgmt/job/mgmt_appstream-components_bionic_user

Our duplicated component QA tech checks that there aren't likely overlaps between "our" components. Because we stack our appstream data on top of Ubuntu's there is a good chance that KDE software changes the component id between what is in Ubuntu and what is in neon. That's where the automated QA comes in. It checks that likely variants of a name do not appear at the same time.

Each component has an "active" ID and permutations thereof. The active ID is the one we have in our dep11 data of archive.neon.kde.org. Permutations are simply likely alternate IDs this may be available as.

When a possible duplicate is found something like this is printed in the QA job:

org.kde.kopete also has permutation: kopete.desktop

this means that the "active" ID org.kde.kopete is conceivably the same as the also available ID kopete.desktop.

To determine what to do about this we'll have to look at where the components come from. One option is to look at the raw xml:

With the commands appstreamcli dump org.kde.kopete and appstreamcli dump kopete.desktop

Looking at the raw XML you might notice that one component uses neon domains and the other ubuntu domains.

Alternatively one can grep the underlying data from dep11

zgrep -i 'ID:' /var/lib/app-info/yaml/* |grep kopete

/var/lib/app-info/yaml/archive.neon.kde.org_user_dists_bionic_main_dep11_Components-amd64.yml.gz:ID: org.kde.kopete
/var/lib/app-info/yaml/archive.ubuntu.com_ubuntu_dists_bionic_universe_dep11_Components-amd64.yml.gz:ID: kopete.desktop

When you've made sure that the permutation is simply coming from ubuntu the solution is easy: mark the component for removal by our appstream-generator. You'll want to add it to the removed-components.json of https://invent.kde.org/sysadmin/neon-appstream-extra-metainfo and note that the old name is from ubuntu in the removed-components.md. Also note the README.md in that repo.

 

Remember
The neon component ID must be the active ID, if it is not there's another bug that needs looking into!

If it is not clear why the components appear as duplicates, or where one of the components actually comes from, the problem will need additional investigation by someone who understands appstream.

Note

Because generating data is always a step behind for changes to apply you need to run the relevant mgmt_appstream-generator_* jobs and then also trigger publishing (see #Generation). You could also wait a couple of days