Migration to storage.kde.org

((TBD: revisit this entire thing vis a vis multiple editions))

New directory layout (within the bucket):

/
- /kde-linux_123.raw [raw image]
- /kde-linux_123.torrent [torrent for raw image - http seed]
- /sysupdate/store [NEW desync chunk store]
  - /sysupdate/store/0000/00000ecfbb68ce84a27f55576156e60ea8b9085103c11140f11913ce0dd7dcb3.cacnk [random chunk]
  - /sysupdate/store/...
- /sysupdate/v2/ [sysupdate tree]
  - /sysupdate/v2/kde-linux_123.efi
  - /sysupdate/v2/SHA256SUMS
  - /sysupdate/v2/SHA256SUMS.gpg
- /sysupdate/v3/ [sysupdate tree]
  - /sysupdate/v3/...

The new /sysupdate/store/ acts as chunk store for desync. It contains content addressable chunks of data. This store is shared across all sysupdate versions since it is content addressed anyway and so doesn't need versioning. It's inconsequential if a given chunk originally belonged to v2 or v3 or v15.

upload-vacuum

Go program to clean up old artifacts

gets changed from sftp to s3
iterates /sysupdate/{v2,v3,...} and cleans them up
for each location it generates a SHA256SUMS based on the sums that are stored in the metadata of the s3 objects
iterates / and cleans the images up
NEW it composes an 'upload-tree/' directory that mimics the path setup on storage.kde.org; this eases uploading
NEW the chunk store gets cleaned up based on all chunk indexes available. this is temporary and needs refactoring into its own tool. this requires downloading all indexes first (each about 4 mib) and listing all chunks in the store. mid term should probably only run once a week. cleanup doesn't directly affect anything since only chunks which are not referenced by an index are cleaned

upload.sh

Shell script to manage the upload

runs upload-vacuum
sets up GPG
NEW moves new files into 'upload-tree/'
adds sysupdate files to the relevant SHA256SUMS
NEW runs uploader

uploader

Go program to upload stuff. Super simple.

iterates the 'upload-tree/'
for each file it generates the sha256
puts the file into s3 with sha256 as metadata X-Amz-Meta-X-Kde-Sha256 (to be used by future vacuum runs)

Plan

Edition Branches // Trusted Branches

All artifacts first get published on storage.kde.org in the kde-linux bucket and may be consumed from there by systemd-sysupdate. For the testing edition (>= daily builds) this is the intended way to consume it. Mid-term this may be assisted by a CDN.

Slower editions get releases cut from storage and then pulled down onto download.kde.org for mirroring.

What is not clear is how updates to the slower editions should be handled. Do we direct them to storage or do we mirror? It's a bit tricky because the /store/ must be synced before the SHA256SUMS{,.gpg} or users will get update errors.

What is also not clear is whether we should put the different editions into different buckets. Supposedly putting everything into one bucket opens more opportunity for chunk sharing in the /store/.

Work Branches

For work branches we put uploads into the ci-artifacts bucket. Versioned by their pipeline id. The tree likely matches that of the regular kde-linux bucket. Work builds may be consumed from here by redirecting the download location in the sysupdate files.

What is not clear is how to handle the erofs. Uploading the chunked data is extremely expensive if we can't reuse existing chunks. As such we'd need to upload the complete erofs and somehow deal with the chunk vs. complete file decision client-side.

Speed Measurements

To test performance implications of chunk size and parallelism it'd make sense to throttle connections a bit.

delay run0 tc qdisc add dev lo root netem delay 500ms

bandwidth run0 ./wondershaper -a lo -d 50000 -u 50000

Too lazy to post a daemon just now, but it's a matter of getting an ssl cert and then a couple lines of go to serve the store directory.

Random Observations

chunk size currently is 32K:64K:128K
with http/1.1 single thread performance is absolutely terrible: ~40 minutes download
with 4 thread concurrency it's between 10 and 15 minutes
increasing chunk size to 1M:2M:4M also is between 10 and 15. slightly more towards 10 when using concurrency 4
with http/2 almost every scenario is between 10 and 15 and large chunk size is slightly more towards 10 minutes
without throttling, concurrency 1 performs terribly (18s) while concurrency 4 performs well enough (8s) with the fastest option being 32 concurrency leading to 6s