KDE Core/Platform 11/FrameworksQA

Warning

This page contains rough working notes from discussion sessions at Platform 11, the contents of which may not accurately reflect any decisions made. Please do not infer anything from these notes, official summaries of the conclusions reached will be made available for discussion as soon as possible.

Frameworks QA

Current situation for kdelibs QA

• Code Review:
  • reviewboard
  • post commit review on the mailing list

• Automated tests
  • 25%-ish of test coverage says gcov, more likely around 12% because of gcov behavior
  • autobuild + automated tests daily for more than a year (on my.cdash.org)
  • the windows guys seems to have also their own dashboard not cdash based
  • those dashboards are some maintainance, but if that's a developer doing it he has the maintainance work for him as well... but on own system

• Binary compatibility? through review, not evaluated

• Criteria for accepting something in the framework:
  • Used by two apps of different modules (but forking to bypass the rule is forbidden), somewhat the old criteria, it is a minimal
  • Now it's more about "common sense"
  • It has to be documented
  • It has to be translated
  • It follows KDE library policy

Current situation for Qt QA

• Code Review:
  • Pastebin (old way)
  • Gerrit so always review (auto-reviewed allowed for trivial cases)

• Automated tests
  • Tests run in reference VMs, VMs can be copied to find out what's going wrong
  • VMs are reset
  • Everything should be tested, but no policy enforcement
  • Some of the Widgets tests are unreliable

• Binary compatibility: evaluated using qt bic tool

• Criteria for accepting something in the framework:
  • Has to have an auto-test
  • Has to be reviewed for coding style, BIC rules
  • Has to pass the auto-tests on all platforms

Aim for our frameworks QA

• Automated tests
  • "continuous" build for the "stamped" frameworks
  • reference VMs? yes but with different build/test setups on top
  • also on developers box with their own setup
  • aim for some level of test coverage? yes for new code, 80% test coverage (if it makes sense, hard for widgets for instance so just having tests is enough)

• Binary compatibility
  • Need to run the check tools
  • Should be part of the autobuild with tests

• Review
  • Auto-test comes with fix in same commit
  • commit log should contain a "Reviewed by" or "REVIEW:"

• Criteria for accepting something in an existing lib:
  • The "two apps rule" should be a minimal
  • The new dependency policy adds to that
  • It should fit in the lib scope
  • It should be written that you commit to maintain that for the rest of your life (around 2 years) or you find someone to take over (which is what we do for apps)

• For new libs,
  • they should meet the license policy
  • they should meet the dependency criteria of their "tier box"
  • they should have a scope, no dumping ground
  • they should meet all the quality criteria before being "stamped"

Plan to get there

• Automated tests
  • talk to the OBS guys?
  • ask to Dirk to get his dashboard submit results
  • get Volker's setup + recommended one by Alex Neundorf, and base first VMs on that
  • setup for autobuilds? we should aim for crowdsourcing on that one, it's better if that comes from our developers as we can help them by discussing if their autobuild gets stuck
  • for the VM the settings should be random: generate a kdesrc-buildrc file distributed by network to the VM asking for it

• VM creation
  • Find the web site creating the VM image
  • Load the VM images with a minimal distro
  • At boot time install packages
  • Downloads the kdesrc-buildrc
  • Then run builds continuously until shutdown including tests using "Volker's script"
  • Work started in git.kde.org:scratch/vkrause/crowdci

• Binary compatibility
  • Ask Lubos about his tool
  • Or run a test based on "gcc ... -fdump-class-hierarchy" like tests/auto/bic in Qt

• Update the library policy with the new constraints coming from our aim on that page, and the whole dependency constraints