Phonon

TODO

• API to allow applications to implement their own effects (independent from the backend)
• encoding to files (or any QIODevice)
• Hardware Mixer API
• kded module to handle tasks like:
  • react on plugged new (never seen before) media device
  • manage software volume controls
  • implement policy like muting/lowering volume of selected applications/categories depending on special events (e.g. incoming call)
• Let applications request specific audio capture formats (32 bit, 8 bit, little endian, etc)
• Documentation!
  • Improve the Phonon/Debugging page before it gets put on techbase

Junior Jobs

• none :(

Done as GSoCs

• High level capture API (needs maturing for Phonon proper)
• Low level PCM I/O

GStreamer

• Ensure streamreader works (technically done, needs thread protection for KIO though)
• Implemenation of Phonon changes for spitfire
• Cleanup MO (maybe introduce statemachine? - MO below 2k lines \o/)
  • Split it up into some smaller classes
    • PluginInstaller::requirePlugins(QStringList) - Returns false if the listed plugins do not exist. If the plugin install api is enabled, the plugins are installed.
    • A state machine
    • Maybe some kind of PipelineBuilder (Plumber?) class that does all the heavy lifting of building the pipelines for various sources instead of making the mediaobject be a swiss army knife of pipeline construction, message handling, source loading, and more.
• Support for more capture sources, such as ximagesrc

4.5 release plan

Hopefully, released on Apr 9th, after camp KDE.

• A thorough testing of the improved PuginInstaller on Kubuntu and Fedora, platforms where pgst is slated to be the primary backend
  • Remove mp3 codecs and try to play an mp3 in amarok.
  • Remove mkv codecs and try to play a video with dragon/simpleplayer.
  • Remove gstreamer-plugins-bad and play a DVD. It should prompt for installation of rsndvdbin and dvdspu.
• Fix the stupid lag between mouse movement and the dvdspu updating dvd menus
• Verify that MRLs are handled correctly
• Bug 269197

VLC

• Ensure streamreader works (works with phonon threaded-streaming branch for QIOD, broken for KIO due to KIO not being threadsafe, needs protection)
• Implementation of Phonon changes for spitfire
• try --no-xlib again, was reverted by markey some time back as it caused problems apparently ... it prevents problems with xthread stuff such as we had with PulseAudio crash on exit

Standing Issues

Effects API returning random identifier namers

Amarok finds an Equalizer by looking at the names of the available effects, for xine that would be KEqualizer (which is what Amarok looks for), Gstreamer however uses completely different names...

3 Approaches suggested:

1. Introduce categories - does not resolve original problem, since a consumer still needs a way to differ between devices, introduces additional complexity

1. Fixed Enum - not very dynamic WRT additional magic supported by specific backends. The capabilities would just return enum values and the consumer then issues creation of e.g. Phonon::Effect::Equalizer

1. Standardize on names.

Apps switching on backend switch

This issue is difficult to describe since it has multiple incarnations. The most occurring one is that at backend switch we can only partially destruct the path, leaving it in an inconsistent state from our POV and implicitly depend on the backend to nuke all nodes (only xine does that). In general the whole backend switching business is difficult to support because at every point in time different parts are (or needed) to be responsible for path deconstruction leading to multiple issues.

In an IRC discussion it was concluded that the backend switching is not a terribly important usecase and that we will stop supporting it.

• The KCM's part for switching needs to be moved to an own KCM that is not listed in systemettings
• Switching will only take effect on app restart, the KCM needs to tell the user about this

DVD Menus (how Dragon should not link against xine)

Currently Phonon does not provide the means for a video player to access the menu of its media (think DVD). This lead to a situation where dragonplayer links against xine to provide (primitive abilities of menu switching). A situation that can easily be prevented. The following implementation was considered: have MediaController get an enum of menus, introduce an interface to allow players to query what menus are available (neverminding if the backend or the medium does not support one). The player can then query what menus are there, since there is a fixed amount of supported menus it then can map the supported ones to (e.g.) QActions and upon action issues MC to instruct the backend to switch the menu. Possible menu types: main, title, root, audio, angle, chapter, subtitle.

State mess

States are a mess right now, not very well defined and attached to some very weird signals (like aboutToFinish)... nothing is reliable and it causes crappy implementations (see Amarok EngineController why that is bad). In particular discussion with some of our major API consumers should be conducted to find a solution. Suggested discussion partners: Amarok, Dragon, Bangerang

Graph destruction & Monster nodes

Amarok's enginecontroller has a function called initializePhonon which gets executed when phonon enters error state (in case of missing codec for example). This function deletes all phonon objects held by amarok and reconstructs the whole thing. While this is not a very good practise at all it raises the general concern that Phonon can not administer global graph destruction and instead depends on the backends to handle this (we also see this issue with runtime backend switches). This is a problem as the logic is only implemented in the xine backend and ends up in undefined behaviour land with other backends (mostly resulting in a segfault at some point). Harald suggested the introduction of a Phonon::MonsterNode which then ought to be used as base for the backend's media nodes. Within the ctor of the MonsterNode it can then register with some sort of global node tracker. Upon destruction the MonsterNode unregisters and takes care of path destruction etc. That way it can be ensured globally that the graph is always in a consistent state.

Signals, Threads and Deletes

All signals that get forwarded by Phonon from the backend to the consumer should be queued. This is to "simulate" threading, as signal within the same thread context (i.e. the main app one) would get executed right away ontop of the currently present stack. If a backend emits statechange to error without queuing this will directly end up in a slot call in for example Amarok (note: no stack unwinding happened at this point, in fact it grew), Amarok then does things to it (at the time of writing: reinitialize everything by deleting all Phonon objects), once Amarok is done the stack unwinds to the emission in the backend, and the backend continues excuting statements (if any). At this point the presen stack operates on not allocated memory! To prevent this all signals must be queued, so that a backend can finish whatever it is doing, and only once the stack returned to the mainloop the slot on Amarok's side of things actually gets called. This allows both the backend and the consumer to do whatever they want to do. General note: it should be enforced that backends must not be running in a different thread context or if they do they must ensure that all used contexts are cleaned up properly (i.e. the dtors must only return once all threads are cleaned up).

QIODevice

We should not forget to QA QIODs as those will be used in Qt-only Phonon. It would appear that there are also some problems there. While ongoing kio related fixes should also bring great improvements to the QIOD side of things there are some special things that should be considered. In general it would appear that reading from a KIO slave before it is ready will lock the application as the read will never get data. Similar behaviour is to be expected with QIOD, which should only be read once it emitted its readyRead signal.

Threading and other madness

It should be evaluated where we should/could/need to introduce threading. Some general possibilities:

• All backends at phonon level (or phonon itself for that matter) - this has unique value as any non-threaded operations in Phonon or the backends will lock the application and flood the apps' event loop with Phonon stuff.
• Selective parts of backends
• Abstractstreaming - through the course of the great KIO wars of 2011 it became apparent that kiostreaming support needs a major remake but most of all that threading could make reader implementations much more flexible and doable than having all in the main thread or even mixed with callbacks from a framework thread. Additionally since KIO is not threadsafe putting up an internal design that forces us to protect our own thread would also protect KIO, which is all the more important considering that current or future backend implementations might get calls from a framework thread and direct action taking on those is potentially dangerous.

Testing

Our unit tests are in a sorry state, they should be cleaned up, and we should try to increase coverage. Also clean up the fake backend, maybe.

De-KDEification

Phonon has a vast amount of KDE specific stuff going on that makes it terrible to use Phonon for Qt. This is becoming a major issue as Qt 4.8 will not have a bundled outdated version of Phonon anymore, but instead point interested parties to our tarballs. Out of the top of the writer's head standing issues here are: installation of backends to kde plugin paths && installation of headers to KDE/Phonon only. For source compatibility reasons we need to maintain the KDE header stuff, but at the same time also need to provide convenient ways to keep includes like <Phonon/MediaObject> working. In addition, a nicely packaged SDK should be made available for folks to download which would install itself alongside the Qt SDK. That will keep everyone happy, as consumers wouldn't need to compile Phonon separately before using it on those platforms.

Master Plan

4.6.0 (git master) - dev name: hyperspeed

Estimated release: before KDE SC 4.7

Changes

• AbstractMediaStreams used for all URLs except file: (i.e. KIO streaming)
• Stabilize effects API to resolve random-naming issue (see above)
• Eliminate runtime backend switching (see above - in KDE)
• Add subtitle support
• Mature Capturing API to move out of experimental
• Fix states (see above)
• Possibly MonsterNodes (see above)
• Evaluate threading options (see above - linked to signals)
• QML plugin (Audio at least)
• New debug system (import from Amarok)
• Figure out how to handle translations (install QTranslator? limited to non-plural translations...)

4.5.1 (git 4.5) - dev name: spitfire

Estimated release: Whenever necessary

Changes

• None worth mentioning :P

4.7.0 (not yet in development) - dev name: skylined

Estimated release: fall 2011

Changes

• Investigate usage of automoc and possibly drop it (rather inconvenient dep for Qt-only API consumers) Possibly in the cmake_tng branch

Phonon Five (TODO accumulation stage) - dev name: Hotel

Phonon5 will use Moby Album names for minor and song names for patch releases.

Changes

• Port to GTK+
• Redo AbstractMediaStream
• phononversion split out of phononnamespace

Getting Involved

If you want to get involved with Phonon backend development please subscribe to [email protected]. If you want to contribute, make comments or useful suggestions you can also write to the KDE Multimedia mailinglist. You can join the mailinglist using the web interface or take a look at the archives.

There's also a #phonon channel on freenode where you may find developers.