PIM/Akonadi/VirtualCollections

All about virtual resources, virtual collections and searching. So far mostly a brain dump of ideas and requirements so we can come up with a nice design for all this. It is based on previous discussions about this topic, those are linked in the last section for reference.

Terms

Storage collection: An Akonadi collections that represents a storage location for items.
Virtual collection: An Akonadi collection that does represent an arbitrary set of items stored in any storage collection. Virtual collections are implemented by "linking" to existing items, comparable to symlinks in a file system.
Virtual resource: An Akonadi resource that creates virtual collections instead of storage collections.

Assumptions

Assumptions all the following is based on.

An Akonadi collection can either be a storage collection or a virtual collection, i.e. it can either contain "real" items or links to items stored elsewhere, never both.
An Akonadi resource can either contain storage collections or virtual collections, never both.
We do not supported linked collections for now, due to lack of use-cases.

Use-Cases for Virtual Collections

Representing the results of a persistent search:
- in a per-session scope, e.g. the currently displayed week in KOrganizer
- in a per-application scope, e.g. user-defined persistent search folders in KMail
- in a global scope, e.g. user-defined persistent search folders shared between KMail, Mailody and LionMail.

The Nepomuk tag resource
- Representing the entire tag-tree

Ideas on the Search Infrastructure

Delegation to resources that support search in the back-ends
- Requires query transformation
- Requires management for live searches
- Requires the ability to report search results (needs protocol extension)
Query language still undecided, current options: XESAM, SPARQL
Back-end still undecided, current options: XESAM, Nepomuk

Akonadi Search Infrastructure (concept)

Virtual Resources

A resource that creates virtual collections instead of storage collections.

A virtual resource can exist in two forms:

Physical virtual resources: I.e. there is a conventional agent instance for that resource that operates like any other resource but instead creates and maintains virtual collections.
- These have the same explicit life-time management by akonadi_control as any other resource.
- A Virtual capability could indicate that they are virtual collections.
- RIDs can only be edited by the resource itself, as usual
Virtual virtual resources: These have no agent instance responsible for their collections, but are used for scoping search folders. The Akonadi server itself is responsible for maintaining those, while the applications decide about their lifetime.
- Lifetime can be session or permanent
- Implicitly created and deleted by the Akonadi server whenever persistent searches are created or deleted.
- Resource identifier is provided by the client, and is used to indicate the visibility of the resulting search folder (session internal, app global, desktop wide, etc).
- RIDs can be edited by anyone to simplify identification of search folders for clients
- Not visible for the AgentManager, shouldn't be a problem as the current internal search resource isn't either.

We definitely need better terms for these...

Virtual Collections

The following lists properties of "normal" Akonadi::Collection objects and maps them to the corresponding semantics of virtual collections:

UID: stays the same
RID: for application use (currently it holds the query, but that's a dirty hack) (search folders only, still used for its original purpose by the tag resource)
Name: stays the same, for display purposes only
Resource Id: see virtual resources
Content mime-types: empty
- used for determining whether DnD or copy/paste etc. should be allowed for a given destination collection, however that has to be done differently for virtual destinations anyway.
- will make them implicitly hidden in ETMs with content mime type filters, which is desired
Parent: Can be either Collection::root() or another virtual collection of the same resource, that is we support nested virtual collections.
ACL: extend by link/unlink operations
- Item creation: not allowed, would lack a storage location
- Item modification: in theory possible if the storage collection allows it
- Item deletion: in theory possible if the storage location allows it
- Collection creation: depends on resources, useful for the Nepomuk tag resource, not for persistent search folders
- Collection modification
  - Name: no problem
  - RID: not allowed once there is a actual resource behind it, not a problem when used by the application
  - Query: requires special support in the server, but no principal problem
  - Parent (that is moving): tricky for the tag resource, not really a problem for search folders, moving to non-virtual resources needs to be prevented.
  - Any other attribute: no problem
- Collection deletion: no effects on stored items/collections, so no problem
Query: for search folders only, should be moved into an dedicated attribute.

Note that virtual collections so far are not explicitly marked as such, to keep them as transparent as possible for the clients. However, since the interaction with them differs from that with storage collections, we need ways to distinguish them at least implicitly. All the below for example is covered by the additional ACL properties for linking/unlinking.

Interaction with Virtual Collections

This is about possible user interaction patterns with virtual collections.

Linking/Unlinking of Items

Copy/DnD a "real" item into a virtual collection: That would mean "linking" the item there.
- Useful for the tag resource
- Makes no sense for search folders
Move/Copy/DnD a "link" out of a virtual collection into a storage one: That would mean moving/copying the item to a different storage location, not changing anything regarding the actual linking.
Moving/Copying/DnDing links between two virtual collections
- Makes no sense for search folders, they are read-only for the clients
- useful for the tag resource though

Modification/Deletion of Items

That requires accessing the ACL of the actual storage collection.

Operations on the Virtual Collection

Modifications
- Renaming
  - Only for display purposes anyway for search folders
  - Rename the tag in case of the tag resource?
Moving
- Only for display purposes for search folders
- Change the tag hierarchy?
Deletion
- Delete persistent search
- Delete tag

Usage in ETM

hidden by default due to empty content mime-types field
can be explicitly subscribed by using the virtual resource identifier, so applications can show only the session-local/application-local/type-global virtual collections relevant to them
- that does map nicely to search folders, not to the tag resource though...

Open Questions

RID usage in virtual collections
What about the semantics of Item::parentCollection() in the context of virtual collections, e.g. when used in ItemFetchJob and Monitor?
The virtual resource sections sounds scary...
How should we handle different item types in the tag resource? Applications probably only want one...

Tasks

Concrete tasks extracted from all the chaos above, for implementing virtual collection support. Topologically sorted by dependencies on each other.

Status	Item	Description
TO DO	Extend collection ACLs	Add CanLinkItem/CanUnlinkItem flags
TO DO	Add virtual capability	Resource capability, propagate to Akonadi server and store it there
TO DO	Fix item retrieval	Make use of the knowledge of what is a virtual and what is a storage collection
TO DO	Move query from RID to attribute	Migrate legacy search folders, adapt persistent search creation
TO DO	Virtual virtual collections	Lifetime management, client-side provided identifiers
TO DO	Adapt tag resource	Set the correct ACLs
TO DO	Interactions	Adapt StandardActionManager/PasteHelper/ETM etc. to respect the extended ACLs
TO DO	Content mime-types	Report as empty, adapt ETM filtering if needed
TO DO	ETM	Allow subscription to several arbitrary virtual resources

References

Past discussions about this topic: