KDE PIM/Meetings/Osnabrueck 4/Proposal Daemon

Tobias's PIM Daemon Proposal

The Proposal

       Abstract for a KDE PIM Daemon
==========================================

During the KDE 3.X development cycle the KDE PIM developer were faced with
new challenges concerning the dimension of data to handle.

When there was just a small address book or calendar file on the users desktop
PC in the past, we'll have to handle LDAP directories with more than 100000
entries and several groupware servers with even more data records in the future.

With libkabc and libkcal, kdepim already provides the tools for managing these amount
of data but there are still two bottlenecks which have to be solved for KDE 4.0:

1) Synchronous access to the data
    - the contacts/events are loaded synchronous from the groupware server
      which blocks the graphical user interface

2) Per process storage of the data
    - every applications which wants to access contacts/events has to load
      _all_ contacts/events into its own address space, which causes a big memory
      footprint and is often unnecessary

Since a few years component based software becomes more and more popular and KDE
should/does profit from it as well. So in the near future the two issues will
become more important, because every pim component would suffer of them.

A possible solution would be a KDE PIM Daemon, the design and API is the topic of this RFC.

Why do we need a PIM daemon?
-----------------------------
  - loading data synchronous blocks the GUI -> asynchronous loading
    -> with asynchronous loading the data aren't available immediately
    -> user has to wait for data
    -> that's bad

  - all data get loaded into the applications address space
    -> no sharing available
    -> 3 applications have loaded the whole address book 3 times
    -> that's bad as well

  - no central change notification
    -> when one application adds a new calendar, another application won't
       get notified about it and can't update its display
    -> not acceptable


What should the PIM daemon look like?
--------------------------------------
The pim daemon (KPimD) is a standalone application or kded module which offers its service
via a DCOP interface. The daemon can either be started on KDE startup or when the
first pim application queries its service.

KPimD handels the address books, calendars and notes of the user, maybe also mails in a later
version. To load/store the data it uses the K Resource Framework which will be rewritten
for KDE 4.0. The application developer will have a thin wrapper library around the dcop service,
but he will be able to use the dcop service directly as well. These libraries will have
a similar API to kabc/kcal.

A new part of KDE PIM 4.0 will be components which utilize common pim tasks (e.g. search for an contact/event)
and make heavy use of the KPimD dcop interface.

So the whole structure will look like this:


    ---------------       ---------------
   < PIM Component >     (  Application  )
    ---------------       ---------------
                \         /
    -------------------------------------
   |       Thin Wrapper Library          |
    -------------------------------------
                     |
                     |
    -------------------------------------
   |          DCOP Interface             |
    .....................................
   |          KDE PIM Daemon             |
    -------------------------------------
                     |
    -------------------------------------
   |       K  Resource Framework         |
    -------------------------------------


How does the PIM daemon solve our problems?
--------------------------------------------
Our main problems where the synchronous/asynchronous loading, the
memory footprint and the missing notification system in current kde pim
libraries.

The first issue is solved by the daemon, because it is a separated process
so it can load all the data and the pim application doesn't have to wait.
Furthermore caching can be implemented in a single place, so also when the daemon
has just started, some (maybe not up-to-data) data are already available.

The second problem is solved by the daemon as well since it will offer a query
interface.
Today KOrganizer loads all events/todos even when it presents only the events of
the current week to the user. KAddressBook also loads all contacts even when the
user just want to see the contacts which starts with an 'A' in the given name.
With the query interface the pim applications can request exactly the data they
are really interested in, so the memory footprint of the single pim applications
stays small.

Since the daemon is a separated and unique process which manages all pim data handling
it knows everything about changes => it can send notifications for all pim related actions.
This feature allows us to develop model-view-controller based components which guarantees
up-to-date views of contacts and calendars all over the desktop.


What shall the APIs look like?
-------------------------------
During KDE 3.X the address book and calendar libraries were designed as a kind
of container classes for contacts and events. The idea behind it was to provide
tools for pim development. The concept was nice but not what's needed in a desktop
environment.
There you need services which can be used by 3rd-party developer and, also it seems to
be the opposite of general development practise, it shouldn't be to general but give
a straight forward idea of how to get stuff done.

For this reason the new APIs shouldn't allow the developer to create new containers
(address books or calendars) as C++ objects and work on them, but rather create new
address books or calendars _in_ the service and work with them by using unique
identifiers.

At the first glance it looks like a big limitation, but it isn't because you can do all
things needed for a desktop environment in this way as well and you get all the above
mentioned features for free.

And what do the APIs look like in detail?

K Resource Framework
......................

The first change has to be done in the K Resource Framework. Here we have to separated
the basic resource framework (located in kdelibs/kresources) from the pim specific framework
(located in kdelibs/kabc and kdepim/libkcal).

The basic framework needs support for
  - subresources
  - asynchronous data handling

The pim framework needs support for
  - categories
  - distribution lists (only KABC::Resource I guess)

So pim resources will be able to load/store categories and distribution lists from/to groupware
servers as well.

KDE PIM Daemon
...............

The pim daemon will provide a dcop interface for creating, changing and deleting address books, calendars,
notebooks and their items, here a short abstract of functionality:

  - AddressBookInterface
    - create address books
    - delete address books
    - lock address books (needed for synchronization)
    - activate/deactivate address books

    - add contact to address books
    - remove contact from address books
    - change contact in address books
    - query contacts

    - add categories to address books
    - remove categories from address books

    - create distribution lists
    - change distribution lists
    - remove distribution lists

    - add custom fields
    - remove custom fields

  - CalendarInterface
    - create calendar
    - delete calendar
    - lock calendar (needed for synchronization)
    - activate/deactivate calendar

    - add event/todo to calendar
    - remove event/todo from calendar
    - change event/todo in calendar
    - query events/todos

    - add categories to calendar
    - remove categories from calendar

    - add custom fields
    - remove custom fields

  - NotesInterface
    - add notebook
    - remove notebook
    - lock notebook (needed for synchronization)
    - activate/deactivate notebook

    - add item to notebook
    - change item in notebook
    - remove item from notebook
    - query items

    - add categories to notebook
    - remove categories from notebook

    - add custom fields
    - remove custom fields

  - Signals
    .... (come later)

The arguments and return values of this methods are primitive types
like booleans, integers, strings and stringlists, so 3rd-party developer
can easily write applications by using the dcop bindings for their preferred
language (even with shell scripting). The data (contacts, events/todos, notes)
are passed as strings in vCard3.0 resp. iCal format.

An example address book interface is attached.

What's new in these interfaces?

The locking of address books, calendars and notebooks is necessary for synchronization, because you need
a static set of data records during the synchronization, otherwise it leads to massive data loose.

The categories and distribution lists are associated to an address book, calendar or notebook, so it will
be possible to store them on groupware servers.

The custom fields are handled in this layer (not in the wrapper libraries) to make them available in
the search queries. That's an often asked feature and of course it makes no sense to provide custom fields
when you can't use them (like it's atm with libkabc/libkcal).

The notebook interface should be the backend for knotes, so it's easier to add support for groupware servers
because the caching and asynchronous data handling is for free, furthermore the API allows a clean way for
synchronization.

There will be dcop signals for every action you can do with the API, so whenever a new address book, calendar
or notebook is created/changed/deleted a signal will be emitted which can be used by model-view-controller
based components.
Errors will be delivered by dcop signals as well.

Are there any limitations?
---------------------------
Of course there are limitations with the new concept, but they are nothing which can't be solved
by doing it in another clean way.

Examples:

How to add a new address book from within the groupwarewizard?

  The old way was to load the resource manager of the address books family
  in the groupwarewizard, create a new resource locally (wich means linking
  against this resource plugin), add the resource to the manager, save the managers
  configuration and delete the resource object.

  That's really bad because you have to link against the plugin and must install it's
  header file in a public place. Furthermore the local instantiation of the resource manager
  can cause race conditions with the current implementation.

  The new way is a lot smarter. To add a new address book for example, you just call
     createAddressBook( "eGroupware", "egroupware", configXML, false );
  in the dcop interface.
  The first argument is the name of the address book, the second one the type identifier, the
  third one the configuration in xml format and the last one tells the pim daemon not
  to show the resource type specific configuration dialog.
  So you just have to create the configuration of the new address book in xml format and pass it
  to the pim daemon, which will configure and add the address book for you.
  The advantages:
    - no linking to the plugin
    - global notification of the new address book

  Since XML is quite flexible we can add version control for the configuration format.

[Please add other questions here]

This RFC is not complete now, constructive discussion is always welcome!

Prototype - Transporting PIM data via DCOP

Here you can find the code of a small prototype to transport PIM data via DCOP. You need a current libkabc from 3.5 branch to get the transportation of contacts done. For serializing events/todos/journals I'll provide a patch for libkcal soon.

API Example

class AddressBookInterface
{
  K_DCOP

  public:
    AddressBookInterface();
    ~AddressBookInterface();

    enum LockType {
      NoLock,
      ReadLock,
      WriteLock,
      ReadWriteLock
    };

  k_dcop:
    /**
      Address Book Management

      The following functions allow you to manage address books ( create, edit, delete ).
     */

    /**
      Returns the identifiers of all available address book types.
     */
    QStringList addressBookTypes() const;

    /**
      Returns the i18n'ed name for a given address book type.
     */
    QString addressBookTypeName( const QString &type ) const;

    /**
      Returns the i18n'ed description for a given address book type.
     */
    QString addressBookTypeDescription( const QString &type ) const;

    /**
      Returns the icon assocciated with the given address book type.
     */
    QPixmap addressBookTypeIcon( const QString &type ) const;

    /**
      Returns the unique identifiers of all available address books.
     */
    QStringList addressBooks() const;

    /**
      Returns the name for a given address book identifier.
     */
    QString addressBookName( const QString &identifier ) const;

    /**
      Returns the type for a given address book identifier.
     */
    QString addressBookType( const QString &identifier ) const;

    /**
      Creates a new address book with the given @param type and the given @param name.

      The @param configuration is an address book type specific xml string which is
      used as default configuration by the new address book.
      If @showConfigDialog is true, a configuration dialog is shown where the user
      can configure the address book manually.

      @returns The unique identifier for the new address book or an empty string
              if the creation was canceled.
     */
    QString createAddressBook( const QString &name, const QString &type, const QString &configuration,
                               bool showConfigDialog = true );

    /**
      Changes the address book with the given unique @param identifier.
      @param configuration The new configuration of this address book.
      @param showConfigDialog If true a configuration dialog is shown where the user
             can configure the address book manually.
     */
    void changeAddressBook( const QString &identifier, const QString &configuration,
                            bool showConfigDialog = true );

    /**
      Deletes the address book with the given unique @param identifier.
     */
    void deleteAddressBook( const QString &identifier );

    /**
      Returns the current lock status of the address book with the given address book @param identifier.
     */
    LockType addressBookLock( const QString &identifier ) const;

    /**
      Sets a lock status for the address book with the given address book @param identifier.
     */
    void addressBookSetLock( const QString &identifier, LockType lockType );

    /**
      Returns whether the address book with the given unique @param identifier
      is active.
     */
    bool addressBookActive( const QString &identifier ) const;

    /**
      Sets the address book with the given identifier active or passive.
     */
    void addressBookSetActive( const QString &identifier, bool active );

    /**
      Contact Management

      The following functions allow you to manage contacts ( create, change, delete ).
     */

    /**
      Adds a new contact to the given address book.
      @param identifier The address book identifier of the address book where the contact shall be added.
      @param vCard contains the contact in vCard 3.0 format.
     */
    void createContact( const QString &identifier, const QString &vCard );

    /**
      Adds a list of new contacts to the given address book.
      @param identifier The address book identifier of the address book where the contacts shall be added.
      @param vCards contains the contacts in vCard 3.0 format.
     */
    void createContacts( const QString &identifier, const QStringList &vCards );

    /**
      Changes the contact with given unique @param identifier.
      @param vCard contains the contact in vCard 3.0 format.
     */
    void changeContact( const QString &identifier, const QString &vCard );

    /**
      Removes the contact with the given unique @param identifier from the address book.
     */
    void deleteContact( const QString &identifier );

    /**
      Removes the contacts with the given unique @param identifiers from the address book.
     */
    void deleteContacts( const QStringList &identifiers );

    /**
      Returns the contact in vCard 3.0 format for the given unique @param identifier.
     */
    QString contact( const QString &identifier );

    /**
      Returns the contacts in vCard 3.0 format for the given unique @param identifiers.
     */
    QStringList contacts( const QStringList &identifiers );

    /**
      Category Management

      The following functions allow you to manage the address book categories.
     */

    /**
      Returns all categories provided by the address book with the given @param identifier.
     */
    QStringList categories( const QString &identifier );

    /**
      Creates a new category for the address book with the given @param identifier.
     */
    void createCategory( const QString &identifier, const QString &category );

    /**
      Deletes the @param category from the address book with the given @param identifier.
     */
    void deleteCategory( const QString &identifier, const QString &category );

    /**
      Custom Fields Management

      The following functions allow you to manage the custom fields.
     */

    /**
      Creates a new custom field.

      @param key The key of this custom field.
      @param label The i18n'ed label for this custom field.
     */
    void createCustomField( const QString &key, const QString &label );

    /**
      Deletes the custom field with the given @param key.
     */
    void deleteCustomField( const QString &key );

    /**
      Returns the list of keys from all custom fields.
     */
    QStringList customFields() const;

    /**
      Returns the label of the custom field with the given @param key.
     */
    QString customFieldLabel( const QString &key ) const;

  k_dcop_signals:
    void addressBookCreated( const QString &identifier );
    void addressBookChanged( const QString &identifier );
    void addressBookDeleted( const QString &identifier );
    void addressBookLocked( const QString &identifier, bool locked );
    void addressBookActivated( const QString &identifier, bool activated );

    void contactCreated( const QString &addressBookIdentifier, const QString &identifier );
    void contactChanged( const QString &addressBookIdentifier, const QString &identifier );
    void contactDeleted( const QString &addressBookIdentifier, const QString &identifier );

    void categoryCreated( const QString &addressBookIdentifier, const QString &category );
    void categoryDeleted( const QString &addressBookIdentifier, const QString &category );

    void customFieldCreated( const QString &key );
    void customFieldDeleted( const QString &key );
};