From January 3rd to 5th 2003, KDE PIM developers met in Osnabrück (Germany) to discuss about and hack on the KDE PIM application family. Here are some reports of the hackfest:

Short Summary Report

This is a short summary report about the Kroupware/Kaplan Hackfest that took place on the first weekend in 2003 in Osnabrück, Germany.

The overall goal of the meeting was reached. Significant progress was made towards bringing the KDE Kolab client functionality towards a better integration with the upcoming architecture of KDE-PIM mainly associated with the Kaplan approach. All participants developed a common vision about how to proceed. No major conceptual differences remained. In addition to this some technical issues could also be resolved right away. For others basically consensus was reached on what needs to be done in the next steps.

People started to drop in on friday and left on sunday afternoon. Travel expenses and the hotel logging was payed by Erfrakon, Klarälvdalens Datakonsult and Intevation, the companies doing the Kroupware project. Bandwidth, drinks and rooms were provided by Intevation.

A kickoff meeting started the hackfest on friday afternoon. To get everyone on common ground presentations were given. After a general overview about the Kroupware project, more technical introductions to the currently implemented KDE Kolab client and the Kaplan approach followed.

At the core of the Kolab architecture stands the disconnected IMAP protocol. Basically this is used to save all sorts of information on the server and allowing offline changes which are later synced. This capability is not special to groupware in any way. The current KDE client implementation based on the upcoming KDE 3.1 made KMail, KOrganizer and KAddressbook save their information using disconnected IMAP controlled by KMail.

Kaplan basically tries to provide access to the capability of the KDE components relating to personal information management. In this sense it needs all KParts done cleanly in a way that the single applications can still work standalone without being embedded in Kaplan. Kaplan therefore does not add much logic itself or solve the problem of interaction. It only forces a clean design and usage of common configuration and interaction methods.

After the problem space was explored work was done in alternating between working groups and meetings for coordination and discussions of new findings. As natural for an event like this, many successful informal conversations also took place. In the following you can read a few lines on what was archived by the different people on this first January weekend.

David Faure and Daniel Molkentin analysed DCOP and found that a better general DCOP service starter is needed which was subsequently implemented. DCOP error handling got examined and a small documentation on how to properly write clean DCOP functions was produced. KAddressbook was ported as an example. It is clear that more general DCOP interfaces need to be defined so that KMail and KOrganizer can act as Kolab client together.

Tobias König presented the (new) resource framework to the group. He progressed its implementation and started porting libkabc to the new framework.

Bo Thorsen and Steffen Hansen worked on various bugfixes of KMail and KOrganiser code and ported most of KMail/Kroupware to HEAD. They, together with Karl-Heinz Zimmer, answered many questions about the code they contributed for the Kroupware project.

Lutz Rogowski made some progress to have KOrganizer as a KPart and made himself familiar with Kitchensync.

Cornelius Schumacher developed a concept for the iCalender parsing and worked on porting KOrganizer to the new resource framework.

Günter Schwann fixed a KParts bug and started on the "imap"-resource for KOrganizer.

Ingo Klöcker helped porting KMail/Kroupware to HEAD and did miscellaneous KMail cleanups. Marc Mutz worked on KMail, especially the IMAP account code and prepared and conducted several code merges to HEAD. Marc also helped to prepare the meeting.

Having many developers on site gave us the good opportunity to discuss the long standing questions whether to move KMail to the kdepim module. The question was approached with structured discussion. In a first step pro and con arguments were collected. Everybody had to wait for the second step to give their opinion on how to weight the arguments. In the end consensus was reached and the move was favoured.

More details on the results will be reported by the respective developers on the appropriate mailinglists.

This report is brought to you by Bernhard Reiter, who was the responsible guy from Intevation on site. He coordinated and moderated the meeting.

Moving KMail, KNode, Korn and related libraries to kdepim

This message was posted to kde-core-devel mailinglist on January 10, 2003 by Ingo Klöcker.

Dear developers,

the question whether KMail should be moved to kdepim has already arisen quite a few times on [email protected] (and probably also elsewhere). Last week end at the Kroupware/Kaplan hack fest (cf. the Announcement on kde-core-devel on 21.12.02) we took the chance to talk about this from face to face. After weighing the pros and the cons we finally came to the unanimous conclusion that KMail and the corresponding libs (and therefore also KNode and Korn which also depend on those libs) should be moved.

The pros were:

Less code duplication. Currently quite a few code is duplicated, e.g. kdepim/kalarm contains copies of kmime_* files which are actually part of kdenetwork/libkdenetwork. Furthermore all the KOrganizer KActions are duplicated in KMail and there is a second iCalendar parser in KMail which would become obsolete if KMail could use libkcal which is in kdepim. The problems of duplicated code could of course also be solved by moving some of it to kdelibs and introducing plugin interfaces, but moving KMail to kdepim means less work now and in the future.
Reduction of dependencies between CVS modules. Although the dependencies within kdepim would become more complex at first, we expect that it will be easier to resolve this when the code is in one place.
We expect the move to foster communication and cooperation of both developer groups and to be of benefit for both, KMail and kdepim.
An email client is part of a PIM solution. And as we now have Kaplan it makes sense to put at least the most important Kaplan parts into one cvs module.

The cons were:

The size of the kdepim cvs module will grow. (OTOH the total size of kdepim and kdenetwork will shrink because some duplicate files can be removed immediately after the move.)
Danger of uncleaner interfaces. If applications are in the same module it's not imperative to make clean interfaces. We might be tempted to introduce some nasty dependencies e.g. between KO and KM just because it's so easy as they are the same cvs module.
Danger of creation of a monolithic application. This is related to the second point. The counter argument to this and the second point is that the Kaplan framework forces the part developers to define clean interfaces. So this should prevent Kaplan from becoming one big fat swiss army knife application.

If there are no serious objections of developers which didn't participate in the Osnabrueck meeting, we will make the move at Friday, 17th of January.

Regards,
Cornelius (for KDE PIM) and Ingo (for KMail)

Participants of the hack fest:

KDE PIM and KMail developers:
- Ingo Klöcker (for KMail)
- Tobias Koenig (for KAddressBook, PIM in general)
- Daniel Molkentin (for Kaplan)
- Cornelius Schumacher (for KOrganizer, PIM in general)
- Günter Schwann (for KOrganizer group scheduling)
from the Kroupware Project:
- David Faure
- Steffen Hansen
- Marc Mutz
- Bernhard Reiter
- Lutz Rogowski
- Bo Thorsen
- Karl-Heinz Zimmer

Additional note Jan 18th: KMail and co are now in kdepim.

Merging kroupware branch into kdepim (mainly korganizer)

This report was contributed by Cornelius Schumacher.

Steffen, Lutz, Günter and Cornelius discussed how to finish the merge of the kroupware branch into the HEAD branch. There is still missing the Free/Busy handling, the configuration, changes to the attendee tab of the todo editor and some minor things.

We agreed on the details how to proceed with the merge (see below). To get all the functionality into the HEAD branch we can temporarily merge the kokroupware file and resolve the remaining isssues in the HEAD branch.

The kroupware developers will send patches. As the HEAD branch is diverging quickly from the 3.1 and the kroupware branch it will be less work, if the merge is done as soon as possible.

Details:

The KNotes view in KOrganizer will be removed. This will be replaced by the KNotes plugin in Kaplan. There is a make_it_cool branch in knotes which saves the data to iCalendar. This should be changed to be able to also use the IMAP resource (see report about new resource framework) to store the notes data to the kolab server.
The signal/slots communication will be replaced by DCOP calls. This will make some ugly things go away like the KMail main window pointer in KOrganizer.
The attendee tab of the event editor with integrated free/busy view is too big for small screen resolutions. The free/busy view will be moved to an own tab which also includes the controls for setting start and end time of an event. Adding/editing attendees in the free/busy view tab will be handled by context menus and maybe inline editing in the gantt widget. The attendee tab will remain as it is in the non-kroupware branches. The 12/24 hour format switch will be replaced by accessing the KDE preferences via KLocale.
The attendee tab of the todo editor has been replaced in the kroupware branch. Currently it uses a #define to switch bewteen the two versions. This will be changed to be configurable at run-time. Which attendee tab is used in the editor will be chosen by the number of attendees associated with the todo. If there is none or only one, the kroupware version is used, if there are more the standard conformat version from HEAD is used.
There are some issues with Outlook interoperability where Outlook doesn't conform to the iCalendar/iTIP standards or does only support a subset of it. There are good reasons to be interoperable with Outlook and this might justify an configuration option to enable Outlook compatible behaviour. This shouldn't limit the ability to behave standards-compliant, though.
Accessing calendar data on the kolab server will be handled by a IMAP resource talking via DCOP to KMail.

New resource framework

This report was contributed by Cornelius Schumacher.

Jan-Pascal van Best has developed a generalization of the resource framework of libkabc for accessing an Exchange server in KOrganizer. This framework provides a modular way to define, implement and use specific resources for calendar, addressbook and similar data. Examples are iCalendar or vCard files and servers using protocols like LDAP, HTTP, IMAP or SQL.

At the meeting the resource framework was moved from kdepim to kdelibs/kresources. Tobias ported libkabc to make use of kresources, Cornelius started to adapt libkcal and KOrganizer to the resource framework and Guenter converted the CalendarIMAP class used to access the calendar data on a kolab server from Korganizer to a resource fitting into the new framework.

Note: In the meantime the KOrganizer and libkcal adaptions have reached a state where it is possible to actually use resources of the new framework in KOrganizer.

Implement DCOP communication between two applications

This report was contributed by Cornelius Schumacher.

The latest version of this documentation is available (as text file) at the KDE CVS.

How to implement DCOP communication between two applications so that

it works when the applications are standalone (separate processes)
it works when the applications are loaded as parts, embedded into kontact
it behaves properly when a separate process exits/crashes.

In the part

Let's say that part 'A' wants to use the interface "Foo", via DCOP. (where Foo is usually a generic name, e.g. Calendar, Mailer, AlarmDaemon, etc.) The services which implement this interface are associated with the service type "DCOP/Foo".

One of those services is application 'B', which implements "Foo" - note that 'B' should make sure that the "Foo" DCOP interface is available both when 'B' is used as standalone process and when 'B' is only loaded as a part. (This means that if the app doesn't use its own part, then both should implement "Foo", like kaddressbook does. Of course it's simpler if the app uses its own part :)

Here are some code snippets that must go into the part (A) that wants to use "Foo":

* Constructor:
 m_foo_stub = 0L;
 kapp->dcopClient()->setNotifications( true );
 connect( kapp->dcopClient(), SIGNAL( applicationRemoved( const QCString&)),
          this, SLOT( unregisteredFromDCOP( const QCString& )) );

* Destructor:
 kapp->dcopClient()->setNotifications( false );
 delete m_foo_stub;

[Note that setNotifications() is implemented with a refcount, this is the correct way to do it and it won't mess up other parts]

* bool connectToFoo() method,
which uses KDCOPServiceStarter::self()->findServiceFor("DCOP/Foo").
See test part for details (plugins/test/test_part.cpp).

* unregisteredFromDCOP( const QCString& appId ) slot, which will be called when
the process implementing Foo exits. The method simply does:
 if ( m_foo_stub && m_foo_stub->app() == appId )
 {
   delete m_foo_stub;
   m_foo_stub = 0;
 }

* Now you can finally use the foo dcop interface. First you need to connect
to it:
 if ( !connectToFoo() )
   return;

Then you can use m_foo_stub to call the DCOP methods. In case of critical methods, where you want to make 100% sure that the DCOP call was correctly done (e.g. the remote app didn't crash during the call), you can use

  if ( !m_foo_stub->ok() )

In the kontact plugin

Don't use dcopClient() until the part is loaded
After loading the part, you might want to create a DCOP stub to use some of its methods (do both in a loadPart() method, e.g.).
Implement createDCOPInterface( const QString& serviceType ), to load the part if the serviceType is one provided by it.

See KAddressbookPlugin (plugins/kaddressbook/*) for a working example.

Don't forget to

Define the service type, using a "Type=ServiceType" .desktop file, with "X-KDE-ServiceType=DCOP/Foo". See e.g. kdepim/kaddressbook/dcopaddressbook.desktop
Add DCOP/Foo to the application's ServiceTypes list, in its .desktop file. See e.g. kdepim/kaddressbook/kaddressbook.desktop
Make sure that X-DCOP-ServiceType and X-DCOP-ServiceName are specified too.

Designing DCOP interfaces

Porting the kroupware signals/slots to DCOP requires some changes. For instance any non-const reference (such as those used for returning values to the caller) has to be changed. If there is more than one value to be returned, you need to

define a structure containing all the returned values
define QDataStream << and >> operators for that structure.

KMail status in kroupware

This report was contributed by Bo Thorsen.

During the Osnabrück meeting, the KMail merge to HEAD was completed. This means we have added Disconnected IMAP, MDN, vacation setup, automatic resource scheduling, an IMAP Resource backend to KMail, and KOrganizer connectivity for groupware scheduling.

Here, I will give an overview of each of these technologies.

The Disconnected IMAP account

Disconnected IMAP - from here on dIMAP - works by synchronizing the IMAP server contents to the local disk. The KMail user will after the sync work on the local cache, and upon the next synchronization, all local changes will be uploaded to the server before new mails are downloaded to the local cache. The basic idea is that after the synchronization, the contents of the server and the local cache are identical.

The user benefits in this is that you do not have to be online to go through the mail boxes. It is also an easy way to make two machines have the same mail setup - for example you workstation and a laptop.

With the current IMAP implementation in KMail no local cache is used which means the user must be online while going through the mail boxes. The biggest user benefit in this is that you local disk usage is almost zero - not a small benefit if you have several gigabytes of email.

The future work for dIMAP includes two enhancements: Selecting a folders to be synchronized, and creating filters for what will be synchronized.

Carsten Burghardt have actually already implemented the folder selection. The idea here is that normally you won't synchronize all folders - users often have old mail archives that are not often changed.

Synchronization filters is a setup where the user can select parts of the folders that shouldn't be downloaded to the local cache (unless explicitly told to do so). Examples include "Don't cache mails larger than X kb", "Don't cache attachments" and "Don't cache mails older than two months".

Comparing the two IMAP implementations show that the current IMAP implementation is almost the same as the behaviour you get when using a dIMAP implementation with a sync filter set to "Don't cache anything" and the user deleting the cache after reading a mail. The current implementation is very handy in the situation where users have very little disk space available. When this isn't the case, the dIMAP implementation should prove more userfriendly.

The implementation of dIMAP is approaching a very stable situation, but the HEAD implementation still experiences crashes. Data loss haven't been seen in testing since november, but the local cache has been seen to be confused. Advice to people wanting to use dIMAP is to backup mails they really don't want to loose. The implementation is definately stable enough for more thorough testing so we can find the remaining bugs that are there. Speed-wise, the implementation isn't as effective as it could be. We have concentrated completely on stability and sacrificed all speedups we could think of to make the implementation as stable as it can be. The effect should be that it's much more stable, but synchronization can take quite a long time. Later work on this should yield big speedups in the synchronization.

dIMAP uses the normal KMail maildir for the local cache, so technically, what we did was to make the synchronization code and not touch the local storage. This means that the local cache is just as stable as having normal local maildir folders. At a later point we hope to decouple the account code completely from the folder code, so the user can choose which type of folder he wants as the local cache.

MDN - Message Disposition Notification

MDNs are a generalization of what is commonly called "read receipt". The message author requests a disposition notification to be sent and the receiver's mail program generates a reply from which the author can learn what happened to his message. Common disposition types include "displayed" (i.e. read), "deleted" and "dispatched" (e.g. forwarded).

In KMail you can request notification when people read the mail you sent, and KMail can send notification when you read mail sent to you with notification requests.

Vacation setup

When your mail server supports Sieve filtering, KMail can set up vacation notifications for you.

Automatic resource scheduling

In groupware mode, you can setup accounts to work as automatic resource scheduling. With this, you set up accounts for each resource you want to schedule. An example could be a room. You make an account that holds the rooms calendar, and people "invite" the room to a meeting. When KMail receives such an invitation, it automatically accepts if the resource is free - if not, it rejects the invitation. We don't mind people calling this a hack, but it works surprisingly well :-)

IMAP Resource backend to KMail

The new resource framework in kdelibs and kdepim have different backend implementations. The obvious one is to use the local disk for the storage, which is the standard implementation. For the addressbook, there is also an LDAP backend, and so on.

We have implemented a resource backend in KMail, so this can be used for storage. This means KOrganizer, KAddressbook and other applications can save their data in KMail folders. This is specifically done with dIMAP in mind, so synchronization in addition to emails also syncs calendar, contacts and notes.

KOrganizer connectivity for groupware scheduling

To complete the groupware functionality, code have been added to KMail to intercept incoming groupware messages.

In KMail, different things happen depending on what is in the message currently being read. We have added the check for mimetypes holding calendar related information. When a message with such a mimetype is seen, the iCal file is given to the groupware code that reformats it to a HTML message which includes the choices the user has. So when an invitation arrives, this is transformed to a message saying you have been invited to a meeting in XX from YY to ZZ. Below that there are four links with different options, the most important being accept and decline. When the user clicks one of these links, KMail will call KOrganizer code with the message and the user choice. After that it is up to KOrganizer to put the information into the right resource (which could be the IMAP resource as described above).

In addition to this, KMail also offers to format and send the messages from KOrganizer, so correct replies and invitations are sent.

The general split here is that KOrganizer handles all iCal related, and KMail handles how the iCal is put into mails or read from mails.

In kroupware_branch all this is working, but in cvs HEAD there are still some issues to be sorted out before this is working. The difference is that in kroupware_branch, all communications is done with Qt signals and slots, where in HEAD it's done with DCOP. This is currently being retrofitted to work in the Kontact framework and is looking very promising.