Revision as of 17:30, 6 November 2009

SingleFileResource Refactoring

Problem Statement

The current (KDE 4.[3,4]) design of the SingleFileResource (SFR) is heavily based on the ical/vcard resources. Those two read the whole file at once and keep the entries in memory as long as they exists. However this is not really in line with the Akonadi design, as Akonadi itself keeps items in chache also and has advanced cache policies even. Also, other resources, like the mbox resource don't keep the whole file in memory but only store pointers to the beginning of each new entry. These two concepts conflict with each other as soon as Akonadi detects that the file on disk has changed. Currently SFR makes a backup of the file by calling writeFile() with a different file name. But this can only succeed correctly if the resource has all data in memory. This is not the case for the mbox resource (and this shouldn't be the case for the other SFR based resources in the future) which will result in a backup file that partly consist of the new data and partly the old data (if it succeeds at all).

Suggested Changes

General

The file parsers should never store the complete data in memory
When items are requested from the file parsers it always should read from the file it is initialized for.
The SFR only reads the file first at retrieveItems.

On item retrieval

Retrieve item first makes a copy of the file (i.e. as done now for remote files but, now its done always)
Create items from the copied file
Call itemsRetrieved

On file change

A second copy of the file is created.
read both copies, compare item by item, using any dirty item in favor of the respective one from copy 1.
do not call synchronize, modify Akonadi like any resource with item level change notification.
finally remove copy 1 and make copy 2 the new copy 1

Drawbacks

Two files are parsed on fileChanged()

Advantages

Reduced memory usage
if dirty items are not part of the change, no need to backup or panic ???
Better data consistency in case files are changed by non-Akonadi programs.

@@ Line 1: / Line 1: @@
 = SingleFileResource Refactoring =
+== Problem Statement ==
+The current (KDE 4.[3,4]) design of the SingleFileResource (SFR) is heavily based on the ical/vcard resources. Those two read the whole file at once and keep the entries in memory as long as they exists. However this is not really in line with the Akonadi design, as Akonadi itself keeps items in chache also and has advanced cache policies even. Also, other resources, like the mbox resource don't keep the whole file in memory but only store pointers to the beginning of each new entry. These two concepts conflict with each other as soon as Akonadi detects that the file on disk has changed. Currently SFR makes a backup of the file by calling writeFile() with a different file name. But this can only succeed correctly if the resource has all data in memory. This is not the case for the mbox resource (and this shouldn't be the case for the other SFR based resources in the future) which will result in a backup file that partly consist of the new data and partly the old data (if it succeeds at all).
+== Suggested Changes ==
+=== General ===
+* The file parsers should never store the complete data in memory
+* When items are requested from the file parsers it always should read from the file it is initialized for.
+* The SFR only reads the file first at retrieveItems.
+=== On item retrieval ===
+* Retrieve item first makes a copy of the file (i.e. as done now for remote files but, now its done always)
+* Create items from the copied file
+* Call itemsRetrieved
+=== On file change ===
+* A second copy of the file is created.
+* read both copies, compare item by item, using any dirty item in favor of the respective one from copy 1.
+* do not call synchronize, modify Akonadi like any resource with item level change notification.
+* finally remove copy 1 and make copy 2 the new copy 1
+=== Drawbacks ===
+* Two files are parsed on fileChanged()
+=== Advantages ===
+* Reduced memory usage
+* if dirty items are not part of the change, no need to backup or panic ???
+* Better data consistency in case files are changed by non-Akonadi programs.