Frameworks/Akonadi: Difference between revisions

From KDE Community Wiki
(→‎Server: Cache and share Collection statistics)
 
(14 intermediate revisions by the same user not shown)
Line 9: Line 9:
  /src
  /src
  /src/agentbase (libKF5AkonadiAgentBase.so - AgentBase, ResourceBase and other classes related to agents. Needs a better name)
  /src/agentbase (libKF5AkonadiAgentBase.so - AgentBase, ResourceBase and other classes related to agents. Needs a better name)
  /src/agentserver (do we actually use it anywhere? Should it just die?)
  /src/agentlauncher (utility for loading and running plugin-based agents)
  /src/akonadictl (akonadictl utility)
  /src/akonadictl (akonadictl utility)
  /src/asapcat (asapcat utility)
  /src/asapcat (asapcat utility)
Line 39: Line 39:
|- {{ FeatureInProgress | Fix coding-style (Guy) | We have to follow the Frameworks coding style }}
|- {{ FeatureInProgress | Fix coding-style (Guy) | We have to follow the Frameworks coding style }}
|- {{ FeatureTodo | Port to Qt 5 | Make client libraries compile against Qt 5 }}
|- {{ FeatureTodo | Port to Qt 5 | Make client libraries compile against Qt 5 }}
|- {{ FeatureTodo | Replace BOOST_STATIC_ASSERT | C++11 has static_assert, to use that }}
|- {{ FeatureTodo | Replace BOOST_STATIC_ASSERT | C++11 has static_assert, so use that }}
 
|}
|}


Line 52: Line 51:
! Branch
! Branch
! Assignee
! Assignee
|- {{ FrameworksTodo | AgentBase | Merge ObserverV2 into Observer | There should be "Observer", "BatchObserver" and "TagsObserver" (not inheriting from each other). BatchObserver would enable the same behavior as ObserverV3 currently, TagsObserver would enable tags notifications. | | frameworks | }}
|- {{ FrameworkTodo | AgentBase | Merge ObserverV2 into Observer | There should be "Observer", "BatchObserver" and "TagsObserver" (not inheriting from each other). BatchObserver would enable the same behavior as ObserverV3 currently, TagsObserver would enable tags notifications. | frameworks | }}
|- {{ FrameworksTodo | AgentSearchInterface | Remove addSearch() and removeSearch() methods | Search is managed on server now, so we only need search() method to get results. add/removeSearch() are no longer used | frameworks | }}
|- {{ FrameworkTodo | AgentSearchInterface | Remove addSearch() and removeSearch() methods. Search is managed on server now, so we only need search() method to get results. add/removeSearch() are no longer used | | frameworks | }}
|- {{ FrameworksTodo | AgentBase | Merge ObserversV* into Observer | There should be "Observer" and "BatchObserver" (not inheriting from each other). BatchObserver would enable the same behavior as ObserverV3 currently | frameworks | }}
|- {{ FrameworkTodo | AgentBase | Merge ObserversV* into Observer | There should be "Observer" and "BatchObserver" (not inheriting from each other). BatchObserver would enable the same behavior as ObserverV3 currently | frameworks | }}
|- {{ FrameworksTodo | ChangeMediator | Remove unused methods | ChangeMediator is only used by ModifyJobs to invalidate entries in Monitor's EntityCache, but it has bunch of stub methods we don't need, so remove them | master | }}
|- {{ FrameworkTodo | ChangeMediator | Remove unused methods | ChangeMediator is only used by ModifyJobs to invalidate entries in Monitor's EntityCache, but it has bunch of stub methods we don't need, so remove them | master | }}
|- {{ FrameworksTodo | Collection | Remove deprecated methods | frameworks | | }}
|- {{ FrameworkTodo | Collection | Remove deprecated methods | | frameworks | }}
|- {{ FrameworksTodo | CollectionFetchJob | Remove deprecated methods | frameworks | | }}
|- {{ FrameworkTodo | CollectionFetchJob | Remove deprecated methods | | frameworks | }}
|- {{ FrameworksTodo | CollectionFetchScope | Remove deprecated methods | frameworks | | }}
|- {{ FrameworkTodo | CollectionFetchScope | Remove deprecated methods | | frameworks | }}
|- {{ FrameworksTodo | CollectionModel | Remove | frameworks | | }}
|- {{ FrameworkTodo | CollectionModel | Remove | | frameworks | }}
|- {{ FrameworksTodo | CollectionSelectJob | Remove? | Performs SELECT before FETCH, STORE, APPEND, ... commands. This mimics IMAP behavior, but since we have diverged from IMAP quite a lot by now, we could just change syntax of the commands to include a "COLLECTION" parameter. This could speed up  FETCH operation a little | master? |  }}
|- {{ FrameworkTodo | CollectionSelectJob | Remove? | Performs SELECT before FETCH, STORE, APPEND, ... commands. This mimics IMAP behavior, but since we have diverged from IMAP quite a lot by now, we could just change syntax of the commands to include a "COLLECTION" parameter. This could speed up  FETCH operation a little | master? |  }}
|- {{ FrameworksTodo | CollectionStatisticsModel | Remove | frameworks | | }}
|- {{ FrameworkTodo | CollectionStatisticsModel | Remove | | frameworks | }}
|- {{ FrameworksTodo | CollectionView | Remove deprecated methods | frameworks | | }}
|- {{ FrameworkTodo | CollectionView | Remove deprecated methods || frameworks | }}
|- {{ FrameworksTodo | DBusConnectionPool | Remove | master? | | }}
|- {{ FrameworkTodo | DBusConnectionPool | Remove || master? | }}
|- {{ FrameworksTodo | Entity | Make it a subclass of AttributeEntity | AttributeEntity was introduced to allow sharing code between Tags and Entities. | frameworks | }}
|- {{ FrameworkTodo | Entity | Make it a subclass of AttributeEntity | AttributeEntity was introduced to allow sharing code between Tags and Entities. | frameworks | }}
|- {{ FrameworksTodo | Entity | Remove AKONADI_DECLARE_PRIVATE | | frameworks | }}
|- {{ FrameworkTodo | Entity | Remove AKONADI_DECLARE_PRIVATE | | frameworks | }}
|- {{ FrameworksTodo | EntityTreeViewStateSaver | Remove | | frameworks | }}
|- {{ FrameworkTodo | Entity | Create ''T* Entity::attribute<T>()'' and ''const T* Entity::attribute<T>() const'' overloads | Needed to preserve const-correctness. Right now there's T* Entity::attribute<T>() const, which allows modifying the attribute (and hence the collection) even though the pointer was returned from const method | frameworks | }}
|- {{ FrameworksTodo | IndexPolicyAttribute | Remove if Baloo Indexer decides to ignore it | | frameworks | }}
|- {{ FrameworkTodo | EntityTreeViewStateSaver | Remove | | frameworks | }}
|- {{ FrameworksTodo | ItemFetchScope | Normalize setter/getter names | | frameworks | }}
|- {{ FrameworkTodo | IndexPolicyAttribute | Remove if Baloo Indexer decides to ignore it | | frameworks | }}
|- {{ FrameworksTodo | ItemModel | Remove | | frameworks | }}
|- {{ FrameworkTodo | ItemFetchScope | Normalize setter/getter names | | frameworks | }}
|- {{ FrameworkTodo | ItemModel | Remove | | frameworks | }}
|- {{ FrameworkTodo | ItemSearchJob | Remove deprecated methods | | frameworks | }}
|- {{ FrameworkTodo | ItemSearchJob | Remove deprecated methods | | frameworks | }}
|- {{ FrameworkTodo | ItemSync | Make private? | Does not seem to be used anywhere | frameworks |  }}
|- {{ FrameworkTodo | ItemSync | Make private? | Does not seem to be used anywhere | frameworks |  }}
Line 75: Line 75:
|- {{ FrameworkTodo | KDSignalBlocker | Remove? | | frameworks | }}
|- {{ FrameworkTodo | KDSignalBlocker | Remove? | | frameworks | }}
|- {{ FrameworkTodo | metatypes.h | Remove | | master? | }}
|- {{ FrameworkTodo | metatypes.h | Remove | | master? | }}
|- {{ FrameworkTodo | MimeTypeChecker | Maybe remove | This class is able to compare mime types, handling aliases, inheritance, etc. AFAIK Volker once told me this should be in Qt 5, so check whether it's there. Might be worth upstreaming, too | frameworks | }}
|- {{ FrameworkTodo | MimeTypeChecker | Maybe remove | This class is able to compare mime types, handling aliases, inheritance, etc. IIRC Volker once told me this should be available in Qt 5, so check whether it's there. Might be worth upstreaming, too | frameworks | }}
|- {{ FrameworkTodo | PersistentSearchAttribute | Remove deprecated methods | | frameworks | }}
|- {{ FrameworkTodo | PersistentSearchAttribute | Remove deprecated methods | | frameworks | }}
|- {{ FrameworkTodo | PluginLoader | Remove if Qt 5 plugin system is powerful enough | | frameworks | }}
|- {{ FrameworkTodo | PluginLoader | Remove if Qt 5 plugin system is powerful enough | | frameworks | }}
Line 100: Line 100:
|- {{ FeatureTodo | (IF POSSIBLE) Share external payload parts helper  | Concentrate external parts support into one place for easier maintanence and less code duplication }}
|- {{ FeatureTodo | (IF POSSIBLE) Share external payload parts helper  | Concentrate external parts support into one place for easier maintanence and less code duplication }}
|- {{ FeatureTodo | Direct streaming of external payload parts  | For parts bigger than 4KB, the server can instead of asking client to provide the data send back a file descriptor that client will use to write the data directly to the file. Server can either verify that sizes match, or rely on client library telling the truth. Memory--, performance++ }}
|- {{ FeatureTodo | Direct streaming of external payload parts  | For parts bigger than 4KB, the server can instead of asking client to provide the data send back a file descriptor that client will use to write the data directly to the file. Server can either verify that sizes match, or rely on client library telling the truth. Memory--, performance++ }}
|- {{ FeatureTodo | Require explicit call to ItemSync::deliveryDone() | Clients should explicitly call ItemSync::deliveryDone() after the sync is over, instead of ItemSync being "clever" and detecting it on it's own. See https://git.reviewboard.kde.org/r/117673/ }}
|}
|}


Line 120: Line 121:
|- {{ FeatureTodo | Refactor DB access | Try to do as much DB interaction as possible in DataStore, so that it can be easily replaced by a FakeDataStore for testing purposes (see "Testable Handlers" below). }}
|- {{ FeatureTodo | Refactor DB access | Try to do as much DB interaction as possible in DataStore, so that it can be easily replaced by a FakeDataStore for testing purposes (see "Testable Handlers" below). }}
|- {{ FeatureTodo |  Document ASAP | Properly document all ASAP commands and responses. Ideally should be part of doxygen documentation rather than a plaintext file. }}
|- {{ FeatureTodo |  Document ASAP | Properly document all ASAP commands and responses. Ideally should be part of doxygen documentation rather than a plaintext file. }}
|- {{ FeatureTodo | Remove exception.h | Share Akonadi::Exception from client libs. }}
|- {{ FeatureDone | Optimize CacheCleaner (1.12.0) | Right now the CacheCleaner requests all collections every minute and checks their cache policies. Instead we should use some form of scheduling queue.  }}
|- {{ FeatureDone | Optimize CacheCleaner (1.12.0) | Right now the CacheCleaner requests all collections every minute and checks their cache policies. Instead we should use some form of scheduling queue.  }}
|- {{ FeatureTodo | Remove LSUB and all subscription related API | Subscribtions are obsoleted by Christian's work on LIST filtering }}
|-{{ FeatureTodo | Remove agentserver | It seems that Agent Server was only used on Maemo 5, otherwise it's disabled at build time. Since we don't really support this anymore, let's kill it. }}
|-{{ FeatureTodo | Fix StorageDebugger DBus iface | The sequence number (first argument) in queryExecuted() signal should be qint64, not double. }}
|-{{ FeatureTodo | Cache and share Collection statistics | To reduce the amount of queries we do whenever collection stats change, we should cache those values and only invalidate the cache when they change. }}
|}
|}


= Other Plans =
= Other Plans =
== Testable Handlers ==
== Testable Handlers ==
Does not really depend on Frameworks, can be done at any time. I 'want' to have it done in Frameworks though. Currently we have no way to test Handlers on the server. This would require having a fake ''DataStore'', fake ''NotificationCollector'', fake ''AkonadiConnection'' and somehow fake backend for ''Entities''. However, as a result we should be able to send various commands to the handlers and observe whether correct data are stored in database and correct signals are emitted. Big problem with external payload parts, would require a fake ''ExternalPartManager'' too.
Does not really depend on Frameworks, can be done at any time. I '''want''' to have it done in Frameworks though. Currently we have no way to test Handlers on the server. This would require having a fake ''DataStore'', fake ''NotificationCollector'', fake ''AkonadiConnection'' and somehow fake backend for ''Entities''. However, as a result we should be able to send various commands to the handlers and observe whether correct data are stored in database and correct signals are emitted. Big problem with external payload parts, would require a fake ''ExternalPartManager'' too.
 
Other option is to start a server instance, send it commands and listen to notifications and check what's in the database, but that's slow, requires lot of code and most importantly it's very difficult to debug. The approach outlined above would be much better.


== Shared protocol helper ==
== Shared protocol helper ==
In a perfect world, we would have Handlers in a shared library and the Handler would be able to parse the command into entities and vice versa. But since the server is using different entities than the client libraries (''Akonadi::Server::Collection'' vs. ''Akonadi::Collection''), this probably won't be possible. However we still should try to share code for handling ImapSets, parsing attributes, payload parts, etc. The first step toward this is to have client jobs use ImapStreamParser.
In a perfect world, we would have Handlers in a shared library and the Handler would be able to parse the command into entities and vice versa. But since the server is using different entities than the client libraries (''Akonadi::Server::Collection'' vs. ''Akonadi::Collection''), this probably won't be possible. However we still should try to share code for handling ImapSets, parsing attributes, payload parts, etc. The first step toward this is to have client jobs use ImapStreamParser.

Latest revision as of 14:40, 14 November 2014

This is a list of things I want to do in kdepimlibs/akonadi for Frameworks 5.

Akonadi repo structure

Merge client libraries and server into one repository. Main advantage is that we will be able to share more code between server and libraries.

Proposed structure:

/autotests
/doc
/src
/src/agentbase (libKF5AkonadiAgentBase.so - AgentBase, ResourceBase and other classes related to agents. Needs a better name)
/src/agentlauncher (utility for loading and running plugin-based agents)
/src/akonadictl (akonadictl utility)
/src/asapcat (asapcat utility)
/src/common (libKF5AkonadiCommon.so - private classes shared between server, core and widgets libraries and other utilities. Does NOT install any (or most of its) public headers)
/src/control (akonadi_control utility)
/src/core (libKF5AkonadiCore.so - non-UI classes, only Jobs, models, Monitor etc.)
/src/core/jobs 
/src/core/models
/src/interfaces (DBus interfaces)
/src/qsqlite (Qt's QSQLITE driver fork - better try to upstream it)
/src/selftest (based on SelfTestDialog class)
/src/server (akonadiserver)
/src/widgets (libKF5AkonadiWidgets.so - UI classes, depends on core)
/src/widgets/views
/src/xml (akonadi/xml library)
/tests/akonaditest (akonaditest utility)
/tests/fake (fake classes to be used for testing)
/tests/searchplugin (akonadi_testsearch_plugin)
/tests/testresource (akonadi_knut_resource)

Client libraries

Code changes

Status Action Comment
IN PROGRESS Fix coding-style (Guy) We have to follow the Frameworks coding style
TO DO Port to Qt 5 Make client libraries compile against Qt 5 <{{{3}}}>
TO DO Replace BOOST_STATIC_ASSERT C++11 has static_assert, so use that <{{{3}}}>

API/ABI changes

Status Class Action Comment Branch Assignee
TODO AgentBase Merge ObserverV2 into Observer There should be "Observer", "BatchObserver" and "TagsObserver" (not inheriting from each other). BatchObserver would enable the same behavior as ObserverV3 currently, TagsObserver would enable tags notifications. frameworks
TODO AgentSearchInterface Remove addSearch() and removeSearch() methods. Search is managed on server now, so we only need search() method to get results. add/removeSearch() are no longer used frameworks
TODO AgentBase Merge ObserversV* into Observer There should be "Observer" and "BatchObserver" (not inheriting from each other). BatchObserver would enable the same behavior as ObserverV3 currently frameworks
TODO ChangeMediator Remove unused methods ChangeMediator is only used by ModifyJobs to invalidate entries in Monitor's EntityCache, but it has bunch of stub methods we don't need, so remove them master
TODO Collection Remove deprecated methods frameworks
TODO CollectionFetchJob Remove deprecated methods frameworks
TODO CollectionFetchScope Remove deprecated methods frameworks
TODO CollectionModel Remove frameworks
TODO CollectionSelectJob Remove? Performs SELECT before FETCH, STORE, APPEND, ... commands. This mimics IMAP behavior, but since we have diverged from IMAP quite a lot by now, we could just change syntax of the commands to include a "COLLECTION" parameter. This could speed up FETCH operation a little master?
TODO CollectionStatisticsModel Remove frameworks
TODO CollectionView Remove deprecated methods frameworks
TODO DBusConnectionPool Remove master?
TODO Entity Make it a subclass of AttributeEntity AttributeEntity was introduced to allow sharing code between Tags and Entities. frameworks
TODO Entity Remove AKONADI_DECLARE_PRIVATE frameworks
TODO Entity Create T* Entity::attribute<T>() and const T* Entity::attribute<T>() const overloads Needed to preserve const-correctness. Right now there's T* Entity::attribute<T>() const, which allows modifying the attribute (and hence the collection) even though the pointer was returned from const method frameworks
TODO EntityTreeViewStateSaver Remove frameworks
TODO IndexPolicyAttribute Remove if Baloo Indexer decides to ignore it frameworks
TODO ItemFetchScope Normalize setter/getter names frameworks
TODO ItemModel Remove frameworks
TODO ItemSearchJob Remove deprecated methods frameworks
TODO ItemSync Make private? Does not seem to be used anywhere frameworks
TODO ItemView Remove deprecated methods frameworks
TODO KDSignalBlocker Remove? frameworks
TODO metatypes.h Remove master?
TODO MimeTypeChecker Maybe remove This class is able to compare mime types, handling aliases, inheritance, etc. IIRC Volker once told me this should be available in Qt 5, so check whether it's there. Might be worth upstreaming, too frameworks
TODO PersistentSearchAttribute Remove deprecated methods frameworks
TODO PluginLoader Remove if Qt 5 plugin system is powerful enough frameworks
TODO ProtocolHelper Share with Server See the Other Plans section below frameworks
TODO RecursiveItemFetchJob Optimize Emit itemsReceived() signal from underlaying fetch jobs master
TODO RenameFavoriteDialog Make private? frameworks
TODO ResourceSettings Remove? frameworks
TODO SearchCreateJob Remove deprecated methods frameworks
TODO SelfTestDialog Make a standalone application SelfTestDialog is the only user of QtSQL in client libraries. By making it a stand-alone app, we can get rid of one dependency from libKF5AkonadiCore.so master
TODO StatisticsProxyModel Remove frameworks
TODO SubscriptionJob Split to CollectionSubscribeJob and CollectionUnsubscribeJob? Keep the naming consistent frameworks
TODO TrashFilterProxyModel Remove? Does not seem to be used anywhere frameworks

Features

Features can mostly be implemented already now in master, but they are something I'd totally love to see in Frameworks.

Status Action Comment
TO DO Use ImapStreamParser in client jobs Will improve performance and memory consumption on client side. Also it's easier to use :) <{{{3}}}>
TO DO (IF POSSIBLE) Share external payload parts helper Concentrate external parts support into one place for easier maintanence and less code duplication <{{{3}}}>
TO DO Direct streaming of external payload parts For parts bigger than 4KB, the server can instead of asking client to provide the data send back a file descriptor that client will use to write the data directly to the file. Server can either verify that sizes match, or rely on client library telling the truth. Memory--, performance++ <{{{3}}}>
TO DO Require explicit call to ItemSync::deliveryDone() Clients should explicitly call ItemSync::deliveryDone() after the sync is over, instead of ItemSync being "clever" and detecting it on it's own. See https://git.reviewboard.kde.org/r/117673/ <{{{3}}}>

Server

Status Action Comment
TO DO Fix coding style We need to follow the Frameworks coding style <{{{3}}}>
TO DO Upstream QSQLITE3 driver I managed to figure out what the actual changes we have in our fork are. They are not implemented optimally, a proper implementation will be more complicated, but generally should ensure better performance. Then we can try to upstream those changes and get rid of the driver for good. <{{{3}}}>
TO DO Remove Nepomuk support We have Baloo now, we don't need Nepomuk anymore <{{{3}}}>
TO DO Remove SELECT command See comment at CollectionSelectJob in the table above <{{{3}}}>
TO DO Remove X-AKLIST and X-AKLSUB commands They are aliases to LIST and LSUB commands from the old days <{{{3}}}>
TO DO Remove EXPUNGE command Not used. We now delete items right away. <{{{3}}}>
TO DO Remove backward-compatibility with older protocol versions Some of the handlers still have several paths for supporting older client libraries. We don't even know whether they still work, so just get rid of them. <{{{3}}}>
TO DO Merge NotificationMessages Merge NotificationMessageV2 and V3 to NotificationMessage <{{{3}}}>
TO DO Remove legacy client support from NotificationManager Don't use the broadcast notify() signal anymore, only support notification subscribers. <{{{3}}}>
TO DO Refactor DB access Try to do as much DB interaction as possible in DataStore, so that it can be easily replaced by a FakeDataStore for testing purposes (see "Testable Handlers" below). <{{{3}}}>
TO DO Document ASAP Properly document all ASAP commands and responses. Ideally should be part of doxygen documentation rather than a plaintext file. <{{{3}}}>
TO DO Remove exception.h Share Akonadi::Exception from client libs. <{{{3}}}>
DONE Optimize CacheCleaner (1.12.0) Right now the CacheCleaner requests all collections every minute and checks their cache policies. Instead we should use some form of scheduling queue.
TO DO Remove LSUB and all subscription related API Subscribtions are obsoleted by Christian's work on LIST filtering <{{{3}}}>
TO DO Remove agentserver It seems that Agent Server was only used on Maemo 5, otherwise it's disabled at build time. Since we don't really support this anymore, let's kill it. <{{{3}}}>
TO DO Fix StorageDebugger DBus iface The sequence number (first argument) in queryExecuted() signal should be qint64, not double. <{{{3}}}>
TO DO Cache and share Collection statistics To reduce the amount of queries we do whenever collection stats change, we should cache those values and only invalidate the cache when they change. <{{{3}}}>

Other Plans

Testable Handlers

Does not really depend on Frameworks, can be done at any time. I want to have it done in Frameworks though. Currently we have no way to test Handlers on the server. This would require having a fake DataStore, fake NotificationCollector, fake AkonadiConnection and somehow fake backend for Entities. However, as a result we should be able to send various commands to the handlers and observe whether correct data are stored in database and correct signals are emitted. Big problem with external payload parts, would require a fake ExternalPartManager too.

Other option is to start a server instance, send it commands and listen to notifications and check what's in the database, but that's slow, requires lot of code and most importantly it's very difficult to debug. The approach outlined above would be much better.

Shared protocol helper

In a perfect world, we would have Handlers in a shared library and the Handler would be able to parse the command into entities and vice versa. But since the server is using different entities than the client libraries (Akonadi::Server::Collection vs. Akonadi::Collection), this probably won't be possible. However we still should try to share code for handling ImapSets, parsing attributes, payload parts, etc. The first step toward this is to have client jobs use ImapStreamParser.