KTp/Tasks/KopeteLogs

Integrating Kopete Logs

The problem/backstory

We have logs in Kopete, we have our own logs. We can't "throw away" the old data.

There is a log program provided telepathy-logger in 0.2 this had an XML file backend, in 0.3 this is moving to MySQLLite.

Telepathy-logger also loads Pidgin logs by a plugin that reads from pidgin files as well as the telepathy logs.

Options

Nepomuk all the things

There are several ways to do this:

Importing from TpLogger

We write something that logs Tp Chats to nepomuk, and imports telepathy stuff to nepomuk.

Problems: We don't know when to import things from TpLogger, unless we constantly just poll it. Logs are duplicated in two places.

Discussion

drdanz: see Zeitgeist option

d_ed: Possible option: We write an observer, that grabs everything from telepathy-logger whenever a channel disconnects. However this is very hacky and messy.

d_ed: I would say unless anyone can solve the issue of "when do you take stuff from tp-logger" we have to rule this out.

With our own logger

We write something that logs Tp Chats to nepomuk, and imports telepathy stuff to nepomuk.

Instead of relying on TpLogger we make our own logger as a nepomuk service text observer.

Problems: If you disable nepomuk at any point you won't get logs for that period. Also if you have tp-logger as well logs are stored in two places completely independantly.

Discussion

drdanz: This is in my opinion not a good option, that requires a lot of effort. I don't see any advantage in logging directly in Nepomuk with a concurrent logger.

d_ed: It may be more effort, but it is a nice clean solution, and if you don't want two logs. Uninstall telepathy-logger.

vHanda: I like this option. It's the same deal as meta contacts. It you don't use Nepomuk, you don't get meta contacts and logs :)

As a storage backend for TpLogger

Problems: I have literally no idea how to do that. I believe it would have to be in Tp-logger.

Discussion

drdanz: This is worthless if Kopete is going to be abandoned.

d_ed: I think you've muddled this with the other later idea. I meant we write a nepomuk backend for tp-logger.

Akonadi all the things

From my understanding of Akonadi this is a far more sensible place for logs.

There is already code for importing Kopete logs AND Logs from Telepathy-logger (old bindings). George G mentored this project and seems to think there's something wrong with it.

It still has all the variants as above, and with all the same advantages/disadvantages.

Additionally if we have all this akonadi feeding to nepomuk which is currently being done everything is stored in /3/ places.

Either nepomuk or this could be merged with other people's code for pulling logs from other places (such as GTalk logs from GMail (Vishesh has sample code for nepomuk) or Facebook messages (which Akonadi supports)

Discussion

Dave: The situation with what goes in nepomuk and akonadi is a complete arbitrary mess based purely on what the developer prefers rather than any actual decision about the data. Which means we have additional data going into both places. This whole idea of copying from akonadi to do neopmuk is completely retarded (/political). You can do a full text search in MySQL. One of them needs to just die. </unhelpful rant>

drdanz Since it's just a mess, I'd just go for having only one dependency between nepomuk and akonadi. And since we will be using the person library based on nepomuk I'd rather discard this option

vHanda: Even I don't like this option. Specially cause it introduces a hard dependency between Akonadi and Telepathy. Considering that people dislike both Akonadi and Nepomuk, do really want to be dependent on both of them? And I agree with Dave. It a messy messy situation.

Use Zeitgeist

I don't know much about Zeitgeist, but at the FOSDEM, Seif suggested to use Zeitgeist for the logging stuff. I just had a chat with him on #zeitgeist:

[11:29:53] <drdanz> seiflotfy1: ping
[11:30:03] <seiflotfy1> hi drdanz
[11:30:08] <drdanz> hi!
[11:30:34] <drdanz> At FOSDEM you spoke in favour of handling telepathy logging with zeitgeist
[11:30:47] <drdanz> but I don't remember exactly what you suggested
[11:31:37] <drdanz> Can you explain me what you already log (about telepathy events) with zeitgeist?
[11:32:19] <drdanz> seiflotfy1: ^
[11:34:13] <seiflotfy1> drdanz: we have a process the listens to telepathy-logger events such as "conversation started/endend" or "msg sent/received" etc...
[11:34:19] <seiflotfy1> and puts them into zeitgeist
[11:34:38] <seiflotfy1> we want to make this hapen from tp-logger itsefl and not via an outside process
[11:37:03] <drdanz> So, we are trying to decide how to get logger data into nepomuk, but at the same time we want to avoid redundancy and data duplication
[11:38:51] <drdanz> But if I understand it correctly zeitgeist doesn't know anything about the conversation, it just knows the "events"
[11:38:56] <seiflotfy1> drdanz: we will finish the zeitgeist extension that forwards stuff to zeitgeist
[11:39:05] <seiflotfy1> drdanz: exactly we dont know
[11:39:14] <seiflotfy1> sorry
[11:39:26] <seiflotfy1> there is a zeitgeist extension that will forward stuff to nepmuk
[11:39:34] <seiflotfy1> hey maybe u are interested in porting it with us
[11:39:35] <seiflotfy1> ?
[11:39:57] <seiflotfy1> so anythign that zeitgeist logs needs to be translated to nuao
[11:40:06] <seiflotfy1> also i think zeitgeist needs to be there for nepomuk
[11:40:23] <seiflotfy1> what events are blacklisted or not
[11:40:31] <seiflotfy1> and how do you attach a location to an event
[11:40:40] <seiflotfy1> with zeitgeist as a proxy that evaluates we can do it for you
[11:40:46] <seiflotfy1> and then store it into nepomuk
[11:45:03] <drdanz> sorry I'm not sure if I understood correctly, because I didn't use zeitgeist (yet! :D ).
[11:45:03] <drdanz> Let me try to summarize: TpLogger logs the events, Zeitgeist gets all the events from TpLogger makes some filtering and pushes the required events into Nepomuk?
[11:47:12] <drdanz> But what kind of data are you going to push in nepomuk? Is, for example, the text of the message logged?
[11:48:03] <seiflotfy1> drdanz: no we dont log the msg but we could tell nepomuk what it is
[11:50:15] <drdanz> seiflotfy1: ok, that's interesting
[11:51:07] <seiflotfy1> still we need first to port the extension that pushes into nepomuk
[11:51:09] <seiflotfy1> we had it in python
[11:51:16] <seiflotfy1> we need it in vala
[11:53:20] <drdanz> ok, I see...
[11:53:54] <drdanz> I will discuss it with the other kde-telepathy guys, perhaps someone knows vala and can help... ;)
[11:54:19] <seiflotfy1> drdanz: tdfischer can help you guys out
[11:54:28] <seiflotfy1> he wrote the extension and is a zeitgeist dev
[11:54:51] <seiflotfy1> drdanz: zeitgeist supports a blacklist extension
[11:55:05] <seiflotfy1> where you can tell zeitgeist what types of events or subject not to log
[11:55:23] <seiflotfy1> so this way if you use zeitgeist as a proxy nepomuk will also be subject to the blacklist
[11:55:30] <seiflotfy1> which is ver good for privacy
[11:57:02] <drdanz> I see... That's interesting! Anyway the data will be logged by tplogger as well
[11:58:29] <drdanz> Anyway I'm still trying to figure out if we should duplicate the message itself (that will be both in tplogger log file and in nepomuk)

Discussion

drdanz: This could solve the problem of the continuous polling. We still have the problem that the content of the logged message is duplicated

mck182: It's not that much of a problem because when Nepomuk indexes files, it basically duplicates its contents. So you already have a serious duplication in your system if you're using Nepomuk. So this is imho not really a big issue.

Merge inside TpLogger

TpLogger has a set of plugins for fetching data from various sources. We make one for kopete.

This is how Pidgin logs are loaded.

Problems: code would have to go in tp-logger. We'd need to write it in glib. I don't know glib.

Merge at the Ktp-log-viewer level

Instead of grabbing the two sets of data and merging them at the data level it could be done at a view level. In KTp-log-viewer we load data from Kopete, and we load data from TpLogger. Job done.

Discussion

Dave: This or above are by far my favourite. It's simple. Job done. We don't touch the data, we've less chance to break it. However we can't do any fancy merging with additional data sources, but does that actually exist?

drdanz: The problem with this is that we will have to maintain forever a part of the code that just reads kopete logs, and imho that sucks.

Write a kopete-tplogger conversion tool

We could just write a simple tool that reads files in kopete format and writes files in TpLogger 0,2 format, and we just import the history in TpLogger once.

Discussion

drdanz: At the moment we are using tp-logger, so until we decide how to import the logs in nepomuk/akonadi or whatever we could just write a simple conversion script, it shouldn't be too hard to do. I don't think that they will abandon the xml format, they will either support both formats at the same time or import the xml in the sqlite db.

Dave: Good idea. There's no public API for doing this, so we would have to just randomly create files in some other app's data store which is a bit naughty, but it would work easily.

vHanda: You could also directly push it into Nepomuk ( if you choose to go with Nepomuk, which I totally think you should, but them I'm biased )

Dave: Visesh, the issue is that kopete logs and our logs need to end up in the same place. Pushing Kopete logs into nepomuk is easy peasy, it's a set of files that won't ever change again. Telepathy logs are not, and is the first 3 options on this page.

Screw Kopete Users

Simplest option. Don't import, live with the hate, 2 years later they'll forget.