Baloo/Architecture: Difference between revisions

Latest revision as of 03:00, 16 November 2023

Baloo is a metadata and search framework by KDE.

Baloo focuses on providing a very small memory footprint along with with extremely fast searching. It also supports storing additional file based metadata via extended attributes.

Mailing List: [email protected]
IRC Channel: #kde-baloo on freenode

Architecture

Outdated architecture
The architecture evolved and Baloo is only a file indexer. The reference to PIM resources (emails, contacts) is historical. KDE PIM uses its own indexing system, which has a similar architecture.

Baloo focuses on decentralization of data and does not have any central database. At its core it is a set of 3 services:

Data Stores
Search Stores
Relations

Baloo replaced Nepomuk, a more elaborate semantic search system and ontology. There may still be information about the older project in techbase::Projects/Nepomuk.

Data Store

A Data Store is a place to permanently store data. Its format and APIs are not dictated by Baloo and can be in any way that is most efficient for the data in question.

Search Store

A Search Store is a plugin which provides search capabilities for a specific kind of data. Each of these stores provide a plugin which can be used to search through the data using a consistent API. At this moment there are 3 search stores that have been implemented -

File Search
Email Search
Contact Search

Each Store can also provide custom APIs for searching the data.

Relations

Baloo can be used to create relations between two uniquely identifiable identifiers. The key aspect is that when creating a relation between A and B, both of them are uniquely identifiable. For most of the PIM data, Akonadi already provides a unique identifier of the 'akonadi:?item=x'. For File data, Baloo provides unique identifiers of the form 'file:x'.

Each relation is stored in an independent store in the desired mechanism. A sample sqlite implementation is provided which maps both the identifiers in a 2 column sqlite database.

For example, there exists a TagRelation which maps a Tag and any unique identifier. Similarly, there can be an ActivityRelation, and relations for other aspects such as mapping a file received via Bluetooth to a Bluetooth device.

Implementation

Files

Baloo concentrates heavily on files and provides both a data store and search store for files. These are stored using a combination of sqlite and xapian.

Akonadi

In the case of PIM, since Akonadi already stores (caches) the data, only a Search Store is required. There are currently separate search stores for contacts and emails. They are both implemented using Xapian.

Baloo and extended attributes

See the Baloo page on the Userbase wiki

@@ Line 9: / Line 9: @@
 {{Remember|2=Outdated architecture|1=The architecture evolved and Baloo is '''only''' a file indexer. The reference to PIM resources (emails, contacts) is historical. KDE PIM uses its own indexing system, which has a similar architecture.}}
-Baloo focuses on decentralization of data, and does not have any central database. At its core it is a set of 3 services:
+Baloo focuses on decentralization of data and does not have any central database. At its core it is a set of 3 services:
 * Data Stores
 * Search Stores
 * Relations
+Baloo replaced Nepomuk, a more elaborate semantic search system and ontology. There may still be information about the older project in [[techbase::Projects/Nepomuk]].
 === Data Store ===
-A Data Store is a place to permanently store data. Its format and APIs are not dictated by Baloo, and can be in any way that is most efficient for the data in question.
+A Data Store is a place to permanently store data. Its format and APIs are not dictated by Baloo and can be in any way that is most efficient for the data in question.
 === Search Store ===
@@ Line 36: / Line 38: @@
 == Implementation ==
 === Files ===
-Baloo concentrates heavily on files, and provides both a data store and search store for files. These are stored using a combination of sqlite and xapian.
+Baloo concentrates heavily on files and provides both a data store and search store for files. These are stored using a combination of sqlite and xapian.
 === Akonadi ===
@@ Line 43: / Line 45: @@
 === Tags ===
 Tags are stored in a custom format. In order to relate tags with files / email, a TagRelation is provided.
-Baloo replaced [[Nepomuk]], see https://community.kde.org/Baloo .
 === Baloo and extended attributes ===
-Applications such as [[Dolphin]] store tags and ratings in xattrs (extended attributes) in the filesystem, and Baloo indexes these so you can search them.
+:''[https://userbase.kde.org/Baloo#Baloo_and_extended_attributes See the Baloo page on the Userbase wiki]
-On Linux, to show them as entered in Dolphin, in a terminal enter:
-<kbd>getfattr ''/path/to/file''</kbd>
-assuming the necessary package ("xattr" in Debian) is installed.
-Some filesystems (e.g. FAT32, the filesystem with which most USB flash drives are formatted) do not support xattrs at all, so copying files to such a volume will lose the tags.
-Most CLI utilities (e.g. cp, tar, rsync) won't copy xattrs without additional flags either, so unless your environment is 'All KDE all the time' and you only use Linux filesystems like ext4, XFS or Btrfs (this is not a complete list), and you modify your backups to specifically include xattrs, your tags may get trashed one way or another.
-Nepomuk (the previous KDE tagging/file metadata system) used to store tags in a separate database, but even after many years of development it could hardly be made to work performantly or reliably so it was replaced with Baloo.
-In some ways using xattrs is 'the right place' to put this tagging info, but Linux doesn't have a 'core API' that would allow a 'xattrs preserved by default, and interpreted consistently' policy. Therefore there are a lot of ways to lose your tags. KDE has been trying to make this work for many years, but it is  difficult given the way UNIX/POSIX has evolved.