Interface the database search engine to an AI-based LLM

Summary

DigiKam’s Advanced Search is powerful, but translating a natural-language request into the right combination of structured filters can be difficult for many users. This project will add a natural-language input to digiKam’s Advanced Search so users can describe queries such as “landscape photos with red labels taken in Paris last summer,” which will then be converted into editable search criteria such as tags, labels, dates, and locations. The implementation will use a lightweight local language model, structured intent parsing, capability-based resolution into supported digiKam search fields, ambiguity handling for unclear terms, and caching for repeated queries, while reusing digiKam’s existing model-management infrastructure. The deliverables include the Advanced Search UI integration, the local inference and parsing pipeline, criteria resolution and validation, safe clarification behavior, and end-to-end integration with digiKam’s existing Advanced Search workflow.

Project Proposal

Blog Posts

The list of all the blog posts for GSoC'26 has been listed here.

Merge Request

The merge requests for this project are:

Timeline

Week 1 - Week 2

Initial merge of the natural-language search pipeline (with a mock back-end and pipeline working end-to-end) into the project branch. Built the full pipeline (SearchQueryEngine, parser, resolver, dictionary, cache, backends, mock backend, model manager).

Pipeline:

User types a query
SearchQueryEngine checks SearchQueryCache first (if it's a hit, return right away)
SearchPromptBuilder builds the prompt (system rules + JSON schema + hints about the user's collection)
SearchLanguageBackend runs it (mock backend for now, llama.cpp later, runs off the GUI thread so the app doesn't freeze)
SearchIntentParser pulls out the JSON and checks every field/operator against a whitelist
SearchIntentResolver and SearchCapabilityDictionary work together to resolve aliases and values
One of three things happens:
- signalIntentReady fills in Advanced Search using Search XML
- signalClarificationRequired asks the user something (e.g. "best" is too vague)
- signalErrorOccurred explains what went wrong, leaves the dialog alone

Week 3

This week I worked on wiring the natural-language search model into digiKam's existing model-management infrastructure (DNNModelManager + FilesDownloader) instead of the placeholder that SearchNlModelManager previously used. It follows the approach: reuse the download half of the existing system, keep the runtime (llama.cpp) separate. The model is not yet downloadable end-to-end: the dnnmodels.conf entry carries placeholder SHA256 and FileSize values until the GGUF is hosted on the KDE infrastructure.

Changes

New: DNNModelNaturalLanguage (download-only model wrapper)

A minimal DNNModelBase subclass. loadModel() only calls the inherited checkFilename() (verifies the file exists and matches the expected size) and marks the model loaded - it performs no OpenCV loading, exactly like DNNModelConfig. Download metadata (getDownloadInformation()) and path resolution (getModelPath()) are inherited unchanged from DNNModelBase.

Registration wiring (DNNModelManager)

Added DNNLoaderNaturalLanguage to DNNLoaderType and DNNUsageNaturalLanguageSearch to DNNModelUsage. dnnmodelmanager.h: added { "naturallanguage", DNNLoaderNaturalLanguage } to the str2loader map so the .conf LoaderType resolves. dnnmodelmanager.cpp: parse natural_language_search in the Usage block; added the DNNLoaderNaturalLanguage case to the construction switch; included the new header.

Model declaration (dnnmodels.conf)

Added a [Qwen2.5-1.5B-Instruct] section with Usage=natural_language_search and LoaderType=NaturalLanguage. SHA256 and FileSize are placeholders pending hosting.

SearchNlModelManager rewrite

Replaced the private path logic (which used its own nlsearch/ directory and a hardcoded asset name) with resolution via DNNModelManager::instance()->getModel(...)->getModelPath(), so the model lives in the shared model directory and is fetched by the central downloader. ensureModelAvailable() no longer pretends to download; it signals that the model is missing so the UI can direct the user to the central downloader (FilesDownloader is the setup dialog, not a silent fetch).

How this was tested: Because the GGUF is not yet hosted, end-to-end download and inference cannot be exercised. The registration and path-resolution logic were verified locally as follows.

Week 4

Added the opt-in checkbox + SystemSettings flag so the NL model appears in the downloader like the other features, and verified the live download + SHA256 against the hosted file.

Week 5 - Week 6

I worked on implementing the real LLM inference backend for natural language search that wires it into the pipeline.

SearchLlamaBackend: loads the quantized Qwen2.5 GGUF and runs greedy decoding through llama.cpp on a dedicated worker thread (slotDoLoad / slotDoInference / slotDoUnload), with an early stop once a balanced JSON object is produced. All llama_* calls stay off the GUI thread. Pipeline wiring: SearchWindow selects the llama backend when built with ENABLE_NLSEARCH_LLAMACPP and the model is installed, and falls back to the mock backend otherwise. Build integration: llama.cpp is built in-tree via add_subdirectory (library-only, CPU-only). This required: HAVE_LLAMACPP config plumbing; keeping the bundled target out of the digikamgui export set (linked via $<TARGET_FILE:> + add_dependencies); excluding thirdparty/ from the recursive HEADER_DIRECTORIES sweep (it was leaking llama headers into unrelated targets); and forcing optimization on llama/ggml in Debug builds (without it, inference is ~30× slower).

Accuracy fixes (found during live testing): The parser now handles JSON numeric/bool values (a numeric rating was previously dropped); the prompt enforces start..end date ranges, injects the current date so relative dates resolve correctly, and more strongly discourages guessing.

Tests: Parser regression tests for the numeric-value and date-range fixes, alongside the existing mock-path pipeline tests.

Verified working end-to-end: Real queries (e.g. "photos from 2023 rated 5 stars", "red label photos rated at least 3 stars") populate the Advanced Search and return the correct photos.

llama.cpp is vendored as full source here, which is not the intended final form. The original plan was a git submodule, but KDE's git server rejects .gitmodules at the pre-receive audit, so I switched to vendoring.

Week 7

Worked on validation, error handling, and unit-test coverage for the pipeline. Hardened the parser and resolver against malformed or out-of-scope model output, and expanded the test suite so the safe-failure behaviour is locked down.

Changes

Parser/resolver: stricter validation so an invented field, an unsupported operator, or a malformed constraint is rejected rather than executed. The model never gets the last word; every field is whitelist-checked before it can reach the search.
Tests: added pipeline tests covering the three safe-failure paths: ambiguous query asks for clarification, unsupported field is rejected, and nonsense maps to nothing, alongside the existing mock-path tests. Also added a real-inference test (gated on an environment variable pointing at a local model, so CI without the GGUF skips it rather than hardcoding a path).

Verified: The pipeline emits exactly one of intent-ready, clarification, or error for every class of input, and never silently guesses on out-of-scope requests.

Week 8 - Week 9

Most of the time was spent benchmarking small local models to settle the model choice with data, and documenting the fine-tuning decision.

Benchmark harness (committed under core/tests/llm/): A standalone Python harness that runs around 40 hand-labelled queries through a model using the exact prompt digiKam sends (transcribed from SearchPromptBuilder, raw prompt with no chat-template wrapping, matching the C++ backend), and scores structured-output accuracy at the parsed-intent level, alongside median latency and peak memory. Ships with the dataset, a runner script (run_benchmark.sh), a README, and a results write-up (RESULTS.md).

Three-way comparison:

Model	Constraint accuracy	Median latency	Peak RAM
TinyLlama-1.1B	18%	~6.4s	~1.3 GB
Qwen2.5-1.5B (chosen)	85%	~2.3s	~2.0 GB
Qwen2.5-3B	79%	~29s	~3.5 GB

TinyLlama is far too inaccurate (malformed JSON, invented fields, schema echo) and, because it rambles, actually slower than the 1.5B model. Qwen2.5-3B scored lower than the 1.5B on core constraints and was ~13× slower (~29s median), so bigger is not better here. Qwen2.5-1.5B is the sweet spot on accuracy and latency together.

Fine-tuning decision: The residual 1.5B errors (orientation, date structure) are the same in the 3B model, so they are a prompt/vocabulary matter already handled by the capability dictionary, not a capacity problem. A LoRA fine-tune would add training/packaging/maintenance overhead for no measurable benefit, so it was documented as not warranted rather than pursued. This closed the conditional the proposal left open for this stage.

Week 10

This week I worked on extending the natural-language support to video and file-format properties.

Changes

New fields mapped end-to-end: video duration, video frame rate, video audio bitrate, and file format. Each follows the established four-layer recipe: prompt schema entry and example, parser whitelist, capability-dictionary aliases, and a writeConstraintToXml branch.
Video properties are emitted as two-value Interval ranges to match what the database query builder requires (duration in seconds, frame rate as doubles, bitrate as integers); file format is matched as an upper-case choice string (JPEG, PNG, TIFF, RAW, ...).
Parser fix: the "between" operator was previously restricted to date ranges and rating, which silently rejected valid video-range queries. Extended the operator validation to allow the video range fields.
Tests: added parser tests covering a videoduration range and a format value, guarding the new field support (in particular the operator-validation fix) against regression.

Verified working end-to-end: queries such as "videos longer than 5 minutes", "videos between 1 and 3 minutes", "videos over 30fps", and "RAW files" populate the corresponding Advanced Search fields and return the correct items.

Week 11

Worked on user documentation and review polish.

Documentation (digikam-doc): added a Natural Language Search section to the Search View chapter (how to use it, the supported properties, the local/offline nature of the feature, translation, and ambiguity handling), and documented the optional local Large Language Model in the "Artificial Intelligence Files Download" section of the quick-start guide and in the "Download Required Binary Data" setting; noted as a separate, optional download using a different engine (llama.cpp) from the OpenCV deep-learning models.

Benchmark MR polish: moved the benchmark into core/tests/llm/ (alongside the existing engine benchmarks) and added the standard digiKam copyright and SPDX license headers to the scripts, per review.

Week 12

Created a Merge Request for the project branch to master for the 9.2.0 packaging and beta testing.