Dynamic Fill Layers in Krita using SeExpr
Layers are one of the core concepts of digital painting. They allow artists to control different parts of their artwork at once, for instance, color, lighting, lineart, as well as texture. A key feature of them is their ability to be resized, composited, renamed, grouped or deleted independently of the rest of the document.
Patterns and textures are also essential components of an artist’s toolbox, allowing them to represent the intricacies of a physical material. They come in two forms: bitmap textures, which are images contained in e.g. PNG or OpenEXR files, or procedural textures, which are generated on the fly using their mathematical representation.
KDE’s Krita painting suite supports using patterns and textures through two types of layers, File or Fill Layers. However, neither of them let artists create dynamically generated content: File Layers are inherently static, and Fill Layers support only color fills (like Paint Layers) or basic pattern rendering.
The goal of this project is to let artists create dynamic content through a new, scriptable Fill Layer. To this effect, I integrated Disney Animation’s SeExpr expression language into Krita.
This project is divided into the following objectives or goals:
- SeExpr-based Fill Layer generator
- Creation and management of SeExpr scripts
- Integration tests
- User documentation
Side goals included:
- Feature support status across all our platforms
- Multithread support for Fill Layers
- ARM support for AppImages (including this project’s deliverables as well)
SeExpr-based Fill Layer generator
Enabling SeExpr as a Fill Layer generator involves two stages.
- Integration of SeExpr as a 3rdparty dependency
- Defining the new generator
- Loading, rendering, and saving the script with the layer
- This commit implements both items.
The SeExpr generator is divided into three components:
SeExprVariableare the glue between SeExpr and Krita; the former contains the script’s execution context, the latter exposes the necessary variables for rendering the script (width, height, and pixel location in normalized coordinates).
KisSeExprGeneratoris the instance of
KisGeneratortasked with rendering the script to the layer’s paint device. It sets up the
SeExprExpressionContextwith the given script and variables. Then, it iterates over the paint device, rendering the script on each pixel and converting the resulting color back to Krita’s color space.
WdgSeExprSavePresetcomprise the UI of the SeExpr generator. These are described in more detail in the next section.
It must be noted that throughout the development of the SeExpr generator, extensive changes were made to Disney’s work itself. Of particular note are the following:
- Warnings and deprecations were solved across all operating systems I worked on.
- The provided UI widgets were completely restyled for better accessibility.
- Unused or irrelevant items were hidden behind flags to further optimize compilation.
ExprEditortext widget was decoupled from
ExprControlCollection(the manipulation widgets list). This enabled a complete migration to Krita’s .ui widget system.
ExprTextEditornow signals when a change is made to the script text field.
- A consistency change was made in the naming of variables.
- By request from Agata Cacko, variable insertion was modified to insert them into a new line, and comment were better differentiated from numbers.
The following bugs in SeExpr were fixed:
- As detailed on these blog posts, SeExpr implicitly assumed that it would be always executed in English-based locales. This was detected by Wolthera van Hövell.
- The autocompletion support was unusable since it blocked numpad entry. This was fixed (commits: 1, 2), along with an extra optimization to use native controls.
- A buffer underflow was fixed in
CCurveScene(the UI widget for color curves) (commits: 1)
- As part of the revision of the documentation, Wolthera van Hövell found a couple of bugs related to incorrect function documentation (commits: 1)
- A crash when initializing vector variables outside the [0, 1] range (commits: 1, 2)
- Lack of promotion of vectors to colors after widget initialization (commits: 1)
Creation and management of SeExpr scripts
When I implemented the generator, the script was saved along with the layer. David Revoy proposed that scripts should be reusable, by making them bundleable as any other resource in Krita.
The initial implementation, shown in the screenshots below, is based on the existing Pattern chooser widget. The underlying
KoResource specialization, which is called
KisSeExprScript and located in
libs/flake/resources, is an almost-straight port of Anna Medonosová’s Gamut Masks.
Data-wise, a SeExpr preset’s script is stored in a plaintext file named
script.se, along with a thumbnail image in
preview.png. These are bundled together in a ZIP of extension
.kse and MIME type
In T13337, I requested feedback from the developers on the UX for creating and editing these presets. Based on the existing workflow for managing brushes, I proposed the following mocks:
Based on suggestions from Agata Cacko in both the Krita Artists feedback thread and the Phabricator task, the final form of the UX is as follows:
By request from Agata, the manipulation widgets and the text editor were placed inside a
QSplitter, to ensure users could freely choose how much space they want to assign each component.
By request from Wolthera, support was added for reporting validation errors. Since SeExpr was not designed with internationalization in mind, extensive refactoring was needed to extracts its strings from the UI components and from the render library, allow their future localization by the KDE team, and finally enable their usage in the widgets. Additionally, variables exposed from Krita were added to the autocompletion system.
While there were not many changes in the management UX, it was simplified into 4 actions. The unique contribution here is Render Script to Thumbnail; as its name says, it creates an instance of the SeExpr fill layer generator, points it to the thumbnail image’s
QPaintDevice, and renders the given preset’s script into the thumbnail.
There are two testable points in the SeExpr generator:
- successful rendering from the given script
- loading and saving from a given
The first is already implemented in the SeExpr generator (commits: 1, 2). The second is NYI.
The first covers the SeExpr generator itself (commits: 1, 2, 3).
The second covers serialization and deserialization from a known working
KoResource (commits: 1).
Up to now, there was no complete reference for creating SeExpr scripts. To build a complete reference for SeExpr scripts, my work created the following documental resources in the Krita user manual:
- an entry to the Fill Layers section
- a specific page for SeExpr scripts in Resource Management
- a tutorial for creating, editing and bundling script presets
- a Quick Reference page detailing the expression language
The latter is one of the most important productions, as it involved porting the existing User Documentation of Disney's (which is only available in precompiled Doxygen form) to reStructuredText, while adapting it to suit the script specification in Krita.
A fifth resource, this one provided with the Krita binaries, is a resource bundle comprising all examples in the Krita Artists thread, “Procedural texture generator (example and wishes)”. The authors of the examples are Wolthera van Hövell and David Revoy; the latter employed an alpha version of this tool in the episode 33 of his webcomic, Pepper&Carrot. (commits: 1, 2, 3, 4)
Multithread support for Fill Layers
This is a stretch goal.
Originally, Fill Layers were rendered in a single, background thread. Although feasible for simple generators, it wastes SeExpr’s potential for parallelization. To maximize the usability and performance of this kind of layers, Dmitry Kazakov suggested that Fill Layers themselves should be converted to a Stroke. Strokes are individual, self-contained sets of jobs, each of which may be performed in parallel.
To achieve this, the canvas is tiled according to the optimal patch size at the moment; for each tile, a job structure is generated that will run the generator over the specified tile. These structures are then submitted to a stroke that runs them in the background.
This feature requires that the generator perform its work on a self-contained basis, i.e. the tile. For those that are unable to do so (e.g. Multipattern), this feature is disabled at compile time.
Live Preview for Fill Layers
(A demo video is available at diode.zone.)
This is a stretch goal.
Before this project, Fill Layers (unlike Filter Layers) were only able to be previewed after being added to the canvas. However, a direct update step was not possible; the multithreaded rendering process implemented above makes use of Krita’s strokes system, and two strokes (layer addition, and layer update) cannot run concurrently. What’s more, all functions that update the graphical state of the layer are booby-trapped: any call to them will in turn trigger an update, the initialization of which breaks the internal rendering state.
In order to enable this functionality, I completely reworked Fill Layers’s update workflow:
- I ported the rendering strategy to a runnable-based one, which allowed me to move the actual work to self-contained lambda functions (commits: 1, 2)
- I defanged all functions dealing with internal state, which allows them to be called without triggering yet another update (commits: 1, 2, 3, 4)
- I decoupled the job generation from the canvas update, allowing them to be inserted into the stroke that adds the layer. Additionally, this step was compressed to trigger only once every 100ms, saving a lot of CPU in unnecessary render jobs (commits: 1, 2, 3)
- Finally, I ensured that Fill Layers were completely re-rendered if the image bounds change, as some generators (like SeExpr) operate over the whole image (commits: 1, 2)
ARM support for AppImages
This is a stretch goal, and is explained in more detail on its blog post.
It was asked on the #krita IRC channel if it was possible to run Krita on ARM-based computers, specifically the Raspberry Pi 3B+. To the best of my knowledge, it has not been tried before.
In the blog post linked above, I prove fully-featured AppImages can be built for ARM-based platforms. Although the original target was 32-bit ARM (also known as
armv7l), my main testing board is a 64-bit, 2GB Le Potato, which forced me to test this process for the
aarch64 architecture as well.
The build process was done on a 16-thread Ryzen 7 under QEMU.
The key results were as follows:
- The KDE Sysadmin’s Docker image works almost out-of-the-box.
- The base image must be changed to one that has QEMU, for instance
- If this path is followed, the
universerepository must be enabled manually.
xkbcommon-devis a key dependency to build Qt’s X11Extras module. It’s missing in
libegl1-mesa-devdoes not bring it in, unless in every other platform.
- There are no official binaries for CMake, patchelf, and linuxdeployqt, so they must be built from scratch and bundled into the image.
- The base image must be changed to one that has QEMU, for instance
- The build process must be multithreaded as much as possible.
- The 3rdparty dependencies
ext_qtmust be forcibly parallelized using the
- The 3rdparty dependencies
- These dependencies need fixes or be disabled altogether:
- OpenColorIO (
ext_ocio) needs this patch applied
- libx265 inside
ext_heifneeds to be upgraded to 3.4 to allow
aarch64builds. It works for
- Python will hang when running its tests, the
BUILD_COMMANDmust be changed to
- GSL needs the shared library in Unix-based systems, not only Android
- Vc does not work altogether
- OpenColorIO (
- The build scripts assume across the whole process that the platform is
x86_64. This must be changed taking into account that
- SeExpr-based Fill Layer generator: merged
- Creation and management of SeExpr scripts: merged
- Integration tests: merged
- User documentation: merged
- Feature support status review: merged
- Multithread support for Fill Layers: merged
- Live Preview for Fill Layers: merged
- ARM support for AppImages: merged
- Official entry for this project at Google Summer of Code
- Blog posts (category: GSoC):
- May 6th: "Hello Planet”
- May 9th–May 13th: “Developing Krita in Visual Studio Code” (part 1, part 2, part 3)
- May 19th: “What is SeExpr about?”
- May 28th: “Status report: Community Bonding”
- June 1st: “Status report: Week 1”
- June 4th: “Status update: Linux”
- June 8th: “First alpha release of my project: looking for feedback!”
- June 22th: “Second alpha release of my project”
- July 3rd: “Third alpha release of my project”
- July 15th: “Compiling Krita for ARM: An AppImage Tale”
- July 27th: “Status update: merged!”
- August 8th: “SeExpr status update!”
- Krita Artists threads:
- Project report at the KDE Community Wiki
- Portfolio entry at my blog
- Phabricator task: T13097, see in particular subtask T13337 for the preset management UI
- Krita Artists threads: “First alpha of my GSoC project: procedural texture generator”, “Procedural texture generator (example and wishes)”
- Fixes and improvements to SeExpr are available at my clone on KDE Invent
- Merge requests for the SeExpr generator: !380, !411
- Merge request for multithreading Fill Layers: !412
- Merge requests for Fill Layers preview: !467, !472
- Merge request for the Krita manual: !145, !151
- Feature support list is available as a Google Docs file