GSoC/2020/StatusReports/LeonardoEmanuelSegovia

Dynamic Fill Layers in Krita using SeExpr

💙 Energy #krita #seexpr (source: David Revoy on Framapiaf)

Abstract

Layers are one of the core concepts of digital painting. They allow artists to control different parts of their artwork at once, for instance, color, lighting, lineart, as well as texture. A key feature of them is their ability to be resized, composited, renamed, grouped or deleted independently of the rest of the document.

Patterns and textures are also essential components of an artist’s toolbox, allowing them to represent the intricacies of a physical material. They come in two forms: bitmap textures, which are images contained in e.g. PNG or OpenEXR files, or procedural textures, which are generated on the fly using their mathematical representation.

KDE’s Krita painting suite supports using patterns and textures through two types of layers, File or Fill Layers. However, neither of them let artists create dynamically generated content: File Layers are inherently static, and Fill Layers support only color fills (like Paint Layers) or basic pattern rendering.

The goal of this project is to let artists create dynamic content through a new, scriptable Fill Layer. To this effect, I integrated Disney Animation’s SeExpr expression language into Krita.

Components

This project is divided into the following objectives or goals:

• SeExpr-based Fill Layer generator
• Creation and management of SeExpr scripts
• Integration tests
• User documentation

Side goals included:

• Feature support status across all our platforms
• Multithread support for Fill Layers
• ARM support for AppImages (including this project’s deliverables as well)

SeExpr-based Fill Layer generator

Enabling SeExpr as a Fill Layer generator involves two stages.

• Integration of SeExpr as a 3rdparty dependency
  • Commits: 1, 2, 3, 4
• Defining the new generator
• Loading, rendering, and saving the script with the layer
  • This commit implements both items.

The SeExpr generator is divided into three components:

• SeExprExpressionContext and SeExprVariable are the glue between SeExpr and Krita; the former contains the script’s execution context, the latter exposes the necessary variables for rendering the script (width, height, and pixel location in normalized coordinates).
• KisSeExprGenerator is the instance of KisGenerator tasked with rendering the script to the layer’s paint device. It sets up the SeExprExpressionContext with the given script and variables. Then, it iterates over the paint device, rendering the script on each pixel and converting the resulting color back to Krita’s color space.
• KritaSeExprGenerator, WdgSeExpr and WdgSeExprSavePreset comprise the UI of the SeExpr generator. These are described in more detail in the next section.

It must be noted that throughout the development of the SeExpr generator, extensive changes were made to Disney’s work itself. Of particular note are the following:

• Warnings and deprecations were solved across all operating systems I worked on.
  • Commits: 1, 2, 3, 4, 5, 6, 7, 8, 9
• The provided UI widgets were completely restyled for better accessibility.
  • Commits: 1, 2, 3, 4, 5, 6, 7, 8, 9
• Unused or irrelevant items were hidden behind flags to further optimize compilation.
  • Commits: 1, 2
• The ExprEditor text widget was decoupled from ExprControlCollection (the manipulation widgets list). This enabled a complete migration to Krita’s .ui widget system.
• ExprTextEditor now signals when a change is made to the script text field.
• A consistency change was made in the naming of variables.
• By request from Agata Cacko, variable insertion was modified to insert them into a new line, and comment were better differentiated from numbers.

The following bugs in SeExpr were fixed:

• As detailed on these blog posts, SeExpr implicitly assumed that it would be always executed in English-based locales. This was detected by Wolthera van Hövell.
  • Commits: 1, 2, 3
• The autocompletion support was unusable since it blocked numpad entry. This was fixed (commits: 1, 2), along with an extra optimization to use native controls.

• A buffer underflow was fixed in CCurveScene (the UI widget for color curves) (commits: 1)
• As part of the revision of the documentation, Wolthera van Hövell found a couple of bugs related to incorrect function documentation (commits: [1](https://invent.kde.org/lsegovia/seexpr/-/commit/b807bfebfec9edc4197ca3561f53625895edc96e))
• A crash when initializing vector variables outside the [0, 1] range (commits: 1, 2)
• Lack of promotion of vectors to colors after widget initialization (commits: 1)

Creation and management of SeExpr scripts

• 

  Example render by David Revoy.

• 

  SeExpr presets management UI.

• 

  Example render by David Revoy.

When I implemented the generator, the script was saved along with the layer. David Revoy proposed that scripts should be reusable, by making them bundleable as any other resource in Krita.

The initial implementation, shown in the screenshots below, is based on the existing Pattern chooser widget. The underlying KoResource specialization, which is called KisSeExprScript and located in libs/flake/resources, is an almost-straight port of Anna Medonosová’s Gamut Masks.

• 

• 

Data-wise, a SeExpr preset’s script is stored in a plaintext file named script.se, along with a thumbnail image in preview.png. These are bundled together in a ZIP of extension .kse and MIME type application/x-krita-seexpr-script.

In T13337, I requested feedback from the developers on the UX for creating and editing these presets. Based on the existing workflow for managing brushes, I proposed the following mocks:

• 

• 

Based on suggestions from Agata Cacko in both the Krita Artists feedback thread and the Phabricator task, the final form of the UX is as follows:

• 

• 

• 

By request from Agata, the manipulation widgets and the text editor were placed inside a QSplitter, to ensure users could freely choose how much space they want to assign each component.

By request from Wolthera, support was added for reporting validation errors. Since SeExpr was not designed with internationalization in mind, extensive refactoring was needed to extracts its strings from the UI components and from the render library, allow their future localization by the KDE team, and finally enable their usage in the widgets. Additionally, variables exposed from Krita were added to the autocompletion system.

• 

• 

While there were not many changes in the management UX, it was simplified into 4 actions. The unique contribution here is Render Script to Thumbnail; as its name says, it creates an instance of the SeExpr fill layer generator, points it to the thumbnail image’s QPaintDevice, and renders the given preset’s script into the thumbnail.

Integration tests

There are two testable points in the SeExpr generator:

• successful rendering from the given script
• loading and saving from a given KoResource instance

The first is already implemented in the SeExpr generator (commits: 1, 2). The second is NYI. The first covers the SeExpr generator itself (commits: 1, 2, 3). The second covers serialization and deserialization from a known working KoResource (commits: 1).

User documentation

• 

  SeExpr Fill Layer entry.

• 

  SeExpr introductory tutorial.

• 

  SeExpr quick reference manual.

• 

  SeExpr resource entry.

Up to now, there was no complete reference for creating SeExpr scripts. To build a complete reference for SeExpr scripts, my work created the following documental resources in the Krita user manual:

• an entry to the Fill Layers section
• a specific page for SeExpr scripts in Resource Management
• a tutorial for creating, editing and bundling script presets
• a Quick Reference page detailing the expression language

The latter is one of the most important productions, as it involved porting the existing User Documentation of Disney's (which is only available in precompiled Doxygen form) to reStructuredText, while adapting it to suit the script specification in Krita.

A fifth resource, this one provided with the Krita binaries, is a resource bundle comprising all examples in the Krita Artists thread, “Procedural texture generator (example and wishes)”. The authors of the examples are Wolthera van Hövell and David Revoy; the latter employed an alpha version of this tool in the episode 33 of his webcomic, Pepper&Carrot. (commits: 1, 2, 3, 4)

Multithread support for Fill Layers

(A demo video is available at diode.zone.)

This is a stretch goal.

Originally, Fill Layers were rendered in a single, background thread. Although feasible for simple generators, it wastes SeExpr’s potential for parallelization. To maximize the usability and performance of this kind of layers, Dmitry Kazakov suggested that Fill Layers themselves should be converted to a Stroke. Strokes are individual, self-contained sets of jobs, each of which may be performed in parallel.

To achieve this, the canvas is tiled according to the optimal patch size at the moment; for each tile, a job structure is generated that will run the generator over the specified tile. These structures are then submitted to a stroke that runs them in the background.

This feature requires that the generator perform its work on a self-contained basis, i.e. the tile. For those that are unable to do so (e.g. Multipattern), this feature is disabled at compile time.

Live Preview for Fill Layers

This is a stretch goal.

Before this project, Fill Layers (unlike Filter Layers) were only able to be previewed after being added to the canvas. However, a direct update step was not possible; the multithreaded rendering process implemented above makes use of Krita’s strokes system, and two strokes (layer addition, and layer update) cannot run concurrently. What’s more, all functions that update the graphical state of the layer are booby-trapped: any call to them will in turn trigger an update, the initialization of which breaks the internal rendering state.

In order to enable this functionality, I completely reworked Fill Layers’s update workflow:

1. I ported the rendering strategy to a runnable-based one, which allowed me to move the actual work to self-contained lambda functions (commits: 1, 2)
2. I defanged all functions dealing with internal state, which allows them to be called without triggering yet another update (commits: 1, 2, 3, 4)
3. I decoupled the job generation from the canvas update, allowing them to be inserted into the stroke that adds the layer. Additionally, this step was compressed to trigger only once every 100ms, saving a lot of CPU in unnecessary render jobs (commits: 1, 2, 3)
4. Finally, I ensured that Fill Layers were completely re-rendered if the image bounds change, as some generators (like SeExpr) operate over the whole image (commits: 1, 2)

In addition to this goal, I completed the move of all documentation to the User Manual (commits: 1, 2).

ARM support for AppImages

This goal is explained in more detail on its blog post.

• 

  The SeExpr AppImage running on a Raspberry Pi 3B+.

• 

  The SeExpr AppImage running on Libre Computer's Le Potato with 2GB RAM.

It was asked on the #krita IRC channel if it was possible to run Krita on ARM-based computers, specifically the Raspberry Pi 3B+. To the best of my knowledge, it has not been tried before.

In the blog post linked above, I prove fully-featured AppImages can be built for ARM-based platforms. Although the original target was 32-bit ARM (also known as armhf and armv7l), my main testing board is a 64-bit, 2GB Le Potato, which forced me to test this process for the aarch64 architecture as well.

The build process was done on a 16-thread Ryzen 7 under QEMU.

The key results were as follows:

• The KDE Sysadmin’s Docker image works almost out-of-the-box.
  • The base image must be changed to one that has QEMU, for instance multiarch/ubuntu-debootstrap.
  • If this path is followed, the universe repository must be enabled manually.
  • xkbcommon-dev is a key dependency to build Qt’s X11Extras module. It’s missing in aarch64 because libegl1-mesa-dev does not bring it in, unless in every other platform.
  • There are no official binaries for CMake, patchelf, and linuxdeployqt, so they must be built from scratch and bundled into the image.
• The build process must be multithreaded as much as possible.
  • The 3rdparty dependencies ext_expat, ext_python, ext_sip, and ext_qt must be forcibly parallelized using the -j${SUBMAKE_JOBS} flag.
• These dependencies need fixes or be disabled altogether:
  • OpenColorIO (ext_ocio) needs this patch applied
  • libx265 inside ext_heif needs to be upgraded to 3.4 to allow aarch64 builds. It works for armhf with -DENABLE_ASSEMBLY=false
  • Python will hang when running its tests, the BUILD_COMMAND must be changed to make build_all.
  • GSL needs the shared library in Unix-based systems, not only Android
  • Vc does not work altogether
• The build scripts assume across the whole process that the platform is x86_64. This must be changed taking into account that arm64 is called aarch64 by AppImageTool.

Status

• SeExpr-based Fill Layer generator: merged
• Creation and management of SeExpr scripts: merged
• Integration tests: merged
• User documentation: merged
• Feature support status review: merged
• Multithread support for Fill Layers: merged
• Live Preview for Fill Layers: merged
• ARM support for AppImages: merged

Communications

• Official entry for this project at Google Summer of Code
• Blog posts (category: GSoC):
  • May 6th: "Hello Planet”
  • May 9th–May 13th: “Developing Krita in Visual Studio Code” (part 1, part 2, part 3)
  • May 19th: “What is SeExpr about?”
  • May 28th: “Status report: Community Bonding”
  • June 1st: “Status report: Week 1”
  • June 4th: “Status update: Linux”
  • June 8th: “First alpha release of my project: looking for feedback!”
  • June 22th: “Second alpha release of my project”
  • July 3rd: “Third alpha release of my project”
  • July 15th: “Compiling Krita for ARM: An AppImage Tale”
  • July 27th: “Status update: merged!”
  • August 8th: “SeExpr status update!”
• Krita Artists threads:
  • “First alpha of my GSoC project: procedural texture generator”
  • “Procedural texture generator (example and wishes)”
• Project report at the KDE Community Wiki
• Portfolio entry at my blog

Deliverables

• Phabricator task: T13097, see in particular subtask T13337 for the preset management UI
• Krita Artists threads: “First alpha of my GSoC project: procedural texture generator”, “Procedural texture generator (example and wishes)”
• Fixes and improvements to SeExpr are available at its clone on KDE Invent
• Merge requests for the SeExpr generator: !380, !411
• Merge request for multithreading Fill Layers: !412
• Merge requests for Fill Layers preview: !467, !472
• Merge request for the Krita manual: !145, !151
• Feature support list is available as a Google Docs file