Goals/Input/Triggers and Actions

Triggers

Common Functionality

Multi-touch, edge, stroke and scroll gestures are often used as global shortcuts - executing an action regardless of the active app/window.

All listed gestures should support triggering only in a particular context, to enable another layer of actions or to avoid getting in the way of a particular app. We anticipate three kinds of context:

• Modifier keys: Only activate the gesture if a given modifier is held down.
• Window matching: Only activate the gesture if the focused window matches a set of rules.
• Effect context: Only activate the gesture if it was registered by an active Desktop Effect. For instance, the Overview effect after being activated will allow swiping up further into its Grid View and from there swipe up once more to return to the regular desktop.

For a given gesture, the one with the most specific context gets activated and all other variants (of the same gesture type and direction/fingers/etc. parameters) are ignored.

For example, the transition gesture inside the active Overview effect will shadow both the 4-finger swipe-up gesture "Cycle through Window Overview and Desktop Grid" and the 4-finger swipe-down gesture "Cycle through Desktop Grid and Window Overview", both of which are otherwise registered as desktop-wide gestures for 4-finger swipe-up and swipe-down.

If there are no context matches for any of the gesture candidate configurations, no gesture gets activated.

In the following sections:

(*) not currently supported by KWin internally

(**) optional addition for later; not in the scope of the NLnet gesture project

Touchpad Gestures

Multi-touch Gestures

• Swipe - 3 and 4 fingers:
  • up
  • down
  • left
  • right
  • up left (*) (**)
  • up right (*) (**)
  • down left (*) (**)
  • down right (*) (**)
• Pinch - 3 and 4 fingers:
  • expand (a.k.a. spread)
  • contract (a.k.a. pinch, squeeze)

macOS describes 3-finger gestures "Pinch with thumb and two fingers", 4-finger gestures like "thumb and three fingers".

• Rotate - 3 and 4 fingers:
  • clockwise (*)
  • counter-clockwise (*)
• Hold - 3 and 4 fingers:
  • minimum seconds: N (*)
  • potentially infinite number of extra gestures depending on number of seconds held. (Longest duration determines which gesture is triggered.) Might be reasonable though to restrict the possible durations to a fixed range, such as max. 5 seconds, as anything above will be impractical to use.

It should be said that macOS provides an option to allow 3-finger drag to move windows on the desktop. Libinput offers a similar option (yet to be exposed through KWin/Plasma) for 3-finger drag emulating button-down while the pointer is moved. Both of these are free-form drags, we don't consider them as gestures/triggers in the context of this document.

Rather, 3-finger drag should be viewed as an alternative to gestures, in particular they conflict with 3-finger swipe gestures. Therefore, while 3-finger drag is enabled in Touchpad settings, 3-finger swipe gestures would not be recognized as such.

• Swipe, Pinch, Rotate, Hold - 2 fingers:
  • generally reserved for applications, unassigned by default
  • if we offer assignment of 2-finger gestures to users, we should encourage constraining applicability through window rules and/or Meta modifier requirement

Touchscreen Gestures

Multi-touch Gestures

Same as Touchpad multi-touch gestures, but independently configurable.

Edge Gestures

• from top
• from bottom
• from left
• from right

KWin may support edge corner gestures in the future:

• from top left (*) (**)
• from top right (*) (**)
• from bottom right (*) (**)
• from bottom left (*) (**)

Samsung phones have recently introduced an alternative, splitting the top edge in the middle. This method would result in the following (likewise 8) possible swipe gestures:

• from top edge, left half (*) (**)
• from top edge, right half (*) (**)
• from bottom edge, left half (*) (**)
• from bottom edge, right half (*) (**)
• from left edge, upper half (*) (**)
• from left edge, lower half (*) (**)
• from right edge, upper half (*) (**)
• from right edge, lower half (*) (**)

This may turn out a more useful division than corner gestures on many device form factors.

Pointer Gestures

Screen Edge Approach (Hot Corners)

• toward top edge
• toward bottom edge
• toward left edge
• toward right edge
• toward top left corner
• toward top right corner
• toward bottom left corner
• toward bottom right corner

Stroke Gestures

Arbitrary stroke paths. We may offer pre-defined paths such as:

• 8 directions straight
• 4 directions there and back
• L-shaped (plus 7 rotations and 1 mirror each)
• circle clockwise / counter-clockwise

We should support these options:

• Gesture activation button: for mice, ideally per device
  • Left mouse button is not available as activation button
• Button-less stroke activation: Start stroke recognition after pressing and releasing the gesture activation button, or following another trigger (e.g. touch hold gesture).
  • Parameter: start and stop time-outs

Scroll Gestures

A.k.a. wheel gestures? But wheels are only available on mouse devices, and perhaps it would be useful to have these on touchpads as well. KWin internally calls it axis shortcuts, which may be too technical of a term for end-user UIs.

Scroll gestures come in two forms:

• Gesture activation button + scroll (+ optional modifier keys). Only available on mouse devices. (*) (**)
• Modifier keys + scroll. Also available on touchpads. (**)

Other trigger types

The Shortcuts KCM (kcm_keys) already allows assigning key combinations to actions. This should arguably be considered as trigger type similar to the ones listed above.

In the future, other inputs may also be desirable as possible triggers:

• Game controller input (e.g. button combos) (*) (**)
• Voice commands (*) (**)
• App-defined triggers? (*) (**)
  • but keep an eye out for desktop-wide key injection malware opportunities

Actions

Desktop Actions

These are provided by the desktop environment itself, via Plasma components such as KWin, plasmashell or PowerDevil.

Desktop Actions with Gesture Integration Requirement

If assigned to a one-to-one capable gesture, these should enable a one-to-one animation. Can also be assigned as one-shot action, if the gesture does not support one-to-one.

• Change virtual desktops (KWin). Currently:
  • 3- and 4-finger swipe right/left (horizontal)
  • 3-finger swipe up/down (vertical)
• Show Overview (KWin). Currently:
  • 4-finger swipe up
  • 4-finger swipe down - context: only when showing Desktop Grid
• Show Desktop Grid (KWin). Currently:
  • 4-finger swipe down
  • 4-finger swipe up - context: only when showing Overview
• Desktop Zoom (accessibility feature) (KWin). Currently:
  • 3-finger pinch expand
  • 3-finger pinch contract

Likely candidates:

• Window management (KWin) - could also be a separate mode (Effect) unto itself
  • Maximize Window
  • Minimize Window
  • Close Window
  • Switch to next/previous window
  • Move window
  • Resize window - expand/shrink/maximize horizontally, vertically, free-form?
• Peek at Desktop (KWin)
• Volume control (Plasma)
  • Interesting case: needs progress callback, but progress can be coarser than KWin's pixel-perfect animation effects.
• Brightness control (PowerDevil/KWin)
  • Also with distinct values in progress callback.

One-shot Actions

Triggered once a certain distance threshold is passed. No content animation before gesture is triggered. This includes all actions that can be user-defined without writing a KWin plugin.

Likely candidates:

• Start button-less stroke gesture
• Mute/Unmute
• Lock screen
• Show Application Launcher
• Show KRunner (or should this be a one-to-one capable action?)
• Show Activity Manager (ditto?)
• Tile window up/down/left/right
• Enter Window Management mode (see "Desktop Actions with gesture integration requirement")

When assigned to gestures with progress callback, Plasma/KWin could preview the action by gradually fading in its icon and text properties, then flipping to an opaque version with "active" color scheme once the gesture activation threshold has been hit. Compare e.g. the 2-finger swipe left gesture in Firefox visualizing its assigned "Back" action.

App-specific Shortcut Actions

Global Shortcuts

Advertised by the app to the desktop, via one of: XDG Global Shortcuts portal, org.kde.KGlobalAccel D-Bus interface, or .desktop file with pre-defined shortcuts (which execute a corresponding command).

Can be invoked by sending the global shortcut command to the relevant application.

Common Actions

May be defined by the desktop/platform. Recognized locally by the app itself, not always in accordance with platform conventions. E.g. Back, Forward, End, New Tab, and more.

Can be invoked by sending the common keyboard shortcut to the active app window (assuming it conforms to conventions).

Other Local Shortcuts

Generally defined by the app with no expectation of universal support in other apps. E.g. "Open Private Window", "Add Node to Line", "Group Objects", or such. The desktop/platform currently has no knowledge of these shortcuts. In order to invoke these actions, the commonly used strategy is to define a user-defined keyboard shortcut which the application supports for this action.

In the last phase of our project, we intend to prototype a solution for the app to advertise its local shortcuts. If successful, these actions can be shown to the user and configured with a user-friendly action label.

User-defined Keyboard Input

All one-shot actions. Can be used for any app-specific shortcut actions or random keyboard emulation.

E.g. "Ctrl+N", "Meta+Shift+T". To be entered through the user's keyboard as in example when assigning. Consecutive sequences of key combinations should also be supported.

KHotKeys used to be able to send the shortcut to one of:

• Active Window: essentially to invoke an existing shortcut through other means, to the currently focused window
• Specific Window: regardless of which window currently has focus, the shortcut is always sent to a matching window. Essentially to create a custom Global Shortcut for a particular app (with crude window filter UI similar to KWin rules)

"Active window" is the obvious default, we should consider supporting "Specific Window" as well in one way or another. Note that "Specific Window" is different from window matching in triggers, because in this case the gesture will always trigger, merely the window target for the emulated keyboard events is changed.

As a special case of emulated key input, it might make sense to provide a different configuration UI and config storage for emulating text snippet input.

Run Command

All one-shot actions.

Very popular. Can also be configured and assigned to a keyboard shortcut in the existing Shortcuts settings page (kcm_keys).

Open URL

All one-shot actions.

Kind of a special case of "Run Command", except with xdg-open as command and the URL as argument?

Invoke D-Bus Method

All one-shot actions.

KHotKeys used to call this "D-Bus Command" in the past. Another special case of "Run Command".