GSoC/2021/StatusReports/LucasWang

From KDE Community Wiki
< GSoC‎ | 2021‎ | StatusReports

KDE Connect iOS <> Status Report & Reflections

Key Links, Commits, People, and Articles

Links:

Key Commits:

People:

Posts, Docs, and Articles:

How It Started:

As of April 2021, the iOS KDE Connect application possesses several issues:

Outdated UI design and features such as optimizing for newer aspect ratios, icons, fonts, and accessibility, etc.

The app is lacking many core functionalities of KDE Connect that other currently available versions have. Since Apple has implemented many of those features into iOS since 2014, the app would now be able to use them to offer the missing functionalities.

The backend portion of the app that governs LAN network communications is currently functioning inconsistently and only under a very specific series of actions.

The goal of this project is to address the problems listed above with an "analysis and rewrite" process of much of the app’s current code base with the newest Apple development language and UI framework, Swift and SwiftUI. Aside from solving the immediate problems with the current app, rewriting the foundation in Swift and SwiftUI would give the codebase a strong footing to expand and adapt to future contributions. SwiftUI’s cross-platform deployment capabilities also give the possibilities for forks of the codebase for other Apple platforms such as macOS and watchOS.

As such, the initial proposal, finalized in early April, outlined the following goals and deliverables:

  • A completely revamped interface of the KDE Connect iOS application written in SwiftUI and adhering to the newest iOS design language.
  • Additional functions were added to the new app to match the core functionalities present on other versions of KDE Connect such as the Android version.
  • A revised network communications backend.
  • Documentation of the unit testing done throughout the project and, using the Android KDE Connect as a reference, document the state and progress of the iOS version.

How It's Going: Key Periods & Moments

Note: Some of these events overlap with each other, and they are also not in strictly sequential order.

May - June 7th: I Discussed and re-checked the project proposal with mentors (timeline, other factors such as project resources, etc.). I joined KDE's Apple Developer Group to assess the situation as KDE Connect iOS 2021 is likely KDE's first project to potentially reach the App Store Connect stage. I also raised the question and potential problem of accessibility to Apple's platforms and frameworks needed to develop and contribute to this project and floated the idea of future consideration of the MacStadium FOSS program which provides FOSS projects with a free Cloud Mac Mini. Having dealt with issues like this in software clubs in school before, I did not want the lack of accessibility to Apple's platforms to be an obstacle for potential contributors.

June 7th - 14th: GSoC's coding period started just as I wrapped up my university spring quarter and was drowning in final projects and exams. Per previous revision with mentors, the coding period for this project will be starting roughly a week later around June 14th.

June 14th - July 7th: The prototype of the UI/UX built completely in SwiftUI was ready for presentation at the upcoming monthly KDE Connect Development meeting. Other components of the app were also re-written in Swift, such as UserDefaults for persistent storage, cross-app data flow, etc.

July 3rd: It was discovered that the huge, breaking change between iOS 12 (which my mentor Inoki was running) and iOS 14 (which I was running) was responsible for the confusing and seemingly unexplainable difficulties that I was having getting UDP broadcast to work. iOS 14 requires the developer to apply for an entitlement to use broadcasting capabilities. I began looking into the complex Apple Developer group structure as the application would require coordination with the KDE Admins.

July 13th: The revamped UI and the Network testing app were presented to the KDE Connect developer group at the monthly meeting. The project was met with positive feedback from the group. KDE Admins were also present so I was able to inform them of the upcoming coordination required to apply for the broadcast entitlement from Apple.

July 1st - July 20th: The prototype of the LAN communication backend built entirely using the native Swift Network framework is functional. The tester app has an intractable selector for changing modes (UDP/TCP) and is capable of sending and receiving both messages as well as sending files via the standard system file picker and receiving files by saving them into the app's document directory to and from any IP address (including broadcasting) and using any port. I found the tester at this point to be quite similar to the command line tools of Netcat and started using it with Netcat on my Linux and Android devices to transfer files and links.

July 27th: Further testing with a backend revision in Swift with the native Network framework was halted by this thread on the Apple Dev Forums. The lack of ability to start TLS on an existing TCP connection without spawning a new one is a dealbreaker for the compatibility of KDE Connect iOS with the Qt and Android versions. Maybe the Network framework can still be used in the future for a new file transfer backend? But even so, we might have spent weeks exploring a framework that we might not end up using. Priorities now are to try to bridge the 2014 Objective-C backend and see what comes out of it.

July 28th - August 11th: After putting aside the work that we've already done regarding the Swift Network framework, we started focusing entirely on getting the existing Objective-C backend from 2014 to work and started integrating it with the new Swift codebase. We managed to get pairing working and optimized as well as to lay down a solid foundation for future work to be done for the Plugins. File transferring is working as well (but could be optimized further). There's still some additional work that needs to be done to the LAN backend, namely the storing and verification of certificates.

August 12th - August 20th: Despite having a solid foundation already in place, we're running out of time to implement the more complex Plugins such as remote mouse control and MPRIS, as well as to implement certificates to be stored in keychains. Realistically, (due to my schedule and other events) August 20th is the last day that I would have to close up GSoC. I've spent most of the weekdays discussing and reflecting on the project with my mentors and getting them in words into blog posts and documentation. I'm going to pre-configure the 3 blog posts to be published over the next 2 days while I'm away.

Key Progress Point: Revamped UI/UX in SwiftUI

Blog post: Opening thoughts and a brief summary of the plans for the KDE Connect iOS project.

A new UI, written entirely in SwiftUI, is a major goal of the project as it would not only modernize the interface’s appearance and functionality but also place the project and new codebase on a solid footing for any future contributions and developments.

SwifUI (introduced in 2019) comes to mind as a strong candidate for building UI that adheres closely to the Cupertino language and also one that provides quite a few nice and (mostly) straightforward abstractions for factors such as implementing accessibility features (voice lines for voiceover, etc.) and its declarative approach (similar to that of Flutter’s) allows for relatively clean code with single sources-of-truth, automatic re-render of views when designated variables change, and involves very little interactions with the ViewControllers present in UIKit and Storyboard.

In addition to adhering to the Cupertino design language, KDE Connect iOS also aims to provide an experience and interactive process that is as close to the Android version as possible by implementing Material design elements into their “equivalents” in Cupertino.

Comparison of the Device List View, current KDE Connect Android on the left, new UI/UX of KDE Connect iOS on the right.
Comparison of the Device Details View, current KDE Connect Android on the left, new UI/UX of KDE Connect iOS on the right.

Key Progress Point: Getting set-up on the Apple Developer Platform & Applying for the Multicast Entitlement

Relevant blog post: Overview and thoughts on the revamped architecture of KDE Connect.

After getting added to the KDE e.V. Apple Developer Group in late May, I spent a fair amount of time exploring and getting used to the Apple Developer Group portal. KDE's portal was mostly empty, with only a certificate for signing the macOS version of Krita, a different KDE project built with Qt. It seems like KDE Connect iOS would likely be the first KDE project built with Apple's native frameworks that makes it to the App Store Connect stage. Therefore, an important goal is to understand the hoops of the Apple Developer portal and its related processes to get the app registered, ready for testing, and eventually onto the App Store. Aside from searching for and reading a considerable amount of documentation, one of the most important factors was the application for the entitlement which enables LAN broadcasting capabilities, which we completed with the help of KDE e.V. Admin Aleix Pol.

Multicast capability enabled.

Key Progress Point: Bridging of the 2014 Backend with the new Swift Components

Relevant blog post: Overview and thoughts on the revamped architecture of KDE Connect.

The programming language(s) of the project is certainly interesting, containing perhaps an equal mix of Swift and Objective-C, two completely different languages in both appearance and styles. Objective-C, the language that KDE Connect iOS 2014 was written entirely in, dates back to the NeXT days before Steve Jobs returned to Apple in the 1990s. Swift, on the other hand, was introduced from scratch in 2014 to serve as the “flagship” programming language for Apple native development going forward. SwiftUI, the youngest of them all, was introduced in 2019 as the new UI/UX framework with a declarative syntax (similar to that of Flutter’s), a ViewController-less architecture, single sources of truth with binding variables, and more. Therefore it is important that we somehow get these seemingly very different frameworks working together to produce the final KDE Connect iOS 2021.

To achieve this, a considerable amount of bridging (more than I had anticipated when I first started the project) was done between the Objective-C and Swift codebase. The general gist of the structure is that: most Objective-C classes and methods can be directly called from Swift (though for some reason Xcode seems to randomly rename the names of classes and methods and provide them in a suggested fix), while Swift classes and methods need to conform to @objc and then be automatically pulled into an Xcode-generated header file for importing into Objective-C code.

The final product of the bridging retained the file structure of the original project, but with Swift and Objective-C replaced and inserted wherever needed. The file structure can be roughly categorized into 6 sections:

LAN backend section: consisting of BaseLink.h/m, LanLink.h/m, BaseLinkProvider.h/m, LanLinkProvider.h/m. Responsible for establishing and managing LAN connections, including sending and receiving of data, handling sockets and connections, loading and verifying TLS identities, etc.

Network package section: consisting of NetworkPackage.h/m. Responsible for defining, serializing, and deserializing Network packets, the fundamental communication blocks of KDE Connect.

Device section: consisting of Device.h/m. Responsible for defining device objects. These are the objects that represent any remote devices that the host has interacted with, including their information, status, and any functionalities that they might have.

App background function: consisting of BackgroundService.h/m. Responsible for defining and executing functionalities that are required somewhere else in the app, such as refreshing device lists, saving remembered devices, taking the appropriate actions when device status changes (pairing, rejected, etc.).

Plugins section: consisting of PluginService.swift and the various .swift plugin files/classes. These classes are responsible for handling individual plugin functions, much like how plugins are handled in the Qt and Android versions of KDE Connect.

Views/UI section: consisting of the various .swift files, written in the SwiftUI frontend framework, to define the visible components

A brief highlight and comparison of the file structures of KDE Connect 2014 KDE Connect 2021.
The pairing process is a heavy mix between the Objective-C backend and Swift functions + UI.

Key Progress Point: Revamping the Plugins in Swift

Relevant blog post: Overview and thoughts on the revamped architecture of KDE Connect.

Similar to the KDE Connect Qt and Android versions, the goal of the Plugins is to tap into device-specific and OS-specific functionalities to send and respond to other devices over the LAN backend. As such, the Plugins need to be structured in a highly modular way that allows for easy changes and additions to be made, which makes it easy to tweak and add more Plugins in future contributions. The Plugins also need to be preferably written in a modern framework that bonds well with iOS’s tendency to introduce potential breaking changes every once in a while.

As such, most Plugins as well as the Plugin service provider that ties BackgroundService with the Plugins were completely re-written in Swift. There’s some tweaking still left to be done and several plugins, namely Remote Mouse Control, Notifications, Media Control, Slideshow Remote, etc. are not yet implemented. Some of these such as Remote Mouse Control is a definite possibility that will have to be worked into the app, while others, such as relaying notifications from the iOS device to other connected KDE Connect devices, is likely impossible currently as iOS provides no system API for apps to listen in on other app’s notifications. But the rewrite should leave the project in a suitable state for future contributions.

A straightforward example of a Plugin revamped in Swift, the Ping Plugin.

Reflections: Obstacles and Detours

Blog post: Reflection on GSoC 2021, what I think could be improved and what I think went well.

Throughout the project, there were a number of detours (work we did that ultimately didn’t make it into the final product of KDE Connect iOS 2021 for GSoC) that caused delays in the timeline of the project.

One of the biggest detours was the decision in July to explore the Swift native Network framework to see how it can be integrated into the backend. A Network testing app with Netcat-like functionalities was made entirely in Swift and SwiftUI as a result of this discovery. However, the discovery towards the end of July more or less ended as the lack of ability to start TLS on an existing TCP connection left it unsuitable for KDE Connect (Perhaps it could be used for file transfer in the future, but that’s not making it into GSoC). Other unexpected time-consuming actions encountered were applying for the entitlement from Apple, which required digging through Apple’s seemingly non-existent documentation on the issue and coordination with KDE e.V. President Aleix Pol (thanks Aleix!), and the fact that kdesrc-build was causing some trouble with the custom-compiled version of KDE Connect.

Though, I hope that in the long term, these difficulties and delays won't be too much of a problem as the project carries forward.

The Network Tester App, built using the Swift native Network framework that ultimately didn't make it into KDE Connect iOS 2021 for GSoC.

Going Forward: Current Project Status and Future Plans

Blog post: Plans for the future of KDE Connect iOS after GSoC 2021: continued development, outreach and accessibility.

Since many crucial parts of the app, e.g the UI/UX, Plugins, etc. were revamped in Swift, this also opens up many possibilities for other interesting additions to the project. The Catalyst framework, which allows cross-compatibility between iOS and macOS (both Apple Silicon and x86) is only a checkbox (and perhaps some UI and backend tweaking) away. And adding Widgets for iOS >=14 and perhaps even a watchOS companion app is also on the table. SwiftUI’s accessibility features are also very straightforward e.g being able to configure lines for the iOS screen reader with a few lines of code. The future potential and possibility of KDE Connect iOS seem broad and wide, and hopefully, we would be able to get to at least some of them.

Retrieved from "https://community.kde.org/index.php?title=GSoC/2021/StatusReports/LucasWang&oldid=98023"