GSoC/2020/StatusReports/KartikRamesh
Digikam : Face Management Workflow Improvements
DigiKam is a KDE Desktop Application for Photo Management. Apart from the standard functionality of being able to view photos, DigiKam provides the user with a lot of added features such as Image Tagging, Photo Editing, Image Metadata viewing/editing. At the heart of DigiKam's commendable functionality is the FaceEngine. DigiKam can detect faces in Photos, and recognize faces in new photos based on prior information. This allows for a great personalized experience for the user.
A major breakthrough in the FaceEngine came last year when Thanh Trung Dinh implemented OpenCV's DNN module to bring great improvements to performance. Igor Antropov implemented many changes to the workflow Interface, to make the overall experience much comfortable for the user.
This project is in essence an extension to the work that Igor did last summer. As such, this project does not intend to implement one Major feature.Instead, it aims to rectify issues in the current workflow, as well as introduce new features in an effort to improve the user experience.
Mentors : Gilles Caulier, Maik Qualmann, Thanh Trung Dinh
Important Links
Project Proposal
DigiKam Face Engine Workflow Improvements
GitLab development branch
Project Goals
This project aims to :
- Provide a Help Box to aid first time users of Facial Recognition.
- Provide notification about results of a Facial Recognition.
- Order People Sidebar, to show tags of Priority first.
- Order Face Item View, to display Unconfirmed Faces before Confirmed Faces.
- Provide new “Ignored” Category for Face Tags.
- Automatically Group Results in Unconfirmed Tag.
- Provide Functionality to reject Face Suggestions.
- Automatically add Icons to newly created face tags.
Work Report
Week 1 : May 11 to May 18
NOTE: Due to the current global situation, there's some uncertainty regarding when my college final term exams will be scheduled. After having discussed this with the mentors, we've decided that starting early with the project is the best course of action.
The first issue I intend to tackle is that of Rejecting Face Suggestions in DigiKam. Face Suggestions are a key part of the Facial Recognition process, and allow the User to categorize their album according to People Identities, while training the Facial Recognition algorithm.
The user interacts with these Facial Rejections by means of the Assign Name Overlay. This overlay appears on hovering over the Face Suggestion and allows the User to confirm (✅) or reject(⛔) the suggestion.
However, in the present version there's really no way to "Reject" a Face Suggestion. Pressing the ⛔ does exactly what pressing ✖ does, it deletes the Face Region from the Database.
This is not ideal. The ⛔ button should technically do the opposite of what ✅ does. It should be the user's way of telling the Facial Recognition Algorithm that it's incorrect.
Present Scenario
- Facial Recognition outputs incorrect suggestion.
- User intuitively presses ⛔, hoping the algorithm realizes the mistake.
- Instead the Face is deleted from the Database.
- To recover the Face, the User re-runs Face Detection and Face Recognition.
- Since the Algorithm was not provided any inputs, it repeats the Incorrect Suggestion.
Desired Scenario
- Facial Recognition outputs incorrect suggestion.
- User intuitively presses ⛔, hoping the algorithm realizes the mistake.
- Face gets moved to "Unknown" Tag.
- If user re-runs Face Recognition immediately, in all likelihood the mistake will be repeated, as it will perform again on identical conditions.
- If user provides some input, in the form of assigning Faces to other photos of the same person, then the mistake will be reconciled.
Implementation
The Confirmation/Rejection happens through an interplay between 3 classes.
- AssignNameWidget : The actual widget which controls the ✅, ⛔ buttons, LineEdit etc.
- AssignNameOverlay : The overlay which controls how it's drawn when user hovers over a face etc. It includes the AssignNameWidget as a member variable.
- DigiKamItemView : Responsible for the main view of DigiKam.
The current workflow is as follows:
- The user presses on the ⛔ button.
- AssignNameWidget sends a reject signal to AssignNameOverlay.
- AssignNameOverlay sends a reject signal to DigiKamItemView.
- DigiKamItemView calls appropriate methods of FaceTagsEditor to delete this face.
The issue is that the Rejection Overlay (✖ Button) is also connect to the same DigiKamItemView::removeFaces(), hence the identical functionality.
I started off with connecting the Reject Signal of AssignNameOverlay to a new function DigiKamItemView::rejectFaces(). The implementation for this function came to be after a discussion with the developers on the mailing list.
In essence, I'm treating Rejection of Face Suggestion as a change of Tag from the current Unconfirmed tag, to an Unknown Tag. To do this I've implemented FaceTagsEditor::changeTags().
Results
Here's how Rejection of Face Suggestions now works:
First I proceed to Reject the Face Suggestion. Notice how this leads to an increment of the count of Unknown Faces, and the Face does show up in the Unknown Category. Hence the Face is not being deleted!
Next, I demonstrate the deletion of face using the (✖) button. This does not change the Unknown Counter, as the Face has been deleted.
Related Links
Week 2 : May 18 to May 25
This week I focused on implementing Ignored Tags in DigiKam. This category is one that has been requested by users multiple times.
DigiKam will often detect and then try to recognize faces in photos that the user perhaps doesn’t recognize himself, or doesn't wish to identify. Such faces will continue to reappear during Face Detection and Recognition, and the user has to keep rejecting the suggestions (using the implementation of last week) every time.
In the past, users have solved this problem by creating a placeholder tag, and assigning all such faces to that tag. Instead, it would be a much greater solution, if DigiKam provided such a placeholder tag. That way we may implement customized logic and functionality for this tag. I've chosen to call this placeholder tag as Ignored.
With the implementation of this new feature, the user could just mark unnecessary faces as Ignored. Faces marked as Ignored will not be detected by the Face Detection process in the future, nor will they be considered during the recognition process.
Implementation Details
Logic behind Implementation and Current Status
Currently, marking of a Face as Ignored is only allowed for Unknown Faces. This implementation was considered after a conversation on the mailing list. This is based on the logic that Confirmed Faces include Faces that the User recognizes. Since they are confirmed, they won't reappear during the Face Detection/Recognition runs. Hence, the problem described in the introduction doesn't arise for Confirmed Faces.
It's important to understand the distinction between a Face and a Person. A person represents a unique identity, who may have several faces in the Photograph collection. The current implementation ignores faces and NOT persons.
For eg. consider a person Jack with 10 photos of him. If the user marks the 4th face of Jack (faces will be created from each photo during Detection) as Ignored, then during subsequent recognitions this Face will not be re-detected or recognized as any Person. However, the Algorithm may try to identify any of the other 9 faces.
The Ignored Implementation may be made Identity Specific, by modifying the underlying Face Recognition algorithm, to take into account similar of a current face (that is trying to be recognized), to Faces marked as Ignored. If high similarity is found, the face won't be recognized. However, this is something for the future, and not something I'll implement in the current project.
1. Allow marking of Faces as Ignored
The Unknown Faces overlay has been modified, so that the Reject Button ⛔ corresponds to marking the Face as Ignored.
This assignment does not clash against the functionality of Reject Suggestions last week, because Suggestions appear only on Unconfirmed Faces. Hence, pressing ⛔ on Unknown Faces as of last week did nothing.
2. Implement Ignored Tags Category in the Database
The next step is to ensure that pressing the ⛔ button on Unknown Faces actually leads to marking the face as Ignored. Instead of opting for a "cheap fix" and just confirming the Face as a Person tag with the name "Ignored", I opted to implement changes in the core level. This separation of Ignored Tag from the standard Confirmed Tag, will allow us to implement custom features for Ignored Tags in the future.
After the implementation, the categories now available to the user are:
- Confirmed Tag
- Unconfirmed Tag
- Unknown Tag
- Ignored Tag
Implementing Ignored Tag as a separate new category in the database was relatively easy as I could follow the blueprint laid out by Unconfirmed and Unknown Tags. Ignored Tag resembles these tags in many aspects. Here are some important commits related to this implementation:
One important thing to take care of is that ignoredPersonTagId() will create the Ignored Tag if one doesn't exist already. This implementation was opted for, to be in accordance with Unconfirmed and Unknown Tags. The issue with this is that it isn't ideal for the Ignored Tag to be automatically created, when the User doesn't want it to.
Since Ignored Tag is not a feature that will be required by most users, it's appropriate to only display it as and when the User requires it. This will help de-clutter the User Interface.
As such unlike Unconfirmed/Unknown Tags, the Ignored Tag isn't created on startup. Developers will also need to ensure that the tag doesn't get created when the user doesn't want to. For eg. if trying to implement a custom functionality for Ignored tag in a particular method (say sorting implementation of the People Sidebar), one would :
if (tagId == FaceTags::ignoredPersonTagId())
{
// custom implementation
}
However this would create the Ignored Tag if it didn't exist. To solve this I've provided FaceTags::existsIgnoredPerson() which may used to check whether an Ignored Tag has been created by the user.
3. Mark the Face as Ignored
Ignoring a Face can be treated as a Change of Tag, where the initially assigned Unknown Tag of a Face is changed to the Ignored Tag. Hence, I made changes to the implementations of changeTag that I had suggested last week. Here are some commits regarding the same:
With this implementation, Ignored persons are stored in the database as:
4. Listing Ignored Faces
Now this problem was something I struggled with for quite some time. Even though I was able to mark Faces as Ignored, I couldn't get them to show up in the Digikam Item View. I suspected this was because the Database did not understand how to display this new category. Indeed that was the case. The Faces that show up for a particular tag (when you click on a name) is activated by an ItemLister job, responsible for providing the ItemView with the ItemInfos for all faces that belong to the tag. All I had to do was add Ignored to the list of Properties that the lister considered. Here's the commit for the same.
5. Metadata support for Ignored Faces
Gilles suggested that it would be a good idea to store information about Ignored faces in the metadata of images. This way, if the user accidentally deleted the database, the information about Faces can be easily recovered. DigiKam provides the option to users to store information about Face Tags, all that needed to be done was plug a similar functionality for Ignored faces.
Here are my commits for this feature: Include Ignored Faces in MetaData
Week 3 : May 25 to June 1
This week I focused on Reordering the People Sidebar. The People Sidebar is an integral part of Face Workflow, and allows the Users to interact with People tags. Currently the People Sidebar is displayed as a plain list, which can make it really difficult for the Users to identify which Tags need attention. During Facial Recognition, an important part of the whole process is to confirm/reject suggestions that the Algorithm provides, so that all (or most) faces are identified. If the number of People Tags in the Database are huge, then this can be very troublesome, as the Tags with new suggestions appear mixed between the tags with no suggestions. The fact that all tags appear in the same font style is also not very helpful for the user.
Here's a video depicting the current status:
Notice how I have to search for the tags that require attention. I aim to improve this using 2 methods: 1. Display People Tags that need attention at the top of the List. 2. Display People Tags that need attention in a different Font Style (or weight).
Implementation Details
1. People Tags of Priority to appear in Bold
To implement this I added a new ItemRole to the Album Model, for Font style. This Role will return Bold for Tags that have Unconfirmed Faces, and Default otherwise. Here are my commits for this feature:
1. Add new ItemRole to AlbumModel
2. Implementation for fontRoleData, which provides a QFont based on the logic mentioned above.
2. People Tags of Priority to appear at the Top
The tags in People Sidebar are displayed using a TagModel, which is sorted using a TagPropertiesFilterModel. This class inherits from AlbumFilterModel, and I modified the lessThan() method of this class, which is responsible for comparing and sorting the Tags. The new sort rule first sorts Tags by the number of Unconfirmed Faces associated with it, and then alphabetically for all Tags that have no Unconfirmed Faces.
Another important part was to enable auto-sorting of the People Sidebar. We are implementing our sorting based on the count of Unconfirmed Faces in each tag, however this may change as the User interacts with each image (whether Confirm or Reject). Hence, I trigger an sorting of the view, whenever the user modifies a face.
Here are my commits regarding the same:
Modify AlbumFilterModel::lessThan()
Enable auto-sort of People Sidebar
Final Results
All said and done, here's how the new People Sidebar looks!
Even though this feature wasn't as tough to implement as the other ones, I'm positive that it would be really improve the User Experience.
About Me
Hello there!