GSoc/2023/StatusReports/KaranjotSingh
MEASURING ENERGY CONSUMPTION USING REMOTE LAB
The Remote Eco Lab project aims to provide a streamlined process for measuring software energy consumption remotely using a CI/CD pipeline. By automating the measurement process and integrating with the OSCAR tool, developers can make informed decisions to improve code efficiency and work towards software eco-certification
Mentors
Volker Krause
Benson muite
Nicolas Fella
Merge Request
The official repository for the project can be found at: https://invent.kde.org/teams/eco/remote-eco-lab
Migration from Remote-Test Repository to Remote-eco-lab: https://invent.kde.org/teams/eco/remote-eco-lab/-/merge_requests/13
Due to the extensive need for Continuous Integration (CI) testing and CI-related modifications, the majority of the work has been carried out in this test repository: https://invent.kde.org/drquark/kecolabtestci
The majority of the alterations related to CI testing can be found detailed in: https://invent.kde.org/drquark/kecolabtestci/-/merge_requests/9
For accessing the most recent artifacts and the final report, please visit: https://invent.kde.org/teams/eco/remote-eco-lab/-/jobs/1157464/artifacts/browse
Blog Posts
Google Summer of Code : KEcolab
Timeline
Week 1 and Week 2
During the community bonding phase, we established a communication channel (accessible at https://matrix.to/#/#kde-eco-dev:kde.org) to encourage transparent collaboration and maintain a public record of our discussions. This channel also served as a means to engage additional individuals in the project.
Furthermore, we organized regular internal meetings, occurring weekly on Thursdays, to discuss project updates within our team. Additionally, we presented project progress to the wider community during our monthly meetups.
In the first and second weeks, with guidance from our mentors, we successfully set up the project infrastructure, which involved tasks such as configuring SSH keys, setting up Raspberry Pi1, and preparing the lab test PC. My specific contribution included configuring the X server on the lab test PC to enable the execution of graphical user interface (GUI) applications. During this timeframe, I started work on implementing the first stage ie installation of Flatpak as part of the Continuous Integration (CI) Pipeline.
I also authored a blog post to showcase our project to the broader KDE community.
Week 3 and Week 4
I successfully finished the implementation of the Flatpak installation phase. Over the course of these two weeks, I also began tackling the energy measurement aspect of the CI Pipeline. However, I encountered some challenges related to copying files (via scp) from the repository to the lab test PC and configuring the necessary environment variables. With the help from the mentors, I was able to resolve these issues. The solutions primarily involved executing all the commands within a single session, simplifying the process.
As we reached the end of the fourth week, I had achieved the ability to execute and complete the energy measurement stage in the pipeline. This stage allowed us to generate artifacts, including power meter readings and hardware data, which are subsequently passed to the next stage for analysis and report generation.
Week 5 and Week 6
I began my work on the final phase of the measurement pipeline, which is the result stage. This phase demanded that I acquire proficiency in the R programming language. Thus, during my fifth week, I dedicated time to learning R and subsequently focused on creating preprocessing scripts tailored for the OSCAR tool(Open-source Software Consumption Analysis in R, was developed by students from Umwelt-Campus Birkenfeld). It's worth noting that OSCAR presents significant challenges, particularly in the area of data formatting. To address these formatting complexities, it became imperative to preprocess the scripts before feeding them into OSCAR.
Within the preprocessing script, my primary tasks revolved around handling user input as script arguments and performing conversions from epoch time, all while ensuring that date and time were correctly formatted for OSCAR.
Additionally, I attempted to install OSCAR and R on our Raspberry Pi. However, this proved impractical due to the substantial resource requirements, including issues related to dependencies and available RAM. Consequently, after talking with mentors, I made the decision to utilize the GitLab CI pipeline for both the installation and execution of OSCAR.
Week 7 and Week 8
During this week, I made changes to the OSCAR script so that it could be used in the GitLab CI Pipeline. This mainly involved fixing problems with how data was organized and presented. For example, there was an issue with the way dates were formatted - the format "%y-%m-%d" is different from "%Y-%m-%d," and this was causing errors in the final results. To fix this, I carefully examined the code line by line and collaborated with Achim, who developed OSCAR, to resolve these issues. These issues were important because they caused the report to have "NA" (not available) values in various columns, making it hard to create a proper report file.
I also tackled a problem where OSCAR couldn't convert the date and time into the POSIX time format, resulting in null timestamps. An important thing to note is that if any value becomes null, it causes all columns and rows to have a value of 0. This problem was similar to the one above related to date formatting.
After dealing with rendering and debugging errors, I successfully managed to generate the report by the end of the eighth week. This report was initially created for the already tested tool Okular, but I adjusted the script to generate reports for any application, and I successfully generated one for Kate as well.
Week 9 and Week 10
After testing the OSCAR R script locally on my computer, I set up the result stage for the GitLab CI pipeline and tested it using a tidyverse image that already had most of the necessary software installed and was lightweight. The script uses an updated OSCAR tool from the official repository. It takes the artifacts from the previous stage and runs the OSCAR tool with these updated artifacts to generate a report.
I faced a challenge related to automatically including the date in the filename for the report so that it could be saved as an artifact. The collectl output was saving the file as 'test1.csv-dd-mm-yy.tab', so to make it downloadable, I needed to dynamically know and replace the current date and time in the filename. I had to handle this for both the testing lab system and the GitLab CI pipeline.
To solve this, I defined variables for both situations. In the testing lab system, I copied the file with a generic name 'test1.csv', and in the GitLab pipeline, I used a variable to export it as an artifact with a name that included the date and time, like 'test1.csv-dd-mm-yy' (this specific format was needed because the preprocessing scripts used the filename to extract the date and time).
I also experimented with regex to export the artifact in a more complex format like "test1.csv-joseph-esprimop957-*.tab.gz". Another challenge was forwarding the artifacts from one job to another, which I solved by adding a dependencies field in the next job and by copying the artifacts to the required directory (since changing directories in a job could cause the loss of the default directory where the artifacts are placed).
With the guidance of mentors, I successfully resolved these issues, and all three jobs ran smoothly, allowing me to generate an energy measurement report in the end.
Week 11 and Week 12
Right now, I'm moving the code from the test repository to the official remote eco lab repository. I'm also in the process of setting up GitLab custom runners for running these pipelines for long hours, and I am working with the sysadmin team at KDE for this.
Additionally, I'm working on creating documentation/Report that explain the work I've done and writing a clear guide (readme) for people who will test or contribute to the official repository in the future.
Contact
- IRC: Karanjot Singh (@drquark:kde.org)
- KDE identity: drquark
- Email: [email protected]