Sysadmin/Project Metadata API

From KDE Community Wiki
Revision as of 13:32, 23 March 2016 by BaloneyGeek (talk | contribs) (Documentation about our upcoming apps.kde.org API service.)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
 
Important
The service described in this wiki page is not yet live. Information described in this page may not be taken as fact and is subject to change


Introduction

KDE has relied on a file called kde_projects.xml to provide metadata about all of KDE's various projects to people and applications that need it. For example, kdesrc-build uses it to know the location of the project in KDE's project hierarchy, as well as where it can find the source code to download and build. KDE's Continuous Integration System uses it to obtain much the same information, which it uses to automatically download, build and run tests on the software. The i18n team uses this file to map which translations go into which versions of the software.

However, kde_projects.xml is a mammoth 1.4 MB text file that must be downloaded in its entirety and parsed by all software that need to use it, even if it needs information about just one project. The file contains a lot of extraneous information, and takes a long time to generate, due to which we cannot generate this file live as and when requested.

Therefore, we're deprecating the kde_projects.xml file and replacing it with a REST API, which should be simpler to use, faster to generate and provide the power to the requester to only request the data they need, nothing more and nothing less.

NOTE: The XML file is deprecated, but will continue to be updated every half-hour until at least 31st December 2016, 23:59:59 UTC. After this deadline, the file will not be updated and may be removed at any time. This deadline may be extended based on the community's requirements.

API Endpoints

Endpoint Base: https://apps.kde.org/api/
Content Type: application/json

All endpoints in this API documentation is relative to https://apps.kde.org/api/. All requests and responses to and from this API are JSON encoded.

The API follows REST standards and returns standard HTTP status codes, as well as a JSON representation of the error if one occurs.

The API is neither rate-limited nor does it require authentication. None of the API's endpoints modify any data on our servers; therefore this API cannot be considered to be writable.

The Components Endpoint

Endpoint: /components/
Methods: GET
Possible Return Codes: 200 OK

Sample Return:
[
    "kdesupport",
    "frameworks",
    "extragear",
    "kde",
    "calligra",
    ...
]

This endpoint simply returns a list of all of KDE's major components. Use the Index API to explore the contents of each component

The Index API

Endpoint: /index/[:path]
Methods: GET
Possible Return Codes: 200 OK, 404 Not Found

Sample Return (/index/kde/kdegraphics):
{
    "gwenview": "kde/kdegraphics/gwenview",
    "okular": "kde/kdegraphics/okular",
    "kolourpaint": "kde/kdegraphics/kolourpaint",
    "spectacle": "kde/kdegraphics/spectacle",
    "libs": {
        "libkexiv2": "kde/kdegraphics/libs/libkexiv2",
        "libkdcraw": "kde/kdegraphics/libs/libkdcraw",
        ...
    },
    ...
}

This endpoint returns the hierarchy of projects rooted at the path supplied. The path is optional and may be omitted. For example, a request to https://apps.kde.org/api/index/ (without a path) returns the entire project tree. A request to https://apps.kde.org/api/index/frameworks returns all projects under the frameworks component.

The return is a tree represented in a JSON Object. All non-leaf nodes are themselves subtrees, while all leaf nodes are valid JSON string values giving the complete path to that particular project. This path may be used as the argument to the Projects API.

The Projects API

The Projects API can be used to obtain detailed data about one or more projects. This API has two endpoints which may either be used to obtain data about a single project only, or multiple projects at once.

The Single Project Endpoint

Endpoint: /project/[:path]/:project
Methods: GET
Possible Return Codes: 200 OK, 404 Not Found

Sample Return (/index/kde/kdegraphics/spectacle):
{
    "name": "Screenshot Capture Utility",
    "description": "The new screenshot capture utility, replaces KSnapshot",
    "maintainers": [ { "username": "bgupta", "displayname": "Boudhayan Gupta" }],
    "weblinks": {
        "gitweb": "https://quickgit.kde.org/?p=spectacle.git",
        "appsdb": "https://apps.kde.org/projects/kde/kdegraphics/spectacle"
    },
    "repository": {
        "hasRepo": true,
        "isActive": true,
        "repoType": "git",
        "repoUrls": {
            "git": { "access": "read-only", "url": "git://anongit.kde.org/spectacle" },
            "ssh": { "access": "read+write", "url": "[email protected]:spectacle" },
            ...
        },
        "i18nBranches": {
            "trunk": "none",
            "stable": "none",
            "trunk_kf5": "master",
            "stable_kf5": "Applications/15.12"
        },
        "allBranches": ["master", "Applications/15.12", "Applications/16.04"]
    }
}

This endpoint provides detailed data about a single project. It contains basic data about the project, such as its name, description and a list of maintainers, as well as links to relevant human-readable HTML sites within the KDE infrastructure (such as a web-based Git repository boriwser, or a landing page for the project).

If the project has an associated repository, then the repository.hasRepo key will be true. This object will then carry additional information, such as links to the repository by protocol, a mapping between version control branches and i18n branches, and a list of all version control branches in the repository.

The above sample return contains all the data that can possibly be returned; all keys are listed.

The Multiple Projects Endpoint

Endpoint: /projects/
Methods: POST
Possible Return Codes: 200 OK, 404 Not Found
Payload: JSON array of paths to projects

Sample Request Payload:
[
    "kde/kdegraphics/spectacle",
    "frameworks/baloo",
    ...
]

Sample Return:
{
    "kde/kdegraphics/spectacle": {
        "name": "Screenshot Capture Utility",
        "description": "The new screenshot capture utility, replaces KSnapshot",
        ...
    },
    "frameworks/baloo": {
        "name": "Baloo",
        "description": "Baloo is a framework for searching and managing metadata.",
    },
    ...
}

Sometimes, it may be desirable to request metadata for multiple projects at once. Instead of making multiple requests for individual projects, it is possible to use this endpoint to send a POST request containing a JSON-encoded list of paths for which to request metadata. An object is returned, in which the key is the path to the project, and the value is the exact same object that is returned at the single-project endpoint.

Even if a single path in the array supplied in the payload is invalid, a 404 Not Found error will be generated. The JSON representation of the error return will contain information about which path was invalid.

IMPORTANT: This request may take a not insignificant time to complete; the execution time rises linearly with the number of paths requested. Please make requests to this endpoint sparingly, and try to make it in small batches of paths. The application backing this service is not multithreaded and will block while serving a single request. While we do have a pool of multiple instances of this application running, long-running requests will result in one instance not being able to serve other requests while one completes. A request to generate data for all 2500+ projects in the system may take well over 30 seconds to complete.