Sysadmin/GitKdeOrgManual: Difference between revisions

From KDE Community Wiki
(commits.kde.org is discontinued (?))
 
(215 intermediate revisions by 48 users not shown)
Line 1: Line 1:
[[File:Mascot konqi-app-system.png|thumbnail|right|[[Konqi]] and the gears.]]
== Overview of facilities ==
== Overview of facilities ==


* '''[http://identity.kde.org/ KDE Identity]''' (identity.kde.org)
* '''[https://invent.kde.org/ Invent]''' (invent.kde.org)
:Account management; notably managing your SSH public keys for read-write developer access.
:Patch review (account sign-up via [http://identity.kde.org/ KDE Identity]), repository browser and other services.


* '''git.kde.org'''
=== Discontinued services ===
:Serves up the repositories, via git:// for anonymous read access and via SSH (on both ports 22 and 443) for read-write developer access (read-only access via HTTP is on the way).
The following services have been discontinued. Usually their role is now provided by other systems, but they typically are missing some important features which the previous systems had. The redesign unfortunately breaks the workflow e.g. in KDevelop which is not yet adapted to the new setup. The culled systems are listed here for reference as they may well still be referenced in older documentation.
 
* '''commits.kde.org'''
:Provides Git commit "short URLs", redirecting to the repository browser pages as appropriate ([http://commits.kde.org/324dd0cd/a8d1175f61e678f61b3643c867f212ad26ce6f44 example]).


* '''[http://projects.kde.org/ KDE Projects]''' (projects.kde.org)
* '''[http://projects.kde.org/ KDE Projects]''' (projects.kde.org)
:Central project hub and primary repository browser.
:Central project hub and primary repository browser.
:Replaced by: the sysadmin/repo-metadata repository (metadata information); phabricator.kde.org (browser)


* '''[http://gitweb.kde.org/ gitweb.kde.org]'''
* '''[http://commitfilter.kde.org commitfilter.kde.org]'''
:Alternative repository browser. At present the only way to view personal clones of project repositories and personal scratch repositories ([[#Personal repositories|see below]]), however the former are planned to appear in projects.kde.org in the future.
:Sends an email with each commit for the projects you want to watch.


* '''[http://git.reviewboard.kde.org/ ReviewBoard]''' (git.reviewboard.kde.org)
* '''git.kde.org'''
:Patch review (account sign-up via [http://identity.kde.org/ KDE Identity]).
:The main git server. Should be used only for pushing new commits to a repository over the SSH protocol.  


* '''commits.kde.org'''
* '''anongit.kde.org'''
:Provides git commit "short URLs", redirecting to [http://projects.kde.org/ KDE Projects] and [http://gitweb.kde.org/ gitweb.kde.org] pages as appropriate ([http://commits.kde.org/324dd0cd/a8d1175f61e678f61b3643c867f212ad26ce6f44 example]).
:Several servers which allow read-only access to the repositories via the git:// and http:// protocols. They are requested to update when anyone pushes to a repo on git.kde.org, so it can be thought of as being always up-to-date.
 
== How to get read-write developer access ==


KDE developer accounts are managed through [http://identity.kde.org/ KDE Identity]. If you already have a KDE SVN developer account, it has been imported into KDE Identity and you may use the Password Reset feature to set a password and manage your SSH public keys. If you don't have a developer account yet, you can request Developer Access in the website's menu upon registering and logging into your account.
* '''[https://cgit.kde.org/ cgit.kde.org]'''
:Repository browser. It shows also personal clones of project repositories and personal scratch repositories ([[#Personal repositories|see below]]),.


== Overview of repository URL schemes ==
* '''[https://phabricator.kde.org/ Phabricator]''' (phabricator.kde.org)
:Patch review (account sign-up via [http://identity.kde.org/ KDE Identity]). [[Infrastructure/Phabricator|Phabricator]], repository browser, calendar and other services. Phabricator is currently being replaced by Invent and only the workboard component has not been migrated to Invent yet.


=== URL prefixes ===


Anonymous read-only access uses the following URL prefix:
== How to get read-write developer access ==


  git://git.kde.org/
KDE developer accounts are managed through [http://identity.kde.org/ KDE Identity]. If you already have a KDE SVN developer account, it has been imported into KDE Identity and you may use the Password Reset feature to set a password and manage your SSH public keys. If you don't have a developer account yet, you can request Developer Access in the website's menu upon registering and logging into your account.


Read-write developer access uses this prefix instead:
== Information For KDE Developers ==


  git@git.kde.org:
For general information visit the page about [[Infrastructure/Git | the use of Git by KDE]].


=== Repository paths ===
To configure Git for your KDE Development environment, please see the [http://techbase.kde.org/Development/Git/Configuration KDE Git Configuration] page on TechBase.


Following the prefix, here are the path schemes for different types of repositories:
You can find some simple step-by-step recipes for using the KDE Git repositories on the [http://techbase.kde.org/Development/Git/Recipes KDE Git Recipes] page on TechBase.


* '''<project identifier>.git'''
== Some notes about repository URLs ==
:A KDE project repository, be it part of the KDE SC, KDE Extragear or KDE Playground.


* '''websites/<address sans leading www. and dots replaced by dashes>.git'''
=== URL prefixes ===
:A KDE website project, e.g. websites/projects-kde-org.git.
(KDevelop 5.x users please note: If you came here via the link in KDevelop, note that KDE projects has been '''discontinued''' and replaced by ''Invent''. So the neat automatic import of KDE projects ''no longer works'' and you have to do everything manually instead. Sorry!)


* '''sysadmin/<repository name>.git'''
=== Let Git rewrite URL prefixes ===
:Non-public repositories used by KDE's sysadmin team.


* '''clones/<original repository path>/<KDE Identity user name>/<user-chosen repository name>.git'''
Instead of remembering URL prefixes, you can also put the following in your <tt>~/.config/git/config</tt>:
:Personal clones of project repositories, e.g. <tt>clones/konversation.git/hein/morecowbell.git</tt> or <tt>clones/websites/projects-kde-org.git/hein/pluginwork.git</tt> ([[#Personal clones of project repositories|more below]]).


* '''scratch/<KDE identity user name>/<user-chosen repository name>.git'''
  [url "https://invent.kde.org/"]
:Personal scratch repositories are a means to start a new project or just to store your favorite .bashrc in a safe location: anything is allowed so long as it is related to KDE or your work for KDE in some way ([[#Personal scratch repositories|more below]]).
      insteadOf = kde:
  [url "ssh://git@invent.kde.org/"]
      pushInsteadOf = kde:


== Overview of server-side commands ==
Then, to clone e.&nbsp;g. the Amarok repo, just do


git.kde.org understands several server-side commands that can be used on the command line via SSH in this fashion:
  $ git clone kde:multimedia/amarok.git


ssh git@git.kde.org <command> [parameters]
By using the <tt>kde:</tt> prefix, read access will automatically happen over HTTPS, and authenticated SSH is only required for pushes.


Here is a list of the commands that are currently available:
== Commit hook keywords ==


*'''info'''
See [[Policies/Commit_Policy#Special_keywords_in_GIT_and_SVN_log_messages | Special keywords in GIT and SVN log messages]]
:Shows a table of repository paths and path patterns you have the permission to see along with details about your access rights to them.


:A brief legend for the permission flags shown in the listing:
== Personal repositories ==


:* '''@R''' - Read permissions.
invent.kde.org currently offers two types of personal repositories: Personal clones of project repositories and personal repositories.
:* '''@W''' - Write permissions.
:* '''@C''' - Create permissions (e.g. the initial push to a newly-created repo).


:If you want to list actual repositories corresponding to patterns listed by <tt>info</tt>, such as your personal [[#Personal scratch repositories|scratch repositories]], see the <tt>expand</tt> command described next.
=== Personal clones of project repositories ===


*'''expand [regex]'''
A personal clone of a project repository can be created using the gitlab web interface.
:Like <tt>info</tt> above, but actually walks through the repositories to verify the information. It's much slower as a result, and should be used if <tt>info</tt> doesn't provide enough information. For example, <tt>info</tt> will list your personal [[#Personal scratch repositories|scratch space]] only in the form of a pattern while <tt>expand</tt> can list the actual repositories located there.


:The output is limited to about 20 rows. The optional regex parameter allows you to filter the listing.
This will create a clone of the source repository in your personal namespace.


*'''<span id="clone">clone <path to source repository> <clone name></span>'''
=== Personal repositories ===
:Can be used to make a personal clone of a project repository.


:An example:
Personal scratch repositories are a means to start a new project or just to store your favorite <tt>.bashrc</tt> in a safe location: anything is allowed so long as it is related to KDE or your work for KDE in some way.


  ssh git@git.kde.org clone konversation.git mykonvi.git
You can create and delete a new personal repository using the gitlab web interface.


:This results in a clone at <tt>clones/konversation.git/<your KDE Identity user name>/mykonvi.git</tt>.
To request your scratch repository be promoted to the status of a KDE Playground project, you currently need to file a [https://go.kde.org/systickets sysadmin repo request]. For details, see the description of an [[Policies/Application Lifecycle|Application Lifecycle]].


:A second example with a longer source repository path:
== Advanced Git ==


  ssh [email protected] clone websites/projects-kde-org.git newtheme.git
=== Safety Precautions ===


:This results in a clone at <tt>clones/websites/projects-kde-org.git/<your KDE Identity user name>/newtheme.git</tt>.
With these techniques, always work on a disposable copy of the repo with all the remotes removed, so if you screw up, it doesn't really matter.  Also, work on a separate branch.  That way, you can usually use <tt>git reset --hard &lt;original-branch&gt;</tt> to get back to the starting state.


:More on personal clones [[#Personal repositories|here]].
Also, make sure there are no grafts around (eg: linking to the old kdelibs history in the case of frameworks). The safest way to do this is to use fresh checkouts.
 
*'''destroy <repository path>'''
:Used to remove a personal clone of a project repository or a personal scratch repository.
 
*'''sudo <KDE Identity user name> <command>'''
:Used by system administrators to run one of the above as another user.
 
*'''able <en|dis> <@all|repository path>'''
:Used by system administrators to enable or disable writes to particular repositories or all repositories, for maintenance.
 
*'''ohnoes <show|recover> <repository path> <gitref>'''
:Used by system administrators to recover deleted branches or mistaken force pushes (rewinds).
 
== Requesting migrations from KDE SVN or Gitorious.org ==
 
To get your project moved from KDE SVN or Gitorious.org to git.kde.org, you have to file a [https://bugs.kde.org/enter_sysadmin_request.cgi sysadmin request] providing the following information:
 
* The name and description of the project.
* The current location of the project.
* Its current or intended module (e.g. playground/utils or extragear/network).
* Which KDE Identity user name(s) should have admin rights to the repository and the entry on [http://projects.kde.org KDE Projects].
* The email address that the [http://git.reviewboard.kde.org/ ReviewBoard] group for the project should send emails to.
* The date and time the migration should take place (can be "asap").
 
When we have completed processing your request, there will be an empty repository at the chosen path ([[#Overview of repository URL schemes|more here]]) that the repository admins can push the data into.
 
== Personal repositories ==
 
git.kde.org currently offers two types of personal repositories: Personal clones of project repositories and personal scratch repositories.
 
=== Personal clones of project repositories ===


A personal clone of a project repository can be created using the server-side clone command on the command line:
=== Merging repositories ===


  ssh git@git.kde.org clone <path to source repository> <clone name>
The <tt>git-merge-repo</tt> script in [https://projects.kde.org/projects/kde/kdesdk/kde-dev-scripts kde-dev-scripts] can merge the tree and history of one git repository into another.


This will result in a clone of the source repository located at <tt>clones/<path to source repository>/<KDE Identity user name>/<clone name></tt>. More examples of <tt>clone</tt> in action can be found [[#clone|here]] in the overview of server-side commands.
First, create a commit in the source repo that removes any files you don't want to copy, and rearrange the remaining files to be as you want them to appear in the target repo. It is important the HEAD of the source and target repositories have completely disjoint trees (so you could copy one tree into the other with no file conflicts).


This scheme makes it very easy to locate all personal clones of a given project and should be preferred over making one in your personal [[#Personal scratch repositories|scratch space]]. (In fact, the server-side clone command won't allow you to clone a project repository into your personal scratch space, but nothing technically prevents you from taking the detour of a local clone to achieve this.)
Then go to the root of the target repository and run


Personal clones of project repositories currently do not show up on [http://projects.kde.org KDE Projects], but we have plans to change that in the future. Until then, you can use [http://gitweb.kde.org/ gitweb.kde.org] to browse these repositories.
/path/to/kde-dev-scripts/git-merge-repo <path to source repo>


=== Personal scratch repositories ===
This will preserve commit identities (unless you filter the source repository - see below).


Personal scratch repositories are a means to start a new project or just to store your favorite .bashrc in a safe location: anything is allowed so long as it is related to KDE or your work for KDE in some way.
:<span style="color:red">'''Note:'''</span> Before pushing such a merge, talk to [http://sysadmin.kde.org/ sysadmin] (ideally on #kde-sysadmin in irc). They can temporarily disable commit hooks (like CCMAIL and BUG) so that people do not get emails about old commits.


Creating one is easily done by just pushing:
=== Filtering ===


  git push --all git@git.kde.org:scratch/<KDE Identity user name>/<repo name of your choosing>.git
<tt>git filter-branch</tt> allows you to edit history. This is useful when you want to merge only a small part of one repository into another. You can trim the tree, and also alter the commit messages (for example to add information about the origin of the commits).


(Or you could use <tt>git remote add</tt> to add a remote to push to.)
A combination of <tt>--tree-filter</tt>, <tt>--prune-empty</tt> and <tt>--msg-filter</tt> generally gets what you want. For example,


Personal scratch repositories can be browsed on [http://gitweb.kde.org/ gitweb.kde.org].
git filter-branch --prune-empty \
                  --tree-filter "find -type f -\! -path './.git/*' -\! -name foo.\* -delete" \
                  --msg-filter 'cat; echo; echo "Commit $GIT_COMMIT in <source-repo>"' \
                  HEAD


If you feel your new project is ready for the wider world and/or wish to signal that it welcomes outside contributors, you may wish to promote it to the status of a KDE Playground project. KDE Playground project repositories are located at the top-level, i.e. the repository will be moved out of your scratch space and may have to be renamed in the event of a collision with an existing repository name. KDE Playground projects are featured on [http://projects.kde.org KDE Projects] and covered by the [https://mail.kde.org/mailman/listinfo/kde-commits kde-commits] mailing list (and thus [http://commitfilter.kde.org CommitFilter]), [http://lxr.kde.org/ LXR], the [http://www.englishbreakfastnetwork.net/ EBN] and [http://cia.vc/ CIA], unlike personal scratch repositories.
This example will remove everything that does not match <tt>foo.*</tt>. Note the <tt>-path</tt> argument to <tt>find</tt> that makes sure you don't delete any of git's own files. <tt>--prune-empty</tt> will remove non-merge commits that no longer have any effect on the tree (you can run <tt>git rebase</tt> after to trim the merge commits if you want). <tt>--msg-filter</tt> adds information about where the commit came from (don't forget to change <tt>&lt;source-repo&gt;</tt>!)


To request your scratch repository be promoted to the status of a KDE Playground project, you currently need to file a [https://bugs.kde.org/enter_sysadmin_request.cgi sysadmin request]. In the future we plan to provide a fully automated facility on [http://projects.kde.org KDE Projects].
More complex filters are possible. Have a look at the man page for <tt>git-filter-branch</tt>. Note that while you could use the commit message filter to neuter commit hook keywords like CCMAIL, it is better to ask a sysadmin to disable the commit hooks temporarily while you push.


Note that we have deliberately decided not to allow the direct creation of KDE Playground projects; the path to existence for a KDE Playground repository project always leads through a personal scratch space first. This is to give you the power to decide whether your project is ready, and also to force you to deliberate whether it truly is.
See [http://whileimautomaton.net/2010/04/03012432 mastering git filter-branch: points to extract a subproject] for more helpful hints.

Latest revision as of 21:49, 26 May 2023

Konqi and the gears.

Overview of facilities

Patch review (account sign-up via KDE Identity), repository browser and other services.

Discontinued services

The following services have been discontinued. Usually their role is now provided by other systems, but they typically are missing some important features which the previous systems had. The redesign unfortunately breaks the workflow e.g. in KDevelop which is not yet adapted to the new setup. The culled systems are listed here for reference as they may well still be referenced in older documentation.

  • commits.kde.org
Provides Git commit "short URLs", redirecting to the repository browser pages as appropriate (example).
Central project hub and primary repository browser.
Replaced by: the sysadmin/repo-metadata repository (metadata information); phabricator.kde.org (browser)
Sends an email with each commit for the projects you want to watch.
  • git.kde.org
The main git server. Should be used only for pushing new commits to a repository over the SSH protocol.
  • anongit.kde.org
Several servers which allow read-only access to the repositories via the git:// and http:// protocols. They are requested to update when anyone pushes to a repo on git.kde.org, so it can be thought of as being always up-to-date.
Repository browser. It shows also personal clones of project repositories and personal scratch repositories (see below),.
Patch review (account sign-up via KDE Identity). Phabricator, repository browser, calendar and other services. Phabricator is currently being replaced by Invent and only the workboard component has not been migrated to Invent yet.


How to get read-write developer access

KDE developer accounts are managed through KDE Identity. If you already have a KDE SVN developer account, it has been imported into KDE Identity and you may use the Password Reset feature to set a password and manage your SSH public keys. If you don't have a developer account yet, you can request Developer Access in the website's menu upon registering and logging into your account.

Information For KDE Developers

For general information visit the page about the use of Git by KDE.

To configure Git for your KDE Development environment, please see the KDE Git Configuration page on TechBase.

You can find some simple step-by-step recipes for using the KDE Git repositories on the KDE Git Recipes page on TechBase.

Some notes about repository URLs

URL prefixes

(KDevelop 5.x users please note: If you came here via the link in KDevelop, note that KDE projects has been discontinued and replaced by Invent. So the neat automatic import of KDE projects no longer works and you have to do everything manually instead. Sorry!)

Let Git rewrite URL prefixes

Instead of remembering URL prefixes, you can also put the following in your ~/.config/git/config:

 [url "https://invent.kde.org/"]
     insteadOf = kde:
 [url "ssh://[email protected]/"]
     pushInsteadOf = kde:

Then, to clone e. g. the Amarok repo, just do

 $ git clone kde:multimedia/amarok.git

By using the kde: prefix, read access will automatically happen over HTTPS, and authenticated SSH is only required for pushes.

Commit hook keywords

See Special keywords in GIT and SVN log messages

Personal repositories

invent.kde.org currently offers two types of personal repositories: Personal clones of project repositories and personal repositories.

Personal clones of project repositories

A personal clone of a project repository can be created using the gitlab web interface.

This will create a clone of the source repository in your personal namespace.

Personal repositories

Personal scratch repositories are a means to start a new project or just to store your favorite .bashrc in a safe location: anything is allowed so long as it is related to KDE or your work for KDE in some way.

You can create and delete a new personal repository using the gitlab web interface.

To request your scratch repository be promoted to the status of a KDE Playground project, you currently need to file a sysadmin repo request. For details, see the description of an Application Lifecycle.

Advanced Git

Safety Precautions

With these techniques, always work on a disposable copy of the repo with all the remotes removed, so if you screw up, it doesn't really matter. Also, work on a separate branch. That way, you can usually use git reset --hard <original-branch> to get back to the starting state.

Also, make sure there are no grafts around (eg: linking to the old kdelibs history in the case of frameworks). The safest way to do this is to use fresh checkouts.

Merging repositories

The git-merge-repo script in kde-dev-scripts can merge the tree and history of one git repository into another.

First, create a commit in the source repo that removes any files you don't want to copy, and rearrange the remaining files to be as you want them to appear in the target repo. It is important the HEAD of the source and target repositories have completely disjoint trees (so you could copy one tree into the other with no file conflicts).

Then go to the root of the target repository and run

/path/to/kde-dev-scripts/git-merge-repo <path to source repo>

This will preserve commit identities (unless you filter the source repository - see below).

Note: Before pushing such a merge, talk to sysadmin (ideally on #kde-sysadmin in irc). They can temporarily disable commit hooks (like CCMAIL and BUG) so that people do not get emails about old commits.

Filtering

git filter-branch allows you to edit history. This is useful when you want to merge only a small part of one repository into another. You can trim the tree, and also alter the commit messages (for example to add information about the origin of the commits).

A combination of --tree-filter, --prune-empty and --msg-filter generally gets what you want. For example,

git filter-branch --prune-empty \
                  --tree-filter "find -type f -\! -path './.git/*' -\! -name foo.\* -delete" \
                  --msg-filter 'cat; echo; echo "Commit $GIT_COMMIT in <source-repo>"' \
                  HEAD

This example will remove everything that does not match foo.*. Note the -path argument to find that makes sure you don't delete any of git's own files. --prune-empty will remove non-merge commits that no longer have any effect on the tree (you can run git rebase after to trim the merge commits if you want). --msg-filter adds information about where the commit came from (don't forget to change <source-repo>!)

More complex filters are possible. Have a look at the man page for git-filter-branch. Note that while you could use the commit message filter to neuter commit hook keywords like CCMAIL, it is better to ask a sysadmin to disable the commit hooks temporarily while you push.

See mastering git filter-branch: points to extract a subproject for more helpful hints.