Sysadmin/GitKdeOrgManual: Difference between revisions

From KDE Community Wiki
No edit summary
(→‎Personal scratch repositories: CIA is no more, remove link)
(42 intermediate revisions by 24 users not shown)
Line 13: Line 13:
:Central project hub and primary repository browser.
:Central project hub and primary repository browser.


* '''[http://gitweb.kde.org/ gitweb.kde.org]'''
* '''[http://quickgit.kde.org/ quickgit.kde.org]'''
:Alternative repository browser. At present the only way to view personal clones of project repositories and personal scratch repositories ([[#Personal repositories|see below]]), however the former are planned to appear in projects.kde.org in the future.
:Alternative repository browser. At present the only way to view personal clones of project repositories and personal scratch repositories ([[#Personal repositories|see below]]), however the former are planned to appear in projects.kde.org in the future.


* '''[http://git.reviewboard.kde.org/ ReviewBoard]''' (git.reviewboard.kde.org)
* '''[http://git.reviewboard.kde.org/ ReviewBoard]''' (git.reviewboard.kde.org)
:Patch review (account sign-up via [http://identity.kde.org/ KDE Identity]). [[#Using ReviewBoard and post-review|Instructions below]].
:Patch review (account sign-up via [http://identity.kde.org/ KDE Identity]). [http://techbase.kde.org/Development/Review_Board|Instructions].


* '''commits.kde.org'''
* '''commits.kde.org'''
Line 28: Line 28:


KDE developer accounts are managed through [http://identity.kde.org/ KDE Identity]. If you already have a KDE SVN developer account, it has been imported into KDE Identity and you may use the Password Reset feature to set a password and manage your SSH public keys. If you don't have a developer account yet, you can request Developer Access in the website's menu upon registering and logging into your account.
KDE developer accounts are managed through [http://identity.kde.org/ KDE Identity]. If you already have a KDE SVN developer account, it has been imported into KDE Identity and you may use the Password Reset feature to set a password and manage your SSH public keys. If you don't have a developer account yet, you can request Developer Access in the website's menu upon registering and logging into your account.
== Information For KDE Developers ==
You can find general information about using Git as a KDE Developer on the [http://techbase.kde.org/Development/Git KDE Git] page on TechBase.
To configure Git for your KDE Development environment, please see the [http://techbase.kde.org/Development/Git/Configuration KDE Git Configuration] page on TechBase.
You can find some simple step-by-step recipes for using the KDE Git repositories on the [http://techbase.kde.org/Development/Git/Recipes KDE Git Recipes] page on TechBase.


== Overview of repository URL schemes ==
== Overview of repository URL schemes ==
Line 68: Line 76:
   [url "[email protected]:"]
   [url "[email protected]:"]
       pushInsteadOf = kde:
       pushInsteadOf = kde:
If you are in a corporate environment (e.g. behind a firewall) which only allows Ports 80 (HTTP) and 443 (HTTPS), use the following configuration instead (i.e. substitute git:// for http://):
  [url "http://anongit.kde.org/"]
      insteadOf = kde:


Then, to clone e. g. the Amarok repo, just do
Then, to clone e. g. the Amarok repo, just do
Line 73: Line 86:
   $ git clone kde:amarok
   $ git clone kde:amarok


By using the <tt>kde:</tt> prefix, read access will automatically happen over Git, and authenticated SSH is only required for pushes.
By using the <tt>kde:</tt> prefix, read access will automatically happen over Git, and authenticated SSH is only required for pushes. Since commits are mirrored to anongit right when you push them, you will not have to worry about anongit being outdated.


== Server-side commands ==
== Server-side commands ==
Line 80: Line 93:


  ssh [email protected] <command> [parameters]
  ssh [email protected] <command> [parameters]
To simplify the first argument to "kde" (i.e. "<tt>ssh kde <command> [parameters]</tt>"), put the following in ~/.ssh/config:
Host kde
    HostName git.kde.org
    User git


The following is a list of the commands that are currently available, broadly divided into categories according to their purpose.
The following is a list of the commands that are currently available, broadly divided into categories according to their purpose.
Line 124: Line 143:
:More on personal clones [[#Personal repositories|here]].
:More on personal clones [[#Personal repositories|here]].


*'''<span id="destroy">destroy <repository path></span>''' <small>[[#destroy|(link here)]]</small>
*'''<span id="rm">D rm <repository path></span>''' <small>[[#destroy|(link here)]]</small>
:Used to delete a personal clone of a project repository or a personal scratch repository. Requires the repository to be unlocked first using the <tt>[[#unlock|unlock]]</tt> command and will additionally ask for confirmation. See also the <tt>[[#trash|trash]]</tt> command as an alternative to outright and irrevocable deletion.
:Used to delete a personal clone of a project repository or a personal scratch repository. Requires the repository to be unlocked first using the <tt>[[#unlock|unlock]]</tt> command and will additionally ask for confirmation. See also the <tt>[[#trash|D trash]]</tt> command as an alternative to outright and irrevocable deletion.


*'''<span id="lock">lock <repository path></span>''' <small>[[#lock|(link here)]]</small>
*'''<span id="lock">D lock <repository path></span>''' <small>[[#lock|(link here)]]</small>


:Locks a repository, causing the <tt>[[#destroy|destroy]]</tt> command to deny deleting it.  
:Locks a repository, causing the <tt>[[#destroy|destroy]]</tt> command to deny deleting it.  
Line 133: Line 152:
:Newly-created repositories are locked by default.
:Newly-created repositories are locked by default.


*'''<span id="unlock">unlock <repository path></span>''' <small>[[#unlock|(link here)]]</small>
*'''<span id="unlock">D unlock <repository path></span>''' <small>[[#unlock|(link here)]]</small>


:Unlocks a repository, making it possible to delete it using the <tt>[[#destroy|destroy]]</tt> command.
:Unlocks a repository, making it possible to delete it using the <tt>[[#destroy|destroy]]</tt> command.
Line 139: Line 158:
====Commands to manage the personal trash area====
====Commands to manage the personal trash area====


*'''<span id="trash">trash <repository path></span>''' <small>[[#trash|(link here)]]</small>
*'''<span id="trash">D trash <repository path></span>''' <small>[[#trash|(link here)]]</small>


:Moves a repository to the personal trash area, creating an entry in the form <tt><repository path>/<timestamp></tt> there. The timestamps, which have second precision, make it possible to have more than one version of a repository in the trash area at the same time.
:Moves a repository to the personal trash area, creating an entry in the form <tt><repository path>/<timestamp></tt> there. The timestamps, which have second precision, make it possible to have more than one version of a repository in the trash area at the same time.
Line 145: Line 164:
:<span style="color:red">'''Note:'''</span> Entries in the personal trash area are automatically removed after 28 days!
:<span style="color:red">'''Note:'''</span> Entries in the personal trash area are automatically removed after 28 days!


*'''<span id="restore">restore <trash area entry></span>''' <small>[[#restore|(link here)]]</small>
*'''<span id="restore">D restore <trash area entry></span>''' <small>[[#restore|(link here)]]</small>


:Restores an entry from the personal trash area (see the <tt>[[#list-trash|list-trash]]</tt> command below for how to list the contents of your personal trash area).
:Restores an entry from the personal trash area (see the <tt>[[#list-trash|list-trash]]</tt> command below for how to list the contents of your personal trash area).
Line 151: Line 170:
:<tt>restore</tt> will deny restoring an entry if doing so would overwrite an existing repository.
:<tt>restore</tt> will deny restoring an entry if doing so would overwrite an existing repository.


*'''<span id="list-trash">list-trash</span>''' <small>[[#list-trash|(link here)]]</small>
*'''<span id="list-trash">D list-trash</span>''' <small>[[#list-trash|(link here)]]</small>


:Lists all entries in the personal trash area, in the form <tt><repository path>/<timestamp></tt>.
:Lists all entries in the personal trash area, in the form <tt><repository path>/<timestamp></tt>.


===Commands related to repository importing===
===Commands related to repository importing===
*'''<span id="hooks-enable">hooks-enable <project repository path></span>''' <small>[[#hooks-enable|(link here)]]</small>
*'''<span id="hooks-enable">hooks enable <project repository path></span>''' <small>[[#hooks-enable|(link here)]]</small>


:An example:
:An example:


   ssh [email protected] hooks-enable konversation
   ssh [email protected] hooks enable konversation


:Available only to repository and system administrators, this command enables several hook scripts that git.kde.org will then execute during a push operation to the specified project repository. Importantly, it also enables write access for non-administrators, which is otherwise disabled along with the hooks scripts.  
:Available only to repository and system administrators, this command enables several hook scripts that git.kde.org will then execute during a push operation to the specified project repository. Importantly, it also enables write access for non-administrators, which is otherwise disabled along with the hooks scripts.  
Line 176: Line 195:
:Used by system administrators to enable or disable writes to particular repositories or all repositories, for maintenance.
:Used by system administrators to enable or disable writes to particular repositories or all repositories, for maintenance.


*'''<span id="hooks-disable">hooks-disable <project repository path></span>''' <small>[[#hooks-disable|(link here)]]</small>
*'''<span id="hooks-disable">hooks disable <project repository path></span>''' <small>[[#hooks-disable|(link here)]]</small>


:Disables the hook scripts git.kde.org normally executes during a push operation to a project repository. While the hook scripts are disabled only repository administrators can push commits to a repository. Both system and repository administrators have the ability to reenable the hook scripts using the <tt>[[#hooks-enable|hooks-enable]]</tt> command.
:Disables the hook scripts git.kde.org normally executes during a push operation to a project repository. While the hook scripts are disabled only repository administrators can push commits to a repository. Both system and repository administrators have the ability to reenable the hook scripts using the <tt>[[#hooks-enable|hooks enable]]</tt> command.


*'''<span id="ohnoes">ohnoes <show|recover> <repository path> <gitref></span>''' <small>[[#ohnoes|(link here)]]</small>
*'''<span id="ohnoes">ohnoes <show|recover> <repository path> <gitref></span>''' <small>[[#ohnoes|(link here)]]</small>
:Used by system administrators to recover deleted branches or mistaken force pushes (rewinds).
:Used by system administrators to recover deleted branches or mistaken force pushes (rewinds).
== Commit hook keywords ==
See [https://techbase.kde.org/Policies/Commit_Policy#Special_keywords_in_GIT_and_SVN_log_messages Special keywords in GIT and SVN log messages (techbase.kde.org)]


== Personal repositories ==
== Personal repositories ==
Line 197: Line 220:
This scheme makes it very easy to locate all personal clones of a given project and should be preferred over making one in your personal [[#Personal scratch repositories|scratch space]]. (In fact, the server-side <tt>clone</tt> command won't allow you to clone a project repository into your personal scratch space, but nothing technically prevents you from taking the detour of a local clone to achieve this.)
This scheme makes it very easy to locate all personal clones of a given project and should be preferred over making one in your personal [[#Personal scratch repositories|scratch space]]. (In fact, the server-side <tt>clone</tt> command won't allow you to clone a project repository into your personal scratch space, but nothing technically prevents you from taking the detour of a local clone to achieve this.)


Personal clones of project repositories currently do not show up on [http://projects.kde.org KDE Projects], but we have plans to change that in the future. Until then, you can use [http://gitweb.kde.org/ gitweb.kde.org] to browse these repositories.
Personal clones of project repositories currently do not show up on [http://projects.kde.org KDE Projects], but we have plans to change that in the future. Until then, you can use [http://quickgit.kde.org/ quickgit.kde.org] to browse these repositories.


=== Personal scratch repositories ===
=== Personal scratch repositories ===


Personal scratch repositories are a means to start a new project or just to store your favorite .bashrc in a safe location: anything is allowed so long as it is related to KDE or your work for KDE in some way.
Personal scratch repositories are a means to start a new project or just to store your favorite <tt>.bashrc</tt> in a safe location: anything is allowed so long as it is related to KDE or your work for KDE in some way.


Creating one is easily done by just pushing:
For the rest of this instruction, let's assume the following names:
* <kde-id> - Your KDE identity user name
* <repo> - Your (new) git repository name


  git push --all git@git.kde.org:scratch/<your KDE Identity user name>/<repo name of your choosing>
If you start a new project then you need to create a git repository out of it before you start to push it to the scratch repositories. This is done by going into the the directory which will contain your project and type:


(Or you could use <tt>git remote add</tt> to add a remote to push to.)
git init


Personal scratch repositories can be browsed on [http://gitweb.kde.org/ gitweb.kde.org].
This will convert the local directory to a git repository. This is of course not necessary if you already have a local git repo containing your project.


If you feel your new project is ready for the wider world and/or wish to signal that it welcomes outside contributors, you may wish to promote it to the status of a KDE Playground project. KDE Playground project repositories are located at the top-level, i.e. the repository will be moved out of your scratch space and may have to be renamed in the event of a collision with an existing repository name. KDE Playground projects are featured on [http://projects.kde.org KDE Projects] and covered by the [https://mail.kde.org/mailman/listinfo/kde-commits kde-commits] mailing list (and thus [http://commitfilter.kde.org CommitFilter]), [http://lxr.kde.org/ LXR], the [http://www.englishbreakfastnetwork.org/ EBN] and [http://cia.vc/ CIA], unlike personal scratch repositories.
To make pushing to the repository convenient, it is recommended to add a remote to push to first:


To request your scratch repository be promoted to the status of a KDE Playground project, you currently need to file a [http://sysadmin.kde.org/svnaccount/repo-request.php sysadmin repo request]. In the future we plan to provide a fully automated facility on [http://projects.kde.org KDE Projects].
  git remote add origin git@git.kde.org:scratch/<kde-id>/<repo>


Note that we have deliberately decided not to allow the direct creation of KDE Playground projects; the path to existence for a KDE Playground repository project always leads through a personal scratch space first. This is to give you the power to decide whether your project is ready, and also to force you to deliberate whether it truly is.
Now you need to create something to push. Otherwise git will refuse the next step. So it is recommended that you create a README for your project and then commit that. Afterward, you perform the first push to the repository as follows:


===Deleting personal repositories===
  git push --all origin


A personal repository can either be deleted outright and irrevocably by using the <tt>[[#destroy|destroy]]</tt> command (which requires you to <tt>[[#unlock|unlock]]</tt> it first to avoid accidental deletion), or you may move it to the personal trash area with the <tt>[[#trash|trash]]</tt> command.  
Now your new scratch repository is created and initialized. It will take about 30 minutes until the creation of the new repository has propagated to the other tools and is visible there.


'''Entries in the personal trash area are kept for 28 days,''' and can be resurrected at any moment during those 28 days by way of the <tt>[[#restore|restore]]</tt> command. You can list the current contents of your personal trash area with the <tt>[[#list-trash|list-trash]]</tt> command.
git push --all does not push tags. You can push them in a second step with


== Using ReviewBoard and post-review ==
  git push --tags ...as above...


A very comfortable way of posting changes for review is [http://git.reviewboard.kde.org ReviewBoard], where every project repository has its own entry.
Personal scratch repositories can be browsed on [http://quickgit.kde.org/ quickgit.kde.org].


=== Creating your changeset ===
You can set a description on your scratch repository by issuing a command


To create your changeset, you probably want to work in a separate branch - or even in your clone. This is actually suggested and the proper way to do changesets in Git. You can create any number of commits, amend them, and do whatever you want to do - it won't affect the next steps, as you will submit the whole branch for review.
  ssh git@git.kde.org desc scratch/<kde-id>/<repo> "Application to do magic"


Before proceeding it is good practice to rebase your branch onto the branch you want to target for the merge. So, supposing you want to target <tt>master</tt>, make sure it is up-to-date with the remote and then run, and want to publish a review for a local branch:
If you feel your new project is ready for the wider world and/or wish to signal that it welcomes outside contributors, you may wish to promote it to the status of a KDE Playground project. KDE Playground project repositories are located at the top-level, i.e. the repository will be moved out of your scratch space and may have to be renamed in the event of a collision with an existing repository name. KDE Playground projects are featured on [http://projects.kde.org KDE Projects] and covered by the [https://mail.kde.org/mailman/listinfo/kde-commits kde-commits] mailing list (and thus [http://commitfilter.kde.org CommitFilter]), [http://lxr.kde.org/ LXR], and [http://www.englishbreakfastnetwork.org/ EBN], unlike personal scratch repositories.


git rebase master
To request your scratch repository be promoted to the status of a KDE Playground project, you currently need to file a [https://sysadmin.kde.org/tickets/index.php?page=tickets&act=add sysadmin repo request]. In the future we plan to provide a fully automated facility on [http://projects.kde.org KDE Projects].


If you want to post a review for merging a non local branch, you might want to run the following:
Note that we have deliberately decided not to allow the direct creation of KDE Playground projects; the path to existence for a KDE Playground repository project always leads through a personal scratch space first. This is to give you the power to decide whether your project is ready, and also to force you to deliberate whether it truly is.


git merge master
===Deleting personal repositories===


=== Using post-review to post changes for review ===
A personal repository can either be deleted outright and irrevocably by using the <tt>[[#rm|rm]]</tt> command (which requires you to <tt>[[#unlock|unlock]]</tt> it first to avoid accidental deletion), or you may move it to the personal trash area with the <tt>[[#trash|trash]]</tt> command.


Once you are done with the above, it is time to post the changes to ReviewBoard. The easiest and most comfortable way to do that is <tt>[http://www.reviewboard.org/docs/manual/dev/users/tools/post-review/ post-review]</tt>, a handy command line tool which takes care of creating review requests for you.
  ssh git@git.kde.org D trash scratch/<username>/<project>


==== Prerequisites ====
'''Entries in the personal trash area are kept for 28 days,''' and can be resurrected at any moment during those 28 days by way of the <tt>[[#restore|restore]]</tt> command. You can list the current contents of your personal trash area with the <tt>[[#list-trash|list-trash]]</tt> command.


The following has to be done only once to make your local clone fit for use with <tt>post-review</tt>.
== Using Review Board and post-review ==


First of all, you have to tell it about the ReviewBoard server. If your project does not ship with a <tt>.reviewboardrc</tt> file (encourage the project manager to add one!), the first thing you have to run is:
A very comfortable way of posting changes for review is [http://git.reviewboard.kde.org Review Board], where every project repository has its own entry.  You can read about how to use Git and Review Board on the [http://techbase.kde.org/Development/Review_Board KDE Review Board] page.


git config reviewboard.url http://git.reviewboard.kde.org
Here's an [http://ericholscher.com/blog/2011/jan/24/using-reviewboard-git/ insightful blog post] that explains how to use post-review.


ReviewBoard currently only knows the project repositories by their git:// URLs, making it necessary to have a remote using the git:// URL in your clone. If your <tt>origin</tt> remote is already using the git:// URL, you are all set. If not you need to add another remote now.  
== Requesting project migrations from KDE SVN or Gitorious.org ==


Let's suppose you are looking to have some changes to [https://projects.kde.org/projects/extragear/multimedia/amarok Amarok] reviewed, and the URL of your <tt>origin</tt> remote is <tt>git@git.kde.org:amarok</tt>. To add another remote using the git:// URL you might run:
To get your project moved from KDE SVN or Gitorious.org to git.kde.org, you have to file a [http://sysadmin.kde.org/svnaccount/repo-request.php sysadmin request]. It will ask you for the following information:


git remote add anonupstream git://anongit.kde.org/amarok
* The name and description of the project.
git fetch anonupstream
* The current location of the project.
 
* Its current or intended module (e.g. playground/utils or extragear/network).
If your <tt>origin</tt> remote was already using the git:// url, substitute <tt>anonupstream</tt> with <tt>origin</tt> throughout the rest of this tutorial.
* Which KDE Identity user name(s) should have admin rights to the repository and the entry on [http://projects.kde.org KDE Projects].
* The email address that the [http://git.reviewboard.kde.org/ ReviewBoard] group for the project should send emails to.
* The date and time the migration should take place (can be "asap").


==== Creating the review request ====
When we have completed processing your request, there will be an empty repository at the chosen path ([[#Overview of repository URL schemes|more here]]) that the repository administrators can push the data into. (When converting from KDE svn to git this typically involves [[../DeveloperAccessForRuleWriting|writing a rule set]], running svn-all-fast-export, and then pushing the created repository into the new git path.) Once you are done pushing everything to the repository, use the <tt>[[#hooks-enable|hooks enable]]</tt> command to enable the commit hooks and allow write access to non-administrators.


You are now ready to create the review request. The <tt>post-review</tt> command should look something like this:
== Advanced Git ==


post-review --parent=master --tracking-branch=anonupstream/master
=== Safety Precautions ===


This command tells <tt>post-review</tt> that your branch is based upon <tt>master</tt>, and it is set to track the remote branch <tt>anonupstream/master</tt>. You can also give <tt>post-review</tt> some more arguments to avoid using the web interface later - have a look at the [http://www.reviewboard.org/docs/manual/dev/users/tools/post-review/ user manual] for more on that.
With these techniques, always work on a disposable copy of the repo with all the remotes removed, so if you screw up, it doesn't really matter.  Also, work on a separate branch. That way, you can usually use <tt>git reset --hard &lt;original-branch&gt;</tt> to get back to the starting state.


After the command has been run a web address will be shown in the terminal, pointing at your review request.
Also, make sure there are no grafts around (eg: linking to the old kdelibs history in the case of frameworks).  The safest way to do this is to use fresh checkouts.


==== Updating a review request ====
=== Merging repositories ===


If you need to update an existing review request you can invoke <tt>post-review</tt> with an additional <tt>-r</tt> argument, which should be the numeric id of the review request you want to update. Supposing you want to update review request 54, you would run:
The <tt>git-merge-repo</tt> script in [https://projects.kde.org/projects/kde/kdesdk/kde-dev-scripts kde-dev-scripts] can merge the tree and history of one git repository into another.


post-review --parent=master --tracking-branch=anonupstream/master -r 54
First, create a commit in the source repo that removes any files you don't want to copy, and rearrange the remaining files to be as you want them to appear in the target repo. It is important the HEAD of the source and target repositories have completely disjoint trees (so you could copy one tree into the other with no file conflicts).


==== Creating a ReviewBoard-compatible diff ====
Then go to the root of the target repository and run


In some rare cases you simply want to generate a diff and submit it to ReviewBoard later. You can do that by running:
/path/to/kde-dev-scripts/git-merge-repo <path to source repo>


post-review --parent=master -n > your-patch.patch
This will preserve commit identities (unless you filter the source repository - see below).


=== Closing a review request ===
:<span style="color:red">'''Note:'''</span> Before pushing such a merge, talk to [http://sysadmin.kde.org/ sysadmin] (ideally on #kde-sysadmin in irc). They can temporarily disable commit hooks (like CCMAIL and BUG) so that people do not get emails about old commits.


To close a review request, you can either use the ReviewBoard web interface or more conveniently, right in a commit. This is done by using the REVIEW keyword followed by the review ID you want to close. For example, to close review 54, you would put
=== Filtering ===


REVIEW: 54
<tt>git filter-branch</tt> allows you to edit history. This is useful when you want to merge only a small part of one repository into another. You can trim the tree, and also alter the commit messages (for example to add information about the origin of the commits).


in your commit. A message will also be posted to ReviewBoard indicating the commit SHA1 that closed the request.
A combination of <tt>--tree-filter</tt>, <tt>--prune-empty</tt> and <tt>--msg-filter</tt> generally gets what you want. For example,


== Requesting project migrations from KDE SVN or Gitorious.org ==
git filter-branch --prune-empty \
                  --tree-filter "find -type f -\! -path './.git/*' -\! -name foo.\* -delete" \
                  --msg-filter 'cat; echo; echo "Commit $GIT_COMMIT in <source-repo>"' \
                  HEAD


To get your project moved from KDE SVN or Gitorious.org to git.kde.org, you have to file a [https://sysadmin.kde.org/svnaccount/repo-request.php sysadmin request]. The form is still in development, but should work fine. It will ask you for the following information:
This example will remove everything that does not match <tt>foo.*</tt>. Note the <tt>-path</tt> argument to <tt>find</tt> that makes sure you don't delete any of git's own files. <tt>--prune-empty</tt> will remove non-merge commits that no longer have any effect on the tree (you can run <tt>git rebase</tt> after to trim the merge commits if you want). <tt>--msg-filter</tt> adds information about where the commit came from (don't forget to change <tt>&lt;source-repo&gt;</tt>!)


* The name and description of the project.
More complex filters are possible. Have a look at the man page for <tt>git-filter-branch</tt>. Note that while you could use the commit message filter to neuter commit hook keywords like CCMAIL, it is better to ask a sysadmin to disable the commit hooks temporarily while you push.
* The current location of the project.
* Its current or intended module (e.g. playground/utils or extragear/network).
* Which KDE Identity user name(s) should have admin rights to the repository and the entry on [http://projects.kde.org KDE Projects].
* The email address that the [http://git.reviewboard.kde.org/ ReviewBoard] group for the project should send emails to.
* The date and time the migration should take place (can be "asap").


When we have completed processing your request, there will be an empty repository at the chosen path ([[#Overview of repository URL schemes|more here]]) that the repository administrators can push the data into. (When converting from KDE svn to git this typically involves [[../DeveloperAccessForRuleWriting|writing a rule set]], running svn-all-fast-export, and then pushing the created repository into the new git path.) Once you are done pushing everything to the repository, use the <tt>[[#hooks-enable|hooks-enable]]</tt> command to enable the commit hooks and allow write access to non-administrators.
See [http://whileimautomaton.net/2010/04/03012432 mastering git filter-branch: points to extract a subproject] for more helpful hints.

Revision as of 18:38, 19 January 2015

Overview of facilities

Account management; notably managing your SSH public keys for read-write developer access.
  • git.kde.org

The main git server. Should be used only for pushing new commits to a repository over the SSH protocol.

  • anongit.kde.org

Several servers which allow read-only access to the repositories via the git:// and http:// protocols. They are requested to update when anyone pushes to a repo on git.kde.org, so it can be thought of as being always up-to-date.

Central project hub and primary repository browser.
Alternative repository browser. At present the only way to view personal clones of project repositories and personal scratch repositories (see below), however the former are planned to appear in projects.kde.org in the future.
Patch review (account sign-up via KDE Identity). [1].
  • commits.kde.org
Provides Git commit "short URLs", redirecting to KDE Projects and gitweb.kde.org pages as appropriate (example).
Sends an email with each commit for the projects you want to watch.

How to get read-write developer access

KDE developer accounts are managed through KDE Identity. If you already have a KDE SVN developer account, it has been imported into KDE Identity and you may use the Password Reset feature to set a password and manage your SSH public keys. If you don't have a developer account yet, you can request Developer Access in the website's menu upon registering and logging into your account.

Information For KDE Developers

You can find general information about using Git as a KDE Developer on the KDE Git page on TechBase.

To configure Git for your KDE Development environment, please see the KDE Git Configuration page on TechBase.

You can find some simple step-by-step recipes for using the KDE Git repositories on the KDE Git Recipes page on TechBase.

Overview of repository URL schemes

URL prefixes

Anonymous read-only access uses the following URL prefix:

 git://anongit.kde.org/

Read-write developer access uses this prefix instead:

 [email protected]:

Repository paths

Following the prefix, here are the path schemes for different types of repositories:

  • <project identifier>
A KDE project repository, be it part of the KDE SC, KDE Extragear or KDE Playground.
  • websites/<address sans leading www. and dots replaced by dashes>
A KDE website project, e.g. websites/projects-kde-org.
  • sysadmin/<repository name>
Non-public repositories used by KDE's sysadmin team.
  • clones/<original repository path>/<KDE Identity user name>/<user-chosen repository name>
Personal clones of project repositories, e.g. clones/konversation/hein/morecowbell or clones/websites/projects-kde-org/hein/pluginwork (more below).
  • scratch/<KDE identity user name>/<user-chosen repository name>
Personal scratch repositories are a means to start a new project or just to store your favorite .bashrc in a safe location: anything is allowed so long as it is related to KDE or your work for KDE in some way (more below).

Let Git rewrite URL prefixes

Instead of remembering the above URL prefixes, you can also put the following in your ~/.gitconfig:

 [url "git://anongit.kde.org/"]
     insteadOf = kde:
 [url "[email protected]:"]
     pushInsteadOf = kde:

If you are in a corporate environment (e.g. behind a firewall) which only allows Ports 80 (HTTP) and 443 (HTTPS), use the following configuration instead (i.e. substitute git:// for http://):

 [url "http://anongit.kde.org/"]
     insteadOf = kde:

Then, to clone e. g. the Amarok repo, just do

 $ git clone kde:amarok

By using the kde: prefix, read access will automatically happen over Git, and authenticated SSH is only required for pushes. Since commits are mirrored to anongit right when you push them, you will not have to worry about anongit being outdated.

Server-side commands

git.kde.org understands several server-side commands that can be used on the command line via SSH in this fashion:

ssh [email protected] <command> [parameters]

To simplify the first argument to "kde" (i.e. "ssh kde <command> [parameters]"), put the following in ~/.ssh/config:

Host kde
    HostName git.kde.org
    User git

The following is a list of the commands that are currently available, broadly divided into categories according to their purpose.

Commands for information retrieval

Shows a table of repository paths and path patterns you have the permission to see along with details about your access rights to them.
A brief legend for the permission flags shown in the listing:
  • @R - Read permissions.
  • @W - Write permissions.
  • @C - Create permissions (e.g. the initial push to a newly-created repo).
If you want to list actual repositories corresponding to patterns listed by info, such as your personal scratch repositories, see the expand command described next.
Like info above, but actually walks through the repositories to verify the information. It's much slower as a result, and should be used if info doesn't provide enough information. For example, info will list your personal scratch space only in the form of a pattern while expand can list the actual repositories located there.
The output is limited to about 20 rows. The optional regex parameter allows you to filter the listing.
  • who-pushed <repository path> <commit sha1 hash> (link here)
Shows the KDE Identity user name of the contributor who pushed the specified commit to the specified repository.

Commands to manage personal repositories

  • clone <path to source repository> <clone name> (link here)
Can be used to make a personal clone of a project repository.
An example:
 ssh [email protected] clone konversation mykonvi
This results in a clone at clones/konversation/<your KDE Identity user name>/mykonvi.
A second example with a longer source repository path:
 ssh [email protected] clone websites/projects-kde-org newtheme
This results in a clone at clones/websites/projects-kde-org/<your KDE Identity user name>/newtheme.
More on personal clones here.
Used to delete a personal clone of a project repository or a personal scratch repository. Requires the repository to be unlocked first using the unlock command and will additionally ask for confirmation. See also the D trash command as an alternative to outright and irrevocable deletion.
Locks a repository, causing the destroy command to deny deleting it.
Newly-created repositories are locked by default.
Unlocks a repository, making it possible to delete it using the destroy command.

Commands to manage the personal trash area

Moves a repository to the personal trash area, creating an entry in the form <repository path>/<timestamp> there. The timestamps, which have second precision, make it possible to have more than one version of a repository in the trash area at the same time.
Note: Entries in the personal trash area are automatically removed after 28 days!
Restores an entry from the personal trash area (see the list-trash command below for how to list the contents of your personal trash area).
restore will deny restoring an entry if doing so would overwrite an existing repository.
Lists all entries in the personal trash area, in the form <repository path>/<timestamp>.

Commands related to repository importing

An example:
 ssh [email protected] hooks enable konversation
Available only to repository and system administrators, this command enables several hook scripts that git.kde.org will then execute during a push operation to the specified project repository. Importantly, it also enables write access for non-administrators, which is otherwise disabled along with the hooks scripts.
The hook scripts in question are the ones reponsible for forwarding commits to the kde-commits mailing list and CIA.vc, and for processing commit message keywords (BUG, CCMAIL, etc.) that may interact with KDE Bugzilla or cause further emails to be sent. As these hook scripts are only available to project repositories, and not to personal repositories, the command only applies to them.
After creating a new, empty project repository for you the system administators will initially disable the hook scripts so you can safely import large numbers of old commits.

Commands for system administrators

Used by system administrators to run one of the above as another user.
Used by system administrators to enable or disable writes to particular repositories or all repositories, for maintenance.
Disables the hook scripts git.kde.org normally executes during a push operation to a project repository. While the hook scripts are disabled only repository administrators can push commits to a repository. Both system and repository administrators have the ability to reenable the hook scripts using the hooks enable command.
  • ohnoes <show|recover> <repository path> <gitref> (link here)
Used by system administrators to recover deleted branches or mistaken force pushes (rewinds).

Commit hook keywords

See Special keywords in GIT and SVN log messages (techbase.kde.org)

Personal repositories

git.kde.org currently offers two types of personal repositories: Personal clones of project repositories and personal scratch repositories.

Personal clones of project repositories

A personal clone of a project repository can be created using the server-side clone command on the command line:

 ssh [email protected] clone <path to source repository> <clone name>

This will create a clone of the source repository at clones/<path to source repository>/<KDE Identity user name>/<clone name>. (See more examples of clone in action here.)

This scheme makes it very easy to locate all personal clones of a given project and should be preferred over making one in your personal scratch space. (In fact, the server-side clone command won't allow you to clone a project repository into your personal scratch space, but nothing technically prevents you from taking the detour of a local clone to achieve this.)

Personal clones of project repositories currently do not show up on KDE Projects, but we have plans to change that in the future. Until then, you can use quickgit.kde.org to browse these repositories.

Personal scratch repositories

Personal scratch repositories are a means to start a new project or just to store your favorite .bashrc in a safe location: anything is allowed so long as it is related to KDE or your work for KDE in some way.

For the rest of this instruction, let's assume the following names:

  • <kde-id> - Your KDE identity user name
  • <repo> - Your (new) git repository name

If you start a new project then you need to create a git repository out of it before you start to push it to the scratch repositories. This is done by going into the the directory which will contain your project and type:

git init

This will convert the local directory to a git repository. This is of course not necessary if you already have a local git repo containing your project.

To make pushing to the repository convenient, it is recommended to add a remote to push to first:

 git remote add origin [email protected]:scratch/<kde-id>/<repo>

Now you need to create something to push. Otherwise git will refuse the next step. So it is recommended that you create a README for your project and then commit that. Afterward, you perform the first push to the repository as follows:

 git push --all origin

Now your new scratch repository is created and initialized. It will take about 30 minutes until the creation of the new repository has propagated to the other tools and is visible there.

git push --all does not push tags. You can push them in a second step with

 git push --tags ...as above...

Personal scratch repositories can be browsed on quickgit.kde.org.

You can set a description on your scratch repository by issuing a command

 ssh [email protected] desc scratch/<kde-id>/<repo> "Application to do magic"

If you feel your new project is ready for the wider world and/or wish to signal that it welcomes outside contributors, you may wish to promote it to the status of a KDE Playground project. KDE Playground project repositories are located at the top-level, i.e. the repository will be moved out of your scratch space and may have to be renamed in the event of a collision with an existing repository name. KDE Playground projects are featured on KDE Projects and covered by the kde-commits mailing list (and thus CommitFilter), LXR, and EBN, unlike personal scratch repositories.

To request your scratch repository be promoted to the status of a KDE Playground project, you currently need to file a sysadmin repo request. In the future we plan to provide a fully automated facility on KDE Projects.

Note that we have deliberately decided not to allow the direct creation of KDE Playground projects; the path to existence for a KDE Playground repository project always leads through a personal scratch space first. This is to give you the power to decide whether your project is ready, and also to force you to deliberate whether it truly is.

Deleting personal repositories

A personal repository can either be deleted outright and irrevocably by using the rm command (which requires you to unlock it first to avoid accidental deletion), or you may move it to the personal trash area with the trash command.

 ssh [email protected] D trash scratch/<username>/<project>

Entries in the personal trash area are kept for 28 days, and can be resurrected at any moment during those 28 days by way of the restore command. You can list the current contents of your personal trash area with the list-trash command.

Using Review Board and post-review

A very comfortable way of posting changes for review is Review Board, where every project repository has its own entry. You can read about how to use Git and Review Board on the KDE Review Board page.

Here's an insightful blog post that explains how to use post-review.

Requesting project migrations from KDE SVN or Gitorious.org

To get your project moved from KDE SVN or Gitorious.org to git.kde.org, you have to file a sysadmin request. It will ask you for the following information:

  • The name and description of the project.
  • The current location of the project.
  • Its current or intended module (e.g. playground/utils or extragear/network).
  • Which KDE Identity user name(s) should have admin rights to the repository and the entry on KDE Projects.
  • The email address that the ReviewBoard group for the project should send emails to.
  • The date and time the migration should take place (can be "asap").

When we have completed processing your request, there will be an empty repository at the chosen path (more here) that the repository administrators can push the data into. (When converting from KDE svn to git this typically involves writing a rule set, running svn-all-fast-export, and then pushing the created repository into the new git path.) Once you are done pushing everything to the repository, use the hooks enable command to enable the commit hooks and allow write access to non-administrators.

Advanced Git

Safety Precautions

With these techniques, always work on a disposable copy of the repo with all the remotes removed, so if you screw up, it doesn't really matter. Also, work on a separate branch. That way, you can usually use git reset --hard <original-branch> to get back to the starting state.

Also, make sure there are no grafts around (eg: linking to the old kdelibs history in the case of frameworks). The safest way to do this is to use fresh checkouts.

Merging repositories

The git-merge-repo script in kde-dev-scripts can merge the tree and history of one git repository into another.

First, create a commit in the source repo that removes any files you don't want to copy, and rearrange the remaining files to be as you want them to appear in the target repo. It is important the HEAD of the source and target repositories have completely disjoint trees (so you could copy one tree into the other with no file conflicts).

Then go to the root of the target repository and run

/path/to/kde-dev-scripts/git-merge-repo <path to source repo>

This will preserve commit identities (unless you filter the source repository - see below).

Note: Before pushing such a merge, talk to sysadmin (ideally on #kde-sysadmin in irc). They can temporarily disable commit hooks (like CCMAIL and BUG) so that people do not get emails about old commits.

Filtering

git filter-branch allows you to edit history. This is useful when you want to merge only a small part of one repository into another. You can trim the tree, and also alter the commit messages (for example to add information about the origin of the commits).

A combination of --tree-filter, --prune-empty and --msg-filter generally gets what you want. For example,

git filter-branch --prune-empty \
                  --tree-filter "find -type f -\! -path './.git/*' -\! -name foo.\* -delete" \
                  --msg-filter 'cat; echo; echo "Commit $GIT_COMMIT in <source-repo>"' \
                  HEAD

This example will remove everything that does not match foo.*. Note the -path argument to find that makes sure you don't delete any of git's own files. --prune-empty will remove non-merge commits that no longer have any effect on the tree (you can run git rebase after to trim the merge commits if you want). --msg-filter adds information about where the commit came from (don't forget to change <source-repo>!)

More complex filters are possible. Have a look at the man page for git-filter-branch. Note that while you could use the commit message filter to neuter commit hook keywords like CCMAIL, it is better to ask a sysadmin to disable the commit hooks temporarily while you push.

See mastering git filter-branch: points to extract a subproject for more helpful hints.