Guidelines and HOWTOs/Build from source/Windows: Difference between revisions

From KDE Community Wiki
(53 intermediate revisions by 19 users not shown)
Line 1: Line 1:
''emerge'' is a tool to build the KDE sources and its third-party requirements on MS Windows. It is the '''easy''' way to build KDE on MS Windows.
''Craft'' is a tool to build the sources and its third-party requirements. It is the '''easy''' way to build software.


== Setting up emerge ==
== Setting up craft ==
=== Setting up a compiler ===
=== Installing python ===
Currently emerge supports both the MinGW and Microsoft Visual Studio (msvc) compiler.
While Craft is capable of installing an embedded Python version sufficient for most packages, we recommend a full installation of Python. The current version is 3.7. [https://www.python.org/downloads/ Download Python 3.6+]  
While MinGW is provided by emerge [https://www.visualstudio.com/ Visual Studio], must be installed by the user.


=== Direct X SDK ===
=== Setting up a powershell ===
In order to compile the Qt5 ''qtbase'' package with MinGW, you will also need to install the [https://www.microsoft.com/en-us/download/details.aspx?id=6812 Microsoft DirectX SDK], make sure to open a new command line window after the installation.
Powershell is required to be at least version 5.0.
If you're running Windows 10 you most probably won't need to update Powershell.


=== Installing Emerge ===
You can check the version by invocating <tt>$PSVersionTable.PSVersion</tt> in a Powershell terminal.
* Start a powershell environment.
* Allow executionm of powershell scripts.
  Set-ExecutionPolicy RemoteSigned
* Install emerge and folow the instructions
iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/KDE/emerge/master/setup/install_emerge.ps1'))


== Using emerge ==
In case you need a newer one, we recommend
To use emerge you need to start a Powershell window, point that to <tt>KDEROOT\emerge</tt> and run the initalization script. For example:
[https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-windows?view=powershell-6 Powershell Core].


C:\KDEROOT\emerge\kdeenv.ps1
The latest version of the standard powershell can be found [https://www.microsoft.com/en-us/download/details.aspx?id=50395 here].


This tells emerge about your environment settings (e.g. paths). It will load your configuration from <tt>KDEROOT\etc\kdesettings.ini</tt>. It should not give any error messages, otherwise emerge will not work as expected. The output should look similar to this one (of course with your paths):
=== Setting up a compiler ===
 
Currently, Craft supports both the MinGW and Microsoft Visual Studio (msvc) compiler.
PS C:\kderoot\emerge>.\kdeenv.ps1
While MinGW can be installed by Craft, [https://www.visualstudio.com/ Visual Studio] must be installed independently by the user.
KDEROOT    : C:\kderoot\emerge
KDECOMPILER : msvc2015
KDESVNDIR  : C:\kderoot\download\svn
KDEGITDIR  : C:\kderoot\download\git
DOWNLOADDIR : C:\kderoot\download
PYTHONPATH  : C:\kderoot\python
PS C:\kderoot\emerge>
 
=== Installing the base system ===
You are now ready to start building KDE, it is recommended to do so progressively, relying on emerge to automatically resolve the required dependencies at each set step:
 
* Enter <tt>emerge qt5</tt>.  This will fetch and install Windows versions of numerous UNIX-like utilities and libraries, then checkout, compile and install Qt.  This will take up to several hours.
* Enter <tt>emerge frameworks</tt>.  This will checkout, compile and install the kde frameworks 5 modules.
 
You will now have successfully installed a base KDE system and can now install other KDE modules as required.
 
Every time you want to update or install a package, you should first update your emerge checkout (simply run
cd C:\kderoot\emerge
git pull
to ensure you are using the latest package recipes.
 
=== Common emerge commands ===
* '''Installing a package and its dependencies''': Simply run <syntaxhighlight lang="text">emerge packagename</syntaxhighlight>
* '''Updating an installed package''': Once you have ''packagename'' built, type <syntaxhighlight lang="text">emerge -i packagename</syntaxhighlight> to update <tt>packagename</tt>.
 
== Troubleshooting and special setups ==
===Vista issues===
*[[User:Jstaniek|jstaniek]] 12:02, 15 January 2008 (CET): UAC has infamous heuristics that make programs like patch.exe treat as installers and try to run them with admin rights (!). This heuristics can be tricked by renaming patch.exe to something like pch.exe ([http://nevali.net/2007/01/update-workaround-for-the-cygwin-uac-problem/ example]) but we did not want to add item to our infrastructure. Instead it is possibleto turn off the heuristics (see the screenshot [http://theinvisiblethings.blogspot.com/2007/02/running-vista-every-day.html here in the security blog calling the heuristics 'severe hole in the design of UAC']). If you happen to disable the UAC, as many annoyed users and devs do (msvc demands admin rights anyway!), patch.exe should already work for you as in older Windows. Alternatively you may want to disable UAC [http://www.howtogeek.com/howto/windows-vista/disable-user-account-controluac-for-administrators-only/ for admins only], but this makes no sense if you are the only user of your machine and use only the admin account.
* [http://ben.versionzero.org/wiki/Fixing_the_way_Vista_Auto-detects_Installers This wiki page] lists instructions on how to use program manifest to disable privilege elevation for a single binary and makes patch play nice with UAC.  This [https://bugs.kde.org/show_bug.cgi?id=186712 should] eventually be integrated to emerge scripts.
=== Windows 10 specifics ===
* To allow scripts such as kdeenv.ps1 to run, you may need to set execution policy for scripts to unrestricted. Of course, this means you should know what you're doing when you run scripts etc. The way to do this is to first get a powershell as administrator:
 
PS> Start-Process powershell -Verb runAs
 
Then, once you get an administrative powershell, run:
 
Set-ExecutionPolicy Unrestricted
 
This should let scripts run
 
* To allow .exe files that are downloaded by emerge to run, you need to have permissions. Otherwise, emerge will give you an "Access denied" error and quit. The details of which program failed can be found by running emerge --verbose <packagename>.
 
There's got to be a better, global way to fix this issue just like in the above case, but a work-around is to go to the containing folder in Windows Explorer, Right click on the .exe file, click properties, go to the permissions tab, say "edit" permissions, and set it to Read & Execute "Allow" for everyone.
 
The author of this blurb had to do this for the following programs:
wget (wget.exe)
7zip (7za.exe)
 
=== Building for 32 bit on a 64 bit machine ===
If you build 32bit binaries with emerge on a 64bit system you need to link boost-python against a 32bit python library. In case your standard python installation is 64bit though, you can specify the python installation by adding a file emerge-boost-config.jam in the <tt>%KDEROOT%\etc</tt> directory to build boost in a specific way, with the following contents:
 
<syntaxhighlight lang="text">
# ---------------------
# Python configuration.
# ---------------------
 
# Configure specific Python version.
using python : 3.2 : C:\\Python32_x86 ;
</syntaxhighlight>
This way boost-python would be using the headers & libraries from C:\\Python32_x86 instead of the default one.
Please remember that for mingw compilers you must regenerate the import library for the python dll (also for the mingw 64bit compiler):
<syntaxhighlight lang="text">
R:\> emerge pexports
R:\> pexports C:\Python32_x86\python32.dll > C:\Python32_x86\libs\python32.def
</syntaxhighlight>
NOTE: In case you generate the import library for the 64bit compiler, add the following line to the file python32.def:
<syntaxhighlight lang="text">
Py_InitModule4 = Py_InitModule4_64
</syntaxhighlight>
For both compilers you should now run:
<syntaxhighlight lang="text">
dlltool -d C:\Python32_x86\libs\python32.def -l C:\Python32_x86\libs\libpython32.dll.a
</syntaxhighlight>
Now you should be able to do emerge -i boost-python-src without errors.
 
== Technical background ==
 
=== What emerge does ===
Emerge can be thought of as performing many of the functions of automated tools like cmake, but in a flexible Python scripting framework.  The benefit of this is that new libraries with idiosyncratic installation procedures, conflicting library and header installation names, and complex rules for building on different setups can be generated automatically, and all directory management should be taken care of without the user's input. 
 
The primary logic for the program is contained in the /bin folder of the Git repository.  The script emerge.py serves as the entry point to the system; it runs appropriate code according to the command line arguments.  The basic command '''emerge ''packageName'' ''' performs the five most important actions <tt>--fetch</tt>,  <tt>--unpack</tt>, <tt>--compile</tt>,  <tt>--install</tt>, and <tt>--qmerge</tt>.  The definition for each of these steps is defined using a flexible system called Portage, after the Gentoo package management system. The basic goals are:
 
1. Fetch action retrieves either a binary or the source code for the package. 
 
2. Unpack action installs the source code in a source folder and applies KDE-specific patches.
 
3. Compile action runs package-dependent configure make steps.
 
4. Install action installs the headers and compiled library and executable outputs.


5. Qmerge updates the Emerge registry and performs finalization steps.
=== Direct X SDK (only for MinGW setups) ===
If you are using the MinGW compiler, in order to compile the Qt5 ''qtbase'' package you will also need to install the [https://www.microsoft.com/en-us/download/details.aspx?id=6812 Microsoft DirectX SDK]. Make sure to open a new command line window after the installation for the environment variables to be set.


=== (Optional) Enable development mode ===
Windows 10 supports the creation of symbolic links, but only if the development mode is activated.
Allowing symbolic links allows faster (much faster) extraction of archives.
[https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development Enable development mode]


Emerge also offers functionality to document dependency trees, create patches to upload tweaks and fixes, and update and clean existing installs.  
=== Installing Craft===
# [https://docs.microsoft.com/en-us/powershell/scripting/setup/starting-windows-powershell?view=powershell-6 Start a PowerShell environment] (Do not use PowerShell ISE, it doesn't work very well with python). It might be required to run it [https://docs.microsoft.com/en-us/powershell/scripting/setup/starting-windows-powershell?view=powershell-6#with-administrative-privileges-run-as-administrator as administrator].
# Allow execution of powershell scripts by executing the following command:
#:<pre>Set-ExecutionPolicy -Scope CurrentUser RemoteSigned</pre>
# Install Craft and follow the instructions by executing the following command:
#: <pre>iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/KDE/craft/master/setup/install_craft.ps1'))</pre>
# Consider adding exception for the entire craft root directory to your antivirus' exceptions, otherwise build of some packages can fail (example: build of <code>kde/frameworks/tier2/kdoctools</code> when AVG is used).


The actual commands for fetching, unpacking etc. are defined by three increasingly specialized levels of logic.  The first level is the code in the /bin folder and determines the overall order, steps should be taken, reading environment variables to configure the build environment and compiler set by <tt>kdeenv.bat</tt>, and parsing the directory tree. 


The second set of logic is found in the /bin/BuildSystem, /bin/Package, /bin/Packager, and /bin/Source folders. This is used to determine general procedures for different classes of packages. For example, the "Source" folder contains the logic for running the <tt>--fetch step</tt> for compressed files, git repositories, SVN, and so on.  The "Package" system contains logic for libraries that need to be configured with e.g. CMake, QMake, or internal make systems.
Note: For Microsoft Visual Studio compiler, it's necessary to have VCTOOLSREDISTDIR environment variable pointing to:
"C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Redist\MSVC\xx.xx.xxxxx" for MSVS 2017 and  
VCINSTALLDIR environment pointing to "C:\Program Files (x86)\Microsoft Visual Studio\2015\Community\VC\" for MSVS 2015.


The final set of logic is at the per-package level.  This is what is contained in the /portage/ directory. Emerge is able to automatically search through the Portage folders to find the name of the package you specify. This is where dependencies, special build configurations and special commands are set up.  Individual patch files and different version configuration information is also stored here.  It is relatively straightforward to add a new package to Portage, especially if the package itself can be downloaded and installed with CMake using minimal configuration.
=== Using the stock Qt SDK===
This will skip all Qt packages and use the official Qt builds instead.
It will work fine for most CMake based recipes but definitely cause problems with QMake based projects.
You will of course also miss all patches we usually apply to Qt.
This is only recommended when you know what you are doing and you won't get support for in our channel.


A good way to prepare a package for wider distribution is to create a simple <tt>CMakeLists.txt</tt> it. You can format the addition of this file as a patch, and create a Portage script which merges the patch into the public code repository.
To activate the SDK mode adapt the [QtSDK] section in your etc/kdesettings.ini to something like:
    [QtSDK]
    ## Whether to use prebuild Qt binaries.
    Enabled = True
    ## The path to the Qt sdk.
    Path = C:\Qt
    ## The version of Qt.
    Version = 5.9
    ## The compiler version, if you are not sure what to use, have a look at the directory set in QtSDK/Path.
    ## The compiler must be of the same type as General/KDECOMPILER.
    ## If you are using mingw please make sure you have installed the mingw using the Qt installer.
    Compiler = msvc2017_64


=== Fixing problems and sharing your work ===
== Using Craft ==
See [[Projects/KDE_on_Windows/Development_Workflow]]
To use Craft you need to start a Powershell window, point that to <tt>KDEROOT\craft </tt> and run the initalization script. For example:


== Philosophy ==
C:\CraftRoot\craft\craftenv.ps1
Emerge is a tool that can build the different parts of KDE and its dependencies under Windows. We created this tool to automate and simplify the build process under Windows. We try to build all packages that we offer in the KDE installer with emerge. That has some advantages for us:
=== Easy for people to join us ===
Before emerge it was quite some work to set a system up for development. There were some quirks, which were documented in some mailing lists, but you had to remember them or you ran into an already solved problem again, etc.
Now to get a development machine you need a Windows computer, need to install Python and Git and do the emerge checkout. Then execute emerge to build what you want to build. This is easy for developers coming from Windows to KDE, and also for KDE developers coming to Windows.
=== Easy for us to do build ===
With emerge you can build the whole software stack (low-level libs, Qt, kdelibs, things above that) with only one command. You can start that build, and some hours later you can check if it worked, or if something broke. So we can spot problems easier and earlier. We can also start with a "naked" Windows computer without any other installed software and bootstrap kde on it. That ensures, that no hidden dependencies on some pieces of software sneak in, because then the builds on a "naked" computer would break and show the problem.
=== Easy to collaborate ===
We can test the same emerge build description for a package on different Windows versions/computers before we do binary releases. People can also add build descriptions for new packages to the Subversion repository.


This emerge tool was inspired by the Gentoo emerge tool.
This tells Craft about your environment settings (e.g. paths). It will load your configuration from <tt>CraftRoot\etc\CraftSettings.ini</tt>. If there are any error messages Craft will not work as expected. The output should look similar to this one (of course with your paths):


PS C:\CraftRoot\craft>.\craftenv.ps1
Craft              : C:\CraftRoot
Version            : master
ABI                : windows-msvc2017_64-cl
Download directory  : C:\CraftRoot\download


[[Category:MS Windows]]
Then [https://community.kde.org/Craft start crafting]

Revision as of 13:21, 23 December 2019

Craft is a tool to build the sources and its third-party requirements. It is the easy way to build software.

Setting up craft

Installing python

While Craft is capable of installing an embedded Python version sufficient for most packages, we recommend a full installation of Python. The current version is 3.7. Download Python 3.6+

Setting up a powershell

Powershell is required to be at least version 5.0. If you're running Windows 10 you most probably won't need to update Powershell.

You can check the version by invocating $PSVersionTable.PSVersion in a Powershell terminal.

In case you need a newer one, we recommend Powershell Core.

The latest version of the standard powershell can be found here.

Setting up a compiler

Currently, Craft supports both the MinGW and Microsoft Visual Studio (msvc) compiler. While MinGW can be installed by Craft, Visual Studio must be installed independently by the user.

Direct X SDK (only for MinGW setups)

If you are using the MinGW compiler, in order to compile the Qt5 qtbase package you will also need to install the Microsoft DirectX SDK. Make sure to open a new command line window after the installation for the environment variables to be set.

(Optional) Enable development mode

Windows 10 supports the creation of symbolic links, but only if the development mode is activated. Allowing symbolic links allows faster (much faster) extraction of archives. Enable development mode

Installing Craft

  1. Start a PowerShell environment (Do not use PowerShell ISE, it doesn't work very well with python). It might be required to run it as administrator.
  2. Allow execution of powershell scripts by executing the following command:
    Set-ExecutionPolicy -Scope CurrentUser RemoteSigned
  3. Install Craft and follow the instructions by executing the following command:
    iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/KDE/craft/master/setup/install_craft.ps1'))
  4. Consider adding exception for the entire craft root directory to your antivirus' exceptions, otherwise build of some packages can fail (example: build of kde/frameworks/tier2/kdoctools when AVG is used).


Note: For Microsoft Visual Studio compiler, it's necessary to have VCTOOLSREDISTDIR environment variable pointing to: "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Redist\MSVC\xx.xx.xxxxx" for MSVS 2017 and VCINSTALLDIR environment pointing to "C:\Program Files (x86)\Microsoft Visual Studio\2015\Community\VC\" for MSVS 2015.

Using the stock Qt SDK

This will skip all Qt packages and use the official Qt builds instead. It will work fine for most CMake based recipes but definitely cause problems with QMake based projects. You will of course also miss all patches we usually apply to Qt. This is only recommended when you know what you are doing and you won't get support for in our channel.

To activate the SDK mode adapt the [QtSDK] section in your etc/kdesettings.ini to something like:

   [QtSDK]
   ## Whether to use prebuild Qt binaries.
   Enabled = True
   ## The path to the Qt sdk.
   Path = C:\Qt
   ## The version of Qt.
   Version = 5.9
   ## The compiler version, if you are not sure what to use, have a look at the directory set in QtSDK/Path.
   ## The compiler must be of the same type as General/KDECOMPILER.
   ## If you are using mingw please make sure you have installed the mingw using the Qt installer.
   Compiler = msvc2017_64

Using Craft

To use Craft you need to start a Powershell window, point that to KDEROOT\craft and run the initalization script. For example:

C:\CraftRoot\craft\craftenv.ps1

This tells Craft about your environment settings (e.g. paths). It will load your configuration from CraftRoot\etc\CraftSettings.ini. If there are any error messages Craft will not work as expected. The output should look similar to this one (of course with your paths):

PS C:\CraftRoot\craft>.\craftenv.ps1
Craft               : C:\CraftRoot
Version             : master
ABI                 : windows-msvc2017_64-cl
Download directory  : C:\CraftRoot\download

Then start crafting