[[PageOutline]] = Latest news = The new multi-platform KDE/CI system designed and implemented by Ben and Scarlett in her [http://scarlettgatelyclark.com/2015/kde-sok-project-final-update/ SOK project] has finally gone live as a [https://build.kde.org drop-in-replacing KDE's old Jenkins CI]. However, in June 2015 it was decided to take down the OSX builders, since they had still a lot of vital dependencies on MacPorts ports in order to make KF5 frameworks and apps build. Such dependencies should not exists for proper CI. Right now KDE e.V. is in the process of setting up proper OSX builders on their own Apple hardware near KDE's core infrastructure. Once that has finished things will proceed further w.r.t. KF5 on OSX, but for now we're at a halt doing MacPorts-based KF5-CI. Furthermore the below descriptions are outdated, since the new KDE/CI system has changed logic to quite an extend which requires to adapt to it by rebasing all changes made here onto the new repositories for [http://quickgit.kde.org/?p=sysadmin%2Fci-master-config.git master] and [http://quickgit.kde.org/?p=sysadmin%2Fci-builder-tools.git builders]. = Current status of setting up a OSX/CI system for Qt5+KF5 = This page shall describe what has to be done to get a CI system for Qt5+KF5 up and running on an OSX host. Partially this is achieved by using quite a few ports from MacPorts. The same could and should be done for Qt4+KDE4, as KDE4 software is going to coexist with KF5 for a longer time. The only hurdle is, that one needs to try and patch Qt4 for the CI system in such a way that it installs on OSX fine (e.g. by integrating all patches applied on MacPorts' qt4-mac port.) == Preparing the OSX host system == === Apple developer tools === Apple's developer tools are a necessary precondition for MacPorts as well as for the CI system. Basically you've got to follow the steps which are a must before you can install MacPorts itself by installing * Xcode from the AppStore * developer command line tools from the command line {{{ $ xcode-select --install }}} and agree actively to the developer license agreement {{{ $ xcodebuild -license }}} and you're ready to go ahead. === Start with a fresh Mavericks/MacPorts install === The setup of an OSX/CI system has only been tested with OSX 10.9 (Mavericks). It is not known, whether this also works for older OSX versions. Using the [https://www.macports.org/install.php current official MacPorts version] is recommended. To be on the safe side one needs to make sure that the disk has at least about 120G space in total, if it is planned to build both KF5 branch-groups 'kf5-qt5' and 'stable-kf5-qt5' with Qt 5.3.2 and 5.4.1, respectively. However, total disk consumption depends on which projects will have to be built by the OSX/CI system. === Getting a MacPorts-compatible bash environment === Install ports for MacPorts' bash {{{ $ sudo port install bash bash-completion }}} and set up the [https://projects.kde.org/projects/playground/sdk/macports-kde/repository/revisions/master/show/contrib/user-setup bash environment for MacPorts use] regarding search paths (PATH, MANPATH) === Tools already present on OSX === Some tools are already installed on OSX due to Xcode (like git, svn, ssh, rsync, make, clang etc.) they don't need to be installed via MacPorts for now. Should a more up-to-date version of any one of these be necessary for the CI system it could be added later. === Some more software supplied by MacPorts === Install ports needed for the CI scripts as well as KF5, e.g.: {{{ $ # Ports for CI: $ sudo port install gnutar py27-lxml bzr mercurial wget automake autoconf cppcheck lcov $ $ # Ports for KF5: $ sudo port install shared-mime-info docbook-xml docbook-xsl giflib boost }}} The full list is being kept in [http://quickgit.kde.org/?p=clones%2Fwebsites%2Fbuild-kde-org%2Fkaning%2Fmp-osx-ci.git&a=blob&h=33f4c83dc7dbe47cef917ed93e83862395f7ceb4&hb=5ecd1848588fa5dc54df6a14c3530bcf503a3905&f=ports-list.txt ~/scripts/ports-list.txt], which can be installed in one go by {{{ $ sudo port install `cat ports-list.txt | egrep -v "^(#.*|\w*)$" | xargs` }}} and this shows which are the hidden dependencies to MacPorts: {{{ $ cat ports-list.txt | egrep -v "^(#.*|\s*)$" | xargs port deps }}} == Setting up the CI system itself == === Basic setup of CI system === In order to start building up a build slave for KDE's Jenkins-based CI system it is in principle needed to clone [http://quickgit.kde.org/?p=websites%2Fbuild-kde-org.git websites/build-kde-org], BUT further down it will become clear, that it is a good idea to track all changes to be made for websites/build-kde-org in a dedicated branch. That is why the [http://quickgit.kde.org/?p=clones%2Fwebsites%2Fbuild-kde-org%2Fkaning%2Fmp-osx-ci.git dedicated repository clone "mp-osx-ci"] has been introduced which has the branch 'mp-osx-ci' containing all changes needed for a CI system on OSX. Create some folders, clone and update CI system (assuming a user "kdeci" with its home directory "/Users/kdeci"): {{{ $ cd $ git clone git://anongit.kde.org/clones/websites/build-kde-org/kaning/mp-osx-ci.git scripts $ cd scripts $ git checkout mp-osx-ci $ ./update-setup.sh # Ignore 404 occurring due to error cloning ECMA262 by hg }}} '''The update shell script created a folder {{{~/scripts}}} in which all subsequent commands will have to be executed.''' (BTW, if you were checking out the original repository from git://anongit.kde.org/websites/build-kde-org you'd have to check out the "production" branch.) ''Note: All configuration settings listed below should already be present in the configuration set up in branch 'mp-osx-ci'.'' === Automatic installation of a CI system === There is ongoing effort to create a script which allows an easy setup of an OSX/CI system from scratch. The script [http://quickgit.kde.org/?p=clones%2Fwebsites%2Fbuild-kde-org%2Fkaning%2Fmp-osx-ci.git&a=blob&h=4541f54a612ac4dd101f4d627d4fb26e853d0a9a&hb=d7317599a2c3f27f937aba6bae1d26b16f7b9a19&f=ci-install.sh ci-install.sh] is not yet complete, but can serve as a starting point for further work in this directions. === SSH setup === Set up SSH for rsync access to build host: {{{ $ cat ~/.ssh/config Host build.kde.org Port 2022 $ ssh-keygen -C YOUR@EMAIL.ADDRESS # <-- send your public SSH key to Ben Cooksley in order to get access to the build server }}} === Helper bash scripts === KDE's Jenkins-based CI system mainly [https://community.kde.org/Frameworks/Epics/Continuous_Integration makes use] of these two python scripts: {{{ tools/prepare-environment.py tools/perform-build.py }}} For qt5 they've further down been used in order to show the workflow. But in daily use it is more practical to consider some helper bash scripts which ease all actions needed to run project builds on an OSX/CI system for which there's not yet the Jenkins integration available. The various projects on the CI system have to be handled for the branch-group 'kf5-qt5' by replacing "PROJECT" with the project name of the corresponding framework or application: {{{ $ python2.7 tools/prepare-environment.py --project PROJECT --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/PROJECT ... $ (cd ~/WC/KDECI-builds/kf5-qt5/PROJECT; git checkout jenkins) ... $ python2.7 tools/perform-build.py --project PROJECT --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/PROJECT ... }}} or alternatively by using [http://quickgit.kde.org/?p=clones%2Fwebsites%2Fbuild-kde-org%2Fkaning%2Fmp-osx-ci.git&a=tree&hb=f19d176025af233c22ac2e7170a8ff7e109289e7 helper bash scripts in ~/scripts/ present in the mp-osx-ci branch of the mp-osx-ci repository] {{{ $ ./prepare.sh PROJECT ... $ ./build.sh PROJECT ... }}} where {{{prepare.sh}}} takes care of creating the build directory if it is not yet existing as well as checks out the branch 'jenkins'. Alternatively one can combine these steps using: {{{ $ ./install.sh LIST_OF_PROJECTS [rebuild] }}} The {{{'rebuild'}}} option can be used to trigger calling {{{build.sh}}} even if {{{prepare.sh}}} didn't signal any changes in the remote project repository. Please note that also here 'kf5-qt5' is the default branch-group! If one want to build for branch-group 'stable-kf5-qt5' one needs to create the file {{{~/scripts/branch-group.sh}}} in order to define the variable 'BRANCH_GROUP' which will then be used by the bash helper scripts later on: {{{ $ cat ~/scripts/branch-group.sh BRANCH_GROUP=stable-kf5-qt5 }}} If you want to revert back to 'kf5-qt5' simply comment out the above definition or delete the file. === 1st step: Qt5 build === ==== Prepare environment for Qt5 build ==== We still need to deactivate two ports and set the correct path to KDE-install directory in config/build/darwin-mavericks.cfg: {{{ $ # Further up ports for Qt should have been installed as listed in ports-list.txt: $ # sudo port install zlib openssl dbus jpeg tiff libmng libpng mysql55 pkgconfig sqlite3 icu pcre libiconv $ $ sudo port deactivate cmake # Remove MacPorts' cmake (which came into the system as build-dependency for mysql55) just to be on the safe side $ sudo port deactivate libarchive # Avoid https://bugreports.qt-project.org/browse/QTBUG-42605 -> https://bugreports.qt-project.org/browse/QTBUG-42870 $ cd ~/scripts $ cat config/build/darwin-mavericks.cfg [DEFAULT] scriptsLocation=/Users/kdeci/scripts opSys=darwin architecture=mavericks compiler=clang configurePlatformArgs=-DCMAKE_INSTALL_BUNDLEDIR="Applications/KF5" -DCMAKE_DISABLE_FIND_PACKAGE_X11=ON [General] installPrefix=/opt/kde/install/%(opSys)s/%(architecture)s/%(compiler)s/ [Test] testsEnabled=False $ python2.7 tools/prepare-environment.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/qt5 ... $ }}} BTW, it could be important to also install freetype for Qt5, but it hasn't been clarified yet whether it should be there or not. ==== Standard paths for Qt5 ==== A very important issue for Qt5 and KF5 is that QStandardPaths gets set appropriately for OSX in general and especially for the OSX/CI system, as this installs all frameworks and apps into separate install roots. To enable Qt5/KF5 apps to find the correct locations for files (for configuration, read-only data, writeable data, etc.) it is necessary to patch QStandardPaths. This is done as a [http://quickgit.kde.org/?p=clones%2Fwebsites%2Fbuild-kde-org%2Fkaning%2Fmp-osx-ci.git&a=blob&h=a9b370d8d70e13a73eba8a59cbd11096f6d39692&hb=cb51af17906562fb89fd93f818ce697eb7e9ca6e&f=patches%2Fqt5%2Fkf5-qt5%2Fpatch-qstandardpaths_mac.cpp.diff temporary hack] in {{{ ~/scripts/patches/qt5/kf5-qt5/patch-qstandardpaths_mac.cpp.diff }}} and still requires more work. The patch enables kdoctools and kdelibs4support to locate their files successfully (which had up to now to be handled using an even nastier workaround), but apart from that there are still problems with config files. Thanks to Jeremy Whiting there is an [https://codereview.qt-project.org/#/c/103277/ ongoing effort to get a QSP patch into the Qt sources]. '''In short: this is still work in progress...''' More information about standard paths is given for [http://qt-project.org/doc/qt-5/qstandardpaths.html Qt5] and [https://community.kde.org/Frameworks/Epics/StandardPathsMigration KF5] as well as in [http://mail.kde.org/pipermail/kde-mac/2014-December/002590.html Ian's post on KDE-MAC]. Note: Due to Qt's bug [https://bugreports.qt.io/browse/QTBUG-41136 41136] there is [http://quickgit.kde.org/?p=clones%2Fwebsites%2Fbuild-kde-org%2Fkaning%2Fmp-osx-ci.git&a=blob&h=5ea27b1c2047b29fd530d34ffc2f4cdf56369781&hb=cb51af17906562fb89fd93f818ce697eb7e9ca6e&f=patches%2Fqt5%2Fkf5-qt5%2Fpatch-QTBUG-41136.diff another patch] needed for Qt < 5.4. ==== Set up build environment for Qt5 ==== It is '''required''' to build the debug version. Also make sure that all libs can be found by adding search paths, use "-dbus-linked" to make the [https://bugs.kde.org/show_bug.cgi?id=337140#c22 session bus work] and avoid inclusion of MacPorts' glib2: {{{ $ cd ~/scripts; cat config/build/qt5/darwin-mavericks.cfg [Build] configureCommand=%(configureExecutable)s -debug -separate-debug-info -system-zlib -system-libpng -system-libjpeg -system-sqlite -dbus-linked -plugin-sql-mysql -nomake examples -confirm-license -opensource -prefix {instPrefix} -no-framework -v -I/opt/local/include -L/opt/local/lib -no-glib }}} Include MySQL's binary path into PATH e.g. in {{{.macports/profile}}} (assuming /opt/local as MacPort's prefix): {{{ export PATH=/opt/local/lib/mysql55/bin:$PATH }}} Start the qt5-build {{{ $ python2.7 tools/perform-build.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/qt5 ... }}} === Installation of cmake === CMake needs a '''manual''' checkout of its sources (which is demonstrated here the explicit way again): {{{ $ python2.7 tools/prepare-environment.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/cmake ... $ git clone git://cmake.org/cmake.git ~/WC/KDECI-builds/kf5-qt5/cmake $ python2.7 tools/perform-build.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/kf5-qt5/cmake ... $ $ # Alternatively you can simply use this script (which does all of the above for you): $ ./install.sh cmake }}} '''Important note:''' * For a while it seemed to be necessary to NOT to install the latest cmake version on OSX, as this causes [http://mail.kde.org/pipermail/kde-frameworks-devel/2014-August/018457.html build problems for framework kcoreaddons] later on. However, the commit 3c0d87645f5861e9fd6fd78855697f77ef77144e was recommended by Cristian Onet which worked fine. * In October 2014 the same regression reappeared, but was fixed in version 3.1.0-rc1 * The latest cmake version (being thoroughly tested against KF5) is secretly being set in {{{prepare.sh}}}. === Setting up the build environment === The branch 'mp-osx-ci' contains all required configuration settings for OSX. The most important shall be pointed out [wiki:/KDEProblems/KDEMacPortsCI/Status/Diff_mp-osx-ci here]: - For KF5 on OSX one has to modify the configuration settings by commenting out the kauth dependency on polkit-qt-1, since that dependency is only needed on Linux. - Problems appeared for some frameworks since kconfig_compiler_kf5.app doesn't yet get installed on OSX (the build scripts error out with "/Applications/KDE/kconfig_compiler_kf5.app/Contents/MacOS/kconfig_compiler_kf5 as location is wrong"). This can be fixed for now introducing a temporary configuration file for kconfig which sets CMAKE_INSTALL_BUNDLEDIR to make sure that the application bundle gets installed below PREFIX/lib/libexec/. - CMAKE_INSTALL_BUNDLEDIR and DATA_INSTALL_DIR need to be set for OSX-like directories. (NOT USED AT THE MOMENT, since all files currently get installed just like on Linux, as long as the [https://codereview.qt-project.org/#/c/103277/ QSP patch is still not finalized].) - Disable searching for X11 by setting CMAKE_DISABLE_FIND_PACKAGE_X11=ON. - Disable running tests for now, in order to save time during rebuilding. - Coverage doesn't work for kdevplatform and plasma-framework The file {{{kde_projects.xml}}} is necessary in order to run the CI scripts. Usually it should be kept up to date in {{{~/scripts}}}, but in some cases it might be corrupted or not existing at all. Then one should download it from [http://projects.kde.org/kde_projects.xml KDE-projects]. === Keeping the OSX host system up-to-date === Updating the OSX operating system as well as the MacPorts installation should not be a problem for the host system as qt5 and all KF5 projects should be binary-compatible. In doubt, or if a project fails because of - indeed possible - library version changes (which is e.g. explicitly checked on MacPorts after installing new ports) simply rebuild all projects. === Jenkins integration === Using {{{all-tiers-install.sh}}} or the other bash scripts allows running the OSX/CI system manually. However, in order to seamlessly integrate this into KDE's automatically functioning CI infrastructure it is necessary to create a [http://ci.openstack.org/jenkins-job-builder/index.html Jenkins builder yaml configuration] which will replace the currently used configuration on [http://build.kde.org] allowing to configure build jobs for other platforms like Windows and OSX. This enormous project has been tackled in [http://scarlettgatelyclark.com/2015/kde-sok-project-final-update/ Scarlett G. Clark's SOK project] and has eventually gone live by [drop-in-replacing KDE's old Jenkins CI https://build.kde.org]. Some experiments have been made to get a Jenkins slave for OSX up and running, which is still using the classic configuration approach. The changes needed for the two builder scripts are kept in [http://quickgit.kde.org/?p=clones%2Fwebsites%2Fbuild-kde-org%2Fkaning%2Fmp-osx-ci.git&a=shortlog&h=d990247ab26a041d126e72decd03713159301a0e branch 'mp-osx-ci-jenkins' in mp-osx-ci]. At the moment only a few projects are configured, which can be built successfully, even the [http://build.kde.org/job/trojita_master_qt5/ multi-variation project Trojita]. == Installation of KF5 frameworks and other projects == Specific projects can be installed using {{{install.sh}}}, but one can also build all projects of a specific tier as a whole using {{{ $ # This builds all frameworks of tier 1 $ ./tier-install.sh 1 ... $ # This builds all frameworks of tier 2, but forces rebuilding every project regardless of $ # whether git pulled in changes or not (useful when e.g. qt5 or cmake was updated) $ ./tier-install.sh 2 rebuild ... $ # This rebuilds all frameworks of tier 3 starting at project kio (i.e. skipping all predecessor frameworks) $ ./tier-install.sh 3 kio rebuild ... }}} All projects of all tiers can be build easily using {{{ $ ./all-tiers-install.sh [ START_TIER [START_PROJECT] ] [rebuild] }}} (optionally) beginning at START_TIER building up to tier 5. Tiers 0 and 5 are just "virtual" tiers - which were introduced to more easily build some projects. '''As of July 5th 2014 [wiki:/KDEProblems/KDEMacPortsCI/Status/KF5 all KF5 frameworks build successfully].''' === Troubleshooting for projects' dependencies === [http://quickgit.kde.org/?p=kde-build-metadata.git Dependency data for the CI system] is being downloaded by ~/scripts/update-setup.sh to ~/scripts/dependencies/. This folder contains files defining the dependencies (see subsections below) as well as some tools. Sometimes it might be needed to add a new entry to the KF5-specific (logical) dependency data in case a project cannot find a required framework, although it is already available on the CI system! Details concerning this can be found on a [wiki:/KDEProblems/KDEMacPortsCI/Status/Dependencies dedicated page]. === Notes concerning some files installed by the frameworks === All notes regarding the placement of certain files (apps, config, data, etc.) have been moved aside to a [wiki:/KDEProblems/KDEMacPortsCI/Status/FileLocations dedicated page] as the patching of Qt5's QStandardPaths is still work in progress. === KF5 software beyond the basic frameworks === Since the basic set of frameworks is now ready for use on OSX one can go ahead and start building other software depending on KF5. A first step is building kdesupport-svn which is a prerequisite for further projects. Important projects are e.g. KDevelop, KMyMoney, but also [https://projects.kde.org/projects/kde/kdeedu KDE-EDU applications] on their [https://community.kde.org/KDEEdu/RouteToKF5 route to KF5] as well as [https://techbase.kde.org/Projects/Games/kdegame_apps_porting_kf5_status KDE games] can serve as good test cases. This is referred to as "Tier 5" and details can be found [wiki:/KDEProblems/KDEMacPortsCI/Status/ProjectsBeyondKF5 here]. Regularly updated information can be found on Christoph Feck's [http://developer.kde.org/~cfeck/portingstatus.html porting status page]. == Some stats about the OSX/CI system == Altogether there are more than two hundred KF5 frameworks and applications being regularly built for OSX (as of March 14th 2015): {{{ $ cat tier*.fw | egrep -v "^#" | wc | awk '{print $1}' 231 }}} of which currently only 13 fail. The roundtrip time for a full check on all projects - if nothing has to be built - is on the 7-core-i7-iMac-OSX/CI-VM about 1h at the moment, which is mainly due to the rsync access to Jenkins master... A full rebuild of all projects (excluding qt5 and cmake) takes about 2,5 hrs (for 118 projects as of November 2014): {{{ $ cd ~/scripts; time ./all-tiers-install.sh rebuild ... real 143m12.696s user 193m56.993s sys 42m30.157s }}} while a great portion of that is also due to the deployment of all built binaries to Jenkins master. Building including qt5 with a rebuild of all projects (except cmake) needs 3 hrs (for 118 projects as of November 2014): {{{ $ cd ~/scripts; time (./install.sh qt5 rebuild && ./all-tiers-install.sh rebuild) ... real 189m14.808s user 328m31.106s sys 64m10.909s }}} where the user time indicates that qt5 increased the parallel building tremendously. With tests enabled building everything would run for about 6 hrs (for 118 projects as of November 2014).