wiki:KDEProblems/KDEMacPortsCI/Status

Version 226 (modified by mkae (Marko Käning), 10 years ago) (diff)

fix shell commands in basic setup of CI system section

Current status of setting up the CI system

This page shall describe what has to be done to get a KDE/CI system up and running on an OSX host. Partially this is achieved by using quite a few ports from MacPorts.

Preparing the OSX host system

Start with a fresh Mavericks/MacPorts install

Currently using the official MacPorts version 2.3.1 is recommended. To be on the safe side make sure your disk has about 60G space in total.

However, currently a Mavericks 10.9.4 KDE/CI system on an i7-iMac uses something like this:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/disk0s2      64G   40G   25G  62% /

that is after all frameworks have been built and installed as shown further down.

Getting a MacPorts-compatible bash environment

Install ports for MacPorts' bash

$ sudo port install bash bash-completion

and set up the bash environment for MacPorts use regarding search paths (PATH, MANPATH)

Apple developer tools

Install

  • Xcode from AppStore (and agree actively to the developer license agreement)
  • developer command line tools
    $ xcode-select --install
    

Tools already present on OSX

Some tools are already installed on OSX due to Xcode (like git, svn, ssh, rsync, make, clang etc.) they don't need to be installed via MacPorts for now. Should a more up-to-date version of any one of these be necessary for the CI system it could be added later.

Some more software supplied by MacPorts

Install ports needed for the CI scripts as well as KF5:

$ # Ports for CI:
$ sudo port install gnutar py27-lxml bazaar mercurial wget automake autoconf cppcheck lcov
$
$ # Ports for KF5:
$ sudo port install shared-mime-info docbook-xml docbook-xsl giflib boost

Setting up the CI system itself

Basic setup of CI system

In order to start building up a build slave for KDE's Jenkins-based CI system it is in principle needed to clone websites/build-kde-org, BUT further down it will become clear, that it is a good idea to track all changes to be made for websites/build-kde-org in a dedicated branch. That is why the dedicated repository clone "mp-osx-ci" has been introduced which has the branch 'mp-osx-ci' containing all changes needed for a CI system on OSX.

Create some folders, clone and update CI system (assuming a user "kdeci" with its home directory "/Users/kdeci"):

$ cd
$ mkdir -p WC/KDECI-builds;
$ git clone git://anongit.kde.org/clones/websites/build-kde-org/kaning/mp-osx-ci.git scripts
$ cd scripts
$ git checkout mp-osx-ci
$ ./update-setup.sh                    # Ignore 404 occurring due to error cloning ECMA262 by hg

The update shell script created a folder ~/scripts in which all subsequent commands will have to be executed.

(BTW, if you were checking out the original repository from git://anongit.kde.org/websites/build-kde-org you'd have to check out the "production" branch.)

Note: All configuration settings listed below should already be present in the configuration set up in branch 'mp-osx-ci'.

Automatic installation of a CI system

There is ongoing effort to create a script which allows an easy setup of an OSX/CI system from scratch. The script ci-install.sh is not yet complete, but can serve as a starting point for further work in this directions.

SSH setup

Set up SSH for rsync access to build host:

$ cat ~/.ssh/config 
Host build.kde.org
   Port 2022
$ ssh-keygen -C YOUR@EMAIL.ADDRESS       # <-- send your public SSH key to Ben Cooksley in order to get access to the build server

Helper bash scripts

KDE's Jenkins-based CI system mainly makes use of these two python scripts:

tools/prepare-environment.py
tools/perform-build.py

For qt5 they've further down been used in order to show the workflow. But in daily use it is more practical to consider some helper bash scripts which ease all actions needed to run project builds on an OSX/CI system for which there's not yet the Jenkins integration available.

The various projects on the CI system have to be handled by replacing "PROJECT" with the project name of the corresponding framework or application:

$ python2.7 tools/prepare-environment.py --project PROJECT --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/PROJECT
...
$ (cd ~/WC/KDECI-builds/PROJECT; git checkout jenkins)
...
$ python2.7 tools/perform-build.py --project PROJECT --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/PROJECT
...

or alternatively by using helper bash scripts in ~/scripts/ present in the mp-osx-ci branch of the mp-osx-ci repository

$ ./prepare.sh PROJECT
...
$ ./build.sh PROJECT
...

where prepare.sh takes care of creating the build directory if it is not yet existing as well as checks out the branch 'jenkins'. Alternatively one can combine these steps using:

$ ./install.sh LIST_OF_PROJECTS [rebuild]

The 'rebuild' option can be used to trigger calling build.sh even if prepare.sh didn't signal any changes in the remote project repository.

1st step: Qt5 build

Prepare environment for Qt5 build

Install additional ports specific for Qt5 as well as KF5, set correct path to KDE-install directory in config/build/darwin-mavericks.cfg (note, some of the to-be-installed ports got already installed further up as dependencies for other tools, they're listed here to make the dependency on them explicit):

$ sudo port install zlib openssl dbus jpeg tiff libmng libpng mysql55 pkgconfig sqlite2 icu pcre libiconv
$ sudo port uninstall cmake            # Remove MacPorts' cmake (which came into the system as build-dependency for mysql55)
$ cd ~/scripts
$ cat config/build/darwin-mavericks.cfg
[DEFAULT]
scriptsLocation=/Users/kdeci/scripts
opSys=darwin
architecture=mavericks
compiler=clang

[General]
installPrefix=/opt/kde/install/%(opSys)s/%(architecture)s/%(compiler)s/
$ python2.7 tools/prepare-environment.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/qt5
...
$

BTW, it could be important to also install freetype for Qt5, but it hasn't been clarified yet whether it should be there or not.

Standard paths for Qt5

A very important issue for Qt5 and KF5 is that QStandardPaths gets set appropriately for OSX in general and especially for the OSX/CI system, as this installs all frameworks and apps into separate install roots. To enable Qt5/KF5 apps to find the correct locations for files (for configuration, read-only data, writeable data, etc.) it is necessary to patch QStandardPaths. This is done as a temporary hack in

~/scripts/patches/qt5/kf5-qt5/patch-qstandardpaths_mac.cpp.diff

and still required more work. The patch enables kdoctools and kdelibs4support to locate their files successfully (which had up to now to be handled using an even nastier workaround), but apart from that there are still problems with config files. In short: this is still work in progress...

More information about standard paths is given for Qt5 and KF5.

Set up build environment for Qt5

It is required to build the debug version. Also make sure that all libs can be found by adding search paths, use "-dbus-linked" to make the session bus work and avoid inclusion of MacPorts' glib2:

$ cd ~/scripts; cat config/build/qt5/darwin-mavericks.cfg
[Build]
configureCommand=%(configureExecutable)s -debug -separate-debug-info -system-zlib -system-libpng -system-libjpeg -system-sqlite -dbus-linked -plugin-sql-mysql -nomake examples -confirm-license -opensource -prefix {instPrefix} -no-framework -v -I/opt/local/include -L/opt/local/lib -no-glib

Include MySQL's binary path into PATH e.g. in .macports/profile (assuming /opt/local as MacPort's prefix):

export PATH=/opt/local/lib/mysql55/bin:$PATH

Start the qt5-build

$ python2.7 tools/perform-build.py --project qt5 --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/qt5
...

Installation of cmake

CMake needs a manual checkout of its sources (which is demonstrated here the explicit way again):

$ python2.7 tools/prepare-environment.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/cmake
...
$ git clone git://cmake.org/cmake.git ~/WC/KDECI-builds/cmake
$ python2.7 tools/perform-build.py --project cmake --branchGroup kf5-qt5 --platform darwin-mavericks --sources ~/WC/KDECI-builds/cmake
...
$ 
$ # Alternatively you can simply use this script (which does all of the above for you):
$ ./install.sh cmake

Important note:

  • The last version is secretly being set in prepare.sh.

Setting up the build environment

The branch 'mp-osx-ci' contains all required configuration settings for OSX. The most important shall be pointed out here:

  • For KF5 on OSX one has to modify the configuration settings by commenting out the kauth dependency on polkit-qt-1, since that dependency is only needed on Linux.
  • Problems appeared for some frameworks since kconfig_compiler_kf5.app doesn't yet get installed on OSX (the build scripts error out with "/Applications/KDE/kconfig_compiler_kf5.app/Contents/MacOS/kconfig_compiler_kf5 as location is wrong"). This can be fixed for now introducing a temporary configuration file for kconfig which sets CMAKE_INSTALL_BUNDLEDIR to make sure that the application bundle gets installed below PREFIX/lib/libexec/.
  • CMAKE_INSTALL_BUNDLEDIR and DATA_INSTALL_DIR need to be set for OSX-like directories.
  • Disable searching for X11 by setting CMAKE_DISABLE_FIND_PACKAGE_X11=ON.
  • Disable running tests for now, until dbus works as expected and test applications can be started normally.
  • DYLD_LIBRARY_PATH has only temporally been added due to wrong linking of phonon (e.g. for kate and kwrite).
  • Disable coverage for plasma-framework for now until a coverage tool for clang is integrated.

Keeping the OSX host system up-to-date

Updating the OSX operating system as well as the MacPorts installation should not be a problem for the host system as qt5 and all KF5 projects should be binary-compatible. In doubt, or if a project fails because of - indeed possible - library version changes (which is e.g. explicitly checked on MacPorts after installing new ports) simply rebuild all projects.

Jenkins integration

Using all-tiers-install.sh or the other bash scripts allows running the OSX/CI system manually.

However, in order to seamlessly integrate this into KDE's automatically functioning CI infrastructure it is necessary to create a Jenkins builder yaml configuration which will replace the currently used configuration on http://build.kde.org allowing to configure build jobs for other platforms like Windows and OSX.

The creation of the required YAML file has up to now not been tackled yet and any support is welcome!

Installation of KF5 frameworks and other projects

Specific frameworks can be installed using install.sh, but one can also build all frameworks of a specific tier as a whole using

$ # This builds all frameworks of tier 1
$ ./tier-install.sh 1
...
$ # This builds all frameworks of tier 2, but forces rebuilding every project regardless of
$ # whether git pulled in changes or not (useful when e.g. qt5 or cmake was updated)
$ ./tier-install.sh 2 rebuild
...
$ # This rebuilds all frameworks of tier 3 starting at project kio (i.e. skipping all predecessor frameworks)
$ ./tier-install.sh 3 kio rebuild
...

All frameworks of all tiers can be build easily using

$ ./all-tiers-install.sh [ START_TIER [START_PROJECT] ] [rebuild]

(optionally) beginning at START_TIER building up to tier 5. Tiers 0 and 5 are just "virtual" tiers - which were introduced to more easily build some projects.

As of July 5th 2014 all KF5 frameworks build successfully.

Troubleshooting for projects' dependencies

Dependency data for the CI system is being downloaded by ~/scripts/update-setup.sh to ~/scripts/dependencies/. This folder contains files defining the dependencies (see subsections below) as well as some tools.

Sometimes it might be needed to add a new entry to the KF5-specific (logical) dependency data in case a project cannot find a required framework, although it is already available on the CI system! Details concerning this can be found on a dedicated page.

Notes concerning some files installed by the frameworks

All notes regarding the placement of certain files (apps, config, data, etc.) have been moved aside to a dedicated page as the patching of Qt5's QStandardPaths is still work in progress.

KF5 software beyond the basic frameworks

Since the basic set of frameworks is now ready for use on OSX one can go ahead and start building other software depending on KF5. A first step is building kdesupport-svn which is a prerequisite for further projects.

Important projects are e.g. KDevelop, KMyMoney, but also KDE-EDU applications on their route to KF5 can serve as good test cases. This is referred to as "Tier 5" and details can be found here.