Different types of cross toolchains and the "multiarch problem"

Sysrooted cross toolchain

The "sysroot" is a folder on the host that contains a copy of the target's filesystem. To be more precise, only those target files needed to compile for the target, mainly libraries and their header files, but also .pc files for pkg-config and so on. A sysrooted compiler prepends the sysroot to all paths automatially. If the compiler is asked to look for includes in /usr/include/blah it will actually translate this into /sysroot/usr/include/blah without any further action needed. The advantage of this kind of cross compiler is, that it will work with unmodified build systems. Just imagine a Makefile with hardcoded include- and library paths.

The multiarch problem

The base of the recommended distribution for the RaspberryPi is Debian Wheezy, which introduces the concept of "multiarch". Other distributions like Arch Linux are much less complicated to cross compile for. The "problem" with multiarch is, that libraries that are usually located in the standard locations /lib and /usr/lib have been moved to /lib/<arch-tuple> and /usr/lib/<arch-tuple>. The compiler included in Debian Wheezy has been patched to make these paths standard, too, because no buildsystem will search standard libraries in these places otherwise. So we need to get that patch into the cross compiler, too. Otherwise it would be neccessary to put these additional paths into the build system as library search paths, but that is not what we want. I dont't want to patch the build system of a project for cross compiling. I dont't want to do that for my own projects and i can't for the others that I just use.

Where to build the cross toolchain

The Raspbian Image uses kernel 3.2 or 3.10, gcc 4.6.3 with eglibc 2.13. All libraries available on RaspberryPi therefore depend on these versions. If crosstool-ng is configured to build a cross toolchain with these versions, the build will not run trough on newer Linux distributions, because some tools are too new there, e.g. autotools. These instructions will work un Ubuntu precise (12.04) and maybe earlier versions, but will fail on Ubuntu 13.10, for example.So you need to build the cross toolchain on e.g. Ubuntu 12.04 (32 bit) and then copy the resulting cross toolchain to your favorite Linux dirtrbution. 

Step 1: Download an build crosstool-ng

Download crosstool-ng version 1.18.0 from http://crosstool-ng.org/

mkdir -p $HOME/Downloads

cd $HOME/Downloads

wget http://crosstool-ng.org/download/crosstool-ng/crosstool-ng-1.18.0.tar.bz2

mkdir -p $HOME/build

cd $HOME/build

tar -xjf $HOME/Downloads/crosstool-ng-1.18.0.tar.bz2

cd $HOME/build/crosstool-ng-1.18.0

./configure --prefix=$HOME/build/ct-ng-1.18-build

Configure will probably abort because it detects that some tools needed for the build are not installed on your system. Just install them and restart the configure step until it succeeds. I had to install the following packets, but depending on what is already installed on your system, you may need to install more or less packets:

• build-essential
• bison
• flex
• gperf
• gawk
• subversion (needed because ct-ng gets eglibc-2.13 out of its subversion repository)
• texinfo (the missing 'makeinfo' tool is in the texinfo package)
• libtool
• autoconf
• automak
• libncurses5-dev
• python-dev

After configure completed successfully, you can start the actual build of crosstool-ng (the 'ct-ng' tool). The tool 'ct-ng' is not yet the cross compiler itself, but the tool to configure and build it, which we will do in step 2.

make && make install

Step 2: configure and build the cross compiler

Then configure crosstool, either manually:

$HOME/build/ct-ng-1.18-build/bin/ct-ng menuconfig

or use these prepared settings: Download here, copy the file raspi.config to $HOME/build/ct-ng-1.18-build/bin and rename it to ".config":

cp $HOME/Downloads/raspi.config $HOME/build/ct-ng-1.18-build/bin/.config

cd $HOME/build/ct-ng-1.18-build/bin

$HOME/build/ct-ng-1.18-build/bin/ct-ng oldconfig

Then start the build (will take about 30 min on a fast maschine):

$HOME/build/ct-ng-1.18-build/bin/ct-ng build

When finished, you wiil find the newly build cross toolchain in $HOME/x-tools . But it is only the C/C++ cross compiler and the corresponding standard libs: libstdc++ and one of glibc, eglibc or dietlibc. For cross compiling more than just "hello world", this is surely not enough.  You can download a tarball with a prebuild toolchain here. Just extract it in $HOME:

cd $HOME && tar -xjf Downloads/x-tools.tar.bz2

Step 3: Additional libraries for use with the cross toolchain

We get all further libraries from the RaspberryPi itself (more details to be added soon):

1. install all needed libraries on the RaspberryPi. They will be needed at runtime anyway.
2. install the corresponding dev packages, too. The include the header files.
3. transfer both to the sysroot folder of the cross toolchain: I use rsync and transfer /lib, /usr/lib and /usr/include
4. adopt symbolic links if they point to an absolute path; make them relative. I have written a python script to do that, available here. 
5. some libs or toolkits are somewhat special, e.g. Qt uses code generators (uic, moc, ...)  that need to run on the host but use target libs => place a qt.conf file next to qmake

cd ~/x-tools/arm-unknown-linux-gnueabihf/arm-unknown-linux-gnueabihf/sysroot

rsync -rav \

--include='/usr' \--exclude='/usr/bin' \

--include='/usr/include/***' \

--exclude='/usr/lib/python*/' \

--exclude='/usr/lib/pypy*/' \

--exclude='/usr/lib/pymodules/' \

--exclude='/usr/lib/pyshared/' \

--exclude='/usr/lib/ruby*/' \

--exclude='/usr/lib/libreoffice/' \

--exclude='/usr/lib/jvm/' \

--include='/usr/lib/***' \

--exclude='/lib/modules/' \

--exclude='/lib/udev/' \

--exclude='/lib/firmware/' \

--exclude='/lib/systemd/' \

--include='/lib/***' \

--include='/etc' \

--include='/etc/ld.so.conf' \

--include='/etc/ld.so.conf.d/***' \

--exclude='/etc/***' \

--exclude='*' \

pi@<raspi-hostname>:/* .

# unfortunately, the cross linker does not find the included files in /etc/ld.so.conf.d/, so include them in in /et/ld.so.conf

#  any symlinks that pointed to absolute paths on the target need to become relative seen from sysroot

~/bin/adpoptsymlinks -s $(pwd)