Cross compiling for embedded systems

Use the power of a modern desktop PC for compiling and deploy the build results to an much more limited embedded system.What sounds straight forward is much more complicated in detail. The desktop system's software usually has totally different release intervals that the ones of the embedded target. Embedded targets often run for tens of years with nearly the same software while desktop systems get their hardware and software updated every three years or so. Even if the target shares the same CPU architecture with the build host, it is wise to use a special cross compiler for the target instead of the host's compiler, just to keep the cross tooling independent of the host. Doing so will enable you to develop software for a specific embedded target and still freely choose your favorite Linux distribution on the host, as well as doing all release updates of that Linux distribution without loosing the ability to build software for the embedded target.The GNU tools on Linux allow easily to run binaries that were built on an older system on a newer one, but not the other way around. So when building a cross toolchain on a very new Linux distribution, it might not run on an older one. In short: You usually cannot use an older Linux distribution for your development PC than the distribution's version the tooling was compiled on. So don't use the newest distribution to build a cross toolchain. Same for 64bit <-> 32bit.

Sysrooted cross toolchain

The "sysroot" is a folder on the host that contains a copy of the target's filesystem. To be more precise, only those target files needed to compile for the target, mainly libraries and their header files, but also .pc files for pkg-config and so on. A sysrooted compiler prepends the sysroot to all paths automatically. If the compiler is asked to look for includes in /usr/include/blah it will actually translate this into /sysroot/usr/include/blah without any further action needed. The advantage of this kind of cross compiler is, that it will work with unmodified build systems. Just imagine a Makefile with hardcoded include- and library paths. The sysroot also seperates the host's libraries and headers from the target's libraries and headers, decopling them completely.  

Impact of kernel version

Compiling a program on one machine with a toolchain that does not match the Kernel version of an other machine where the program is to be run will lead to the following error when starting the program:

FATAL: kernel too old

What is the reason? From time to time, new features get included in the Linux Kernel that are then used by the glibc to implement glibc's functions. Whether such a feature is available and can be used or not is decided when compiling glibc, not at runtime. One example is the introduction of the futex, the fast userspace mutex, in the Linux Kernel. It allows glibc to implement the well known pthread mutex with less context switches. The API used by the developer does not change at all, a pthread_mutex_t is still the same, no source code needs to be changed. Here again: compiling against a new Linux Kernel (on a new system) can prevent software from running on systems with an older kernel in such a case.

Impact of library versions

Most programs use shared libraries. When cross comiling such a program, it will meet these libraries two times: once at compile time when the program is finally linked and once at runtime when the program ist started. The libraries must match in both cases, of cause. When building all target libraries from source with a cross compiler and installing them on the target, then the final application was linked against exactly that library it is run with. In this case, you can choose your favourite version for the cross compiler. But if you use an of-the-shelf Linux distribution on the target, like Raspbian on the RaspberryPi, you have to use a cross compiler that matches the toolchain that was used to build all the Raspbian libraries. Easiest way is to use the same version. The standard C and C++ library (glibc,eglibc or uClibc and libstdc++) will be build together with the compiler. The Sstandard C library glibc is a serpate package and can be combined with different gcc versions, but the stdandard C++ library 'libstdc++' is part of the g++ package, so you cannot easily combine other versions thatn packaged.

• To get the same version of libstdc++ as on the target you need to use the same version of g++ as on the target for cross compiling.
• gcc and g++ is the same compiler, so the gcc version is also specified by libstdc++
• To avoid ELF symbol versioning conflicts, use the same version of glibc/eglib for the cross compiler as used by the target

Just using the newest GCC version as cross compiler works for building small "hello world"-like programs as the use nearly no libraries or statically linked programs. But if you intend to use some more advanced libraries like libavcodec, libavformat and possibly also C++, like boost, Qt etc. get those versions right!