Linux

Toolchain From Scratch

About

“Toolchain From Scratch” (TFS) is a set of Makefiles and patches to build a toolchain, further programs and libraries from scratch. The compilers support the C and C++ programming languages. The build procedure is based on the build of the temporary system in “Linux From Scratch” (LFS). TFS aims at a standalone toolchain that is independent from the build system on which it was built and that can coexist with the toolchain(s) on the build system. TFS is intended to teach how to build a toolchain and to provide most recent compilers and libraries for testing purposes.

TFS up to version 3.6 has been successfully run on a x86_64 (64-bit Intel) system under Debian 5.0 and TFS version 3.7 as well as 3.8 under Debian 6.0.

WARNING

TFS is not intended to be used for or on production systems. TFS itself and toolchains built with TFS can contain serious bugs and security holes. The author of TFS will not provide any patches for bugs or security holes found in the programs of the toolchain. The user is responsible for that on his own.

Download

TFS can be downloaded from here:

toolchain_from_scratch_3.8.0.tar.gz

It is published under the terms of the GNU GPL v3 (see http://www.gnu.org/licenses/).

Prerequisites

Note: It is strongly recommended to create a new user for running TFS and the programs from the TFS toolchain, respectively. This way the risk of damaging the build system due to bugs in either TFS or a program in the toolchain can be minimized.

The build system must have a linux kernel with version >= 2.6.0. Additionally, TFS requires several tools installed on the build system, beyond others, the GCC C compiler, binutils, tar, gzip, bzip2, bash, make, autoconf, automake etc. A minimalist test for required tools is run at the very beginning of TFS. The list of these tools is stored in $(TOOLS_HOST) in make.config.

Furthermore, TFS requires source packages of the programs that shall be built. These packages have to be downloaded from the respective project websites either as .tar.gz or .tar.bz2 file and stored in $(TOOLS_SRCDIR) (defaults to sources directory in the directory with TFS Makefiles; see make.config). The required packages are defined by the variables named $(TOOL_PACKAGENAME) in the respective section headed "TFS VERSION --- files and patches" in make.config.

All the required patches are shipped with TFS and stored in the $(TOOLS_PATCHDIR) directory (defaults to patches in the directory with TFS Makefiles). There is no need to change this directory.

During the build, the sources will be unpacked and compiled in directories under $(TOOLS_TEMPDIR)/build and temporarily installed and packaged under $(TOOLS_TEMPDIR)/packages. $(TOOLS_TEMPDIR) defaults to the directory with TFS Makefiles. A build of a toolchain for the i686 or x86_64 target requires almost 30 GB of disk space in $(TOOLS_TEMPDIR) (for TFS version 3.7)! You might want to change this directory (see below).

The installation directory for the toolchain is defined by $(TOOLS_INSTDIR) in make.config. It defaults to /opt/tfs/VERSION/SYSTEMS/ and must be writable by the user running TFS. As it is strongly discouraged to run TFS as root, you will have to create $(TOOLS_INSTDIR) (or at least /opt/tfs) beforehand as root and chown it for the user running TFS.

WARNING: Do not use /, /usr, /usr/local, or any other default filesystem tree as installation directory of TFS! Use a new, empty directory as basis of the toolchain installation. If you override your distribution's C library, the whole system can become unusable!

Build of a toolchain

TFS can be run on i686 and x86_64 build system. The built toolchain is intended to run on the so-called host system and generates code for the target system. The build and target systems are configured by the variables $(TOOLS_BUILD) and $(TOOLS_TARGET), respectively. The host system is chosen by TFS depending on both the build and target system (see below).

Valid build system names are:

i686-pc-linux-gnu: for a i686 system (32-bit Intel)
x86_64-unknown-linux-gnu: for x86_64 system (64-bit Intel); this is the default build system

Valid target system names are:

avr: for Atmel's AVR system; the host system will be identical to the build system in this case
i686-tfs-linux-gnu: for i686 system; the host system will be identical to the target systems (= i686)
x86_64-tfs-linux-gnu: for x86_64 system; the host system will be identical to the target system (= x86_64); this is the default target system

The toolchain for the AVR target system will be a cross-compile toolchain, because the host system on which the compiler is run is different from the target system for which the code is generated.

The compilers for i686 and x86_64 are built in a cross-compile process even if the build and target are both i686 or x86_64, respectively. This is due to the changed target (= host) system names: pc or unknown, respectively, in the build system name are substituted by tfs in the target system names. This ensures that there will be no dependencies on the build system later. However, for the final toolchain the host system is identical to the target system and its compiler will not be a cross-compiler.

Note: If the target is i686 or x86_64, the build system must be able to run programs of the target (= host) system. The reason is that the compiler and programs of the toolchain from an early built stage will be used to build further programs and libraries of the toolchain.

After fulfilling all prerequisites, TFS can be started. Change to the directory with TFS Makefiles and run make.

Example 1: build of a toolchain for the AVR target system on a x86_64 build system.

$ make TOOLS_TEMPDIR=/scratch_dir \
  TOOLS_BUILD=x86_64-unknown-linux-gnu \
  TOOLS_TARGET=avr

Example 2: build of a toolchain for the i686 target system on a x86_64 build system.

$ make TOOLS_TEMPDIR=/scratch_dir \
  TOOLS_BUILD=x86_64-unknown-linux-gnu \
  TOOLS_TARGET=i686-tfs-linux-gnu

Example 3: build of a toolchain for the i686 target system on a i686 build system.

$ make TOOLS_TEMPDIR=/scratch_dir \
  TOOLS_BUILD=i686-pc-linux-gnu \
  TOOLS_TARGET=i686-tfs-linux-gnu

Example 4: build of a toolchain for the x86_64 target system using i686 tools (read notes below!).

$ make TOOLS_TEMPDIR=/scratch_dir \
  TOOLS_BUILD=i686-pc-linux-gnu \
  TOOLS_TARGET=x86_64-tfs-linux-gnu

As the build system must be able to run the programs from the newly created TFS toolchain, the last example will only work, if the computer system has a x86_64 hardware running a 64-bit kernel. However, the toolchain installed on the build system used to compile first parts of the new TFS toolchain must be of i686 type (as configured by TOOLS_BUILD=i686-pc-linux-gnu). The toolchain built by TFS will be pure x86_64 (due to TOOLS_TARGET=x86_64-tfs-linux-gnu).

If the build is successful, the toolchain will be installed in $(TOOLS_INSTDIR) and packages can be found in $(TOOLS_PACKDIR) (= $(TOOLS_TEMPDIR)/packages). The packages can be used to install the packages on other systems. For the i686 and x86_64 target system, packages ending in pass1 and pass2 will not be required anymore except for the linux-...-pass1.tar.bz2 package.

Installation of a toolchain

(This section may be skipped, if the toolchain has been built and installed before.)

The installation directory $(TOOLS_INSTDIR) must not coincide with an existing directory tree (especially / or /usr). Change to the root directory / and untar the packages.

# cd /
# tar -xjf binutils-....tar.bz2
# tar -xjf gcc-....tar.bz2
# tar -xjf glibc-....tar.bz2
# tar -xjf gmp-....tar.bz2
# tar -xjf linux-....tar.bz2
# tar -xjf mpfr-....tar.bz2
# tar -xjf ...

At this point you can chown the toolchain for root.

Configuration of a toolchain

This section only applies to toolchains for the i686 and x86_64 systems. For these systems you can configure parts of glibc.

Locales: Install the required locales for your system. E.g., set ${TOOLS_INSTDIR} appropriately under your shell and issue the following commands:

# mkdir -pv "${TOOLS_INSTDIR}/lib/locale"
# "${TOOLS_INSTDIR}/bin/localedef" -i de_DE -f UTF-8 de_DE
# "${TOOLS_INSTDIR}/bin/localedef" -i en_US -f ISO-8859-1 en_US
# "${TOOLS_INSTDIR}/bin/localedef" ...

NSSWITCH: Configure nsswitch by creating

File: `${TOOLS_INSTDIR}/etc/nsswitch.conf`

passwd: files
group: files
shadow: files

hosts: files dns
networks: files

protocols: files
services: files
ethers: files
rpc: files

Timezone: If you do not know your timezone, you can run

# "${TOOLS_INSTDIR}/bin/tzselect"

and answer some questions. To configure the timezone, e.g. Europe/Berlin, issue the command

# cp -v --remove-destination \
  "${TOOLS_INSTDIR}/share/zoneinfo/Europe/Berlin" \
  "${TOOLS_INSTDIR}/etc/localtime"

Dynamic Loader: Add all directories other than ${TOOLS_INSTDIR}/lib that should be searched for dynamic libraries to ${TOOLS_INSTDIR}/etc/ld.so.conf (one per line) and issue:

# "${TOOLS_INSTDIR}/sbin/ldconfig"

WARNING: Libraries in ${TOOLS_INSTDIR}/lib will be searched after the directories in ${TOOLS_INSTDIR}/etc/ld.so.conf. As a consequence it is usually not possible to add /lib or /usr/lib to ${TOOLS_INSTDIR}/etc/ld.so.conf. Additionally, never try to add ${TOOLS_INSTDIR}/lib to your system's dynamic loader configuration /etc/ld.so.conf!

Usage of a toolchain

To use the tools, you can add ${TOOLS_INSTDIR}/bin and ${TOOLS_INSTDIR}/sbin to your $PATH environmental variable:

$ export PATH="${TOOLS_INSTDIR}/bin:${TOOLS_INSTDIR}/sbin:${PATH}"

Be aware that headers are looked up in ${TOOLS_INSTDIR}/include by default and binaries compiled with the compilers of the toolchain use ${TOOLS_INSTDIR}/lib/ld-linux.so.2 as dynamic linker by default. Make sure that the required libraries are installed on any system that should run binaries created with the toolchain.