world leader in high performance signal processing
Trace: » release-creation

Making a Toolchain Release

This page covers the release process for the Blackfin GNU Toolchain. It is mostly an internal checklist for developers, but other people may find it interesting.

Overview

You will need:

  • An old x86 (32bit) distribution
  • RPM tools
  • Access to the Blackfin source repository
  • Access to the Blackfin FTP server

Step by Step

Build distro

An old 32bit (x86) distribution is needed in order to create binary files that will be usable across a wide range of customer platforms. Historically, this has been SUSE 8 because it builds things with:

  • Glibc 2.2.5 (released Jan 2002)
  • GCC 3.2 (no shared libgcc_s.so)

This means the resulting toolchain should be usable on all Linux distros built around 2002. Further, the lack of a shared libgcc_s.so dependency provides gcc version independence.

The only other required libraries date back to similar time frames:

  • Ncurses 5 (libncurses.so.5)
  • Zlib (libz.so.1)

With some releases where the TCL/TK insight frontend is included, GDB will require:

  • X11 (libX11.so.6)

The remaining 3rd party libraries are included and compiled statically to avoid host requirements:

32bit vs 64bit

Most modern 64bit (x86_64) distros can execute 32bit code natively, so you shouldn't need to install a dedicated machine and/or a 32bit kernel. Simply use setarch i386 before chrooting into the old distro:

setarch i386 chroot /path/to/chroot

If your distro does not provide setarch, then try running just i386.

Setup

You can find a copy of such a distro here: http://blackfin.uclinux.org/uimages/suse8.tar.bz2

So find a directory with lots of space (say a few gigs) and run:

wget http://blackfin.uclinux.org/uimages/suse8.tar.bz2
tar xjf suse8.tar.bz2 --numeric-owner

Then you'll have an old distro in $PWD/suse/ that can be chrooted into when building things.

We'll refer to this as the build chroot from now on, but you won't need it right away.

Usage

A chroot will need certain directories mounted. A script to simplify this can be found in the toolchain rpm dir. Simply copy this into the chroot dir and run it.

SCM

Every major release is branched from SVN trunk. Every single release is then tagged from that branch.

Setup

First create the bare svn tree (replace the URL with whatever you need to access things). This avoids checking out all the branches and tags you don't care about while leaving the framework to create new ones.

svn co -N svn+ssh://$USER@sources.blackfin.uclinux.org/svn/toolchain
cd toolchain
svn up -N tags branches
svn up trunk

Branching

Now you're ready to actually branch things from trunk and commit it:

svn cp trunk branches/2011R1
svn commit -m 'branch 2011R1' branches/2011R1

Kernel Headers

Usually the toolchain build is given the entire kernel source, but for the release process, we cache just the required headers.

So first grab the equivalent version of the kernel source (meaning branch). Then (obviously replace </paths> with the actual relevant paths) export the required headers with:

mkdir branches/2011R1/kbuild_output
make -C </absolute/path/to/kernel/branch> headers_install \
    INSTALL_HDR_PATH=$PWD/branches/2011R1/kbuild_output/usr \
    KBUILD_OUTPUT=$PWD/branches/2011R1/kbuild_output
find branches/2010R1/kbuild_output -type f -name '.*' -exec rm {} +

Then you need to add that new directory to the branch:

svn add branches/2011R1/kbuild_output
svn commit -m 'import 2011R1 kernel headers to 2011R1 toolchain branch' branches/2011R1

Tagging

The branch is a step closer to the release sources, but we don't use it directly for an actual release. You have to tag the specific release:

svn cp branches/2011R1 tags/2011R1-RC1
svn commit -m 'tag 2011R1-RC1' tags/2011R1-RC1

RPM

The toolchain releases are created with RPM. Then the RPM is turned into all the other file formats (tar/deb/etc…).

Source Prep

First you will need to create tarballs of each subdirectory.

cd tags/2010R1-RC1
tar jcf bfin-gcc-4.3.tar.bz2 gcc-4.3 --exclude=.svn
tar jcf bfin-gcc-4.5.tar.bz2 gcc-4.5 --exclude=.svn
tar jcf gmp.tar.bz2 gmp --exclude=.svn
tar jcf mpfr.tar.bz2 mpfr --exclude=.svn
tar jcf mpc.tar.bz2 mpc --exclude=.svn
tar jcf binutils.tar.bz2 binutils-2.17 binutils-2.21 --exclude=.svn
tar jcf kbuild.tar.bz2 kbuild_output --exclude=.svn
tar jcf buildscript.tar.bz2 buildscript --exclude=.svn
tar jcf elf2flt.tar.bz2 elf2flt --exclude=.svn
tar jcf expat-2.0.1.tar.bz2 expat-2.0.1 --exclude=.svn
tar jcf libdsp.tar.bz2 libs --exclude=.svn
tar jcf ldr-utils.tar.bz2 ldr-utils --exclude=.svn
tar jcf uClibc.tar.bz2 uClibc --exclude=.svn
tar jcf libftdi.tar.bz2 libftdi-1.0 --exclude=.svn
tar jcf libusb.tar.bz2 libusb --exclude=.svn
tar jcf urjtag.tar.bz2 urjtag --exclude=.svn
tar jcf gdbproxy.tar.bz2 gdbproxy --exclude=.svn
tar jcf readline.tar.bz2 readline --exclude=.svn

Note that QEmu isn't currently included, as it doesn't build on the release build machine. Additionally, you will need the u-boot source tree. So get the entire branch and run:

tar jcf uboot.tar.bz2 -C </path/to/u-boot/branch> . --exclude=.svn --transform=s:^.:u-boot:

Build Prep

With all these files in hand, you're ready to create some RPMs. Move all the tarballs you just created into the build chroot.

mv *.tar.bz2 </path/to/chroot>/usr/src/rpm/SOURCES/

You probably need to copy some supplemental files as well:

cp distribution/rpm/*.diff </path/to/chroot>/usr/src/rpm/SOURCES/

Spec Prep

The RPM supplemental files all live in the toolchain svn at distribution/rpm/.

You will need the .spec files, but you will want to update the version info first. These are the fields you'll need to update:

Field Value Description Breakdown
Version 2011R1 Base version (chop off the leading two digits) 2011R1-RC12011R1
Release RC1 The RC number only 2011R1-RC1RC1

Then copy them to the build chroot:

cp distribution/rpm/blackfin-*.spec </path/to/chroot>/usr/src/rpm/SPECS/

Build

At this point you will need to switch to the build chroot or machine. So chroot or ssh into it to proceed.

From within the build machine, you want to run rpm to create the RPM packages:

rpmbuild -ba /usr/src/rpm/SPECS/blackfin-toolchain.spec

This will take a while as it's going to compile the whole Blackfin toolchain. If everything completes successfully, you'll end up with the RPMs in /usr/src/rpm/RPMS/<arch>/ and the SRPMs in /usr/src/rpm/SRPMS/.

You will want to gather all of the RPMs that start with blackfin-toolchain- and have the appropriate version number.

Filesystem Layout

If everything doesn't work, here are the misc paths you'll need to investigate things.

Path Description
/usr/src/rpm/SPECS/ The .spec files live here
/usr/src/rpm/SOURCES/ The .spec file searches for source tarballs here
/usr/src/rpm/BUILD/<package>/ The source tarballs are unpacked/compiled here
/opt/uClinux/ The toolchain is installed here as it builds
/usr/src/rpm/RPMS/ The final binary RPMs are stored here
/usr/src/rpm/SRPMS/ The final source RPMs are stored here

More specific paths can be discovered by looking at the output from the building process. The BuildToolChain script shows all the relevant paths.

Website

Once the packages have been created, they need to be posted to the website for people to access.

FTP

Upload the RPMs and SRPMs via anonymous ftp to blackfin.uclinux.org and place them in the /toolchain/ directory.

Website

Start with the toolchain FRS admin page.

Click Add Release and fill out the form like so:

Field Contents
Release Name Something like 2011R1-RC1
Release Notes URL to the release notes in the wiki like http://docs.blackfin.uclinux.org/doku.php?id=toolchain:release-notes:2011r1
Changes Any changes to the last RC release in this series (if you feel like it)
Released Do not check
Development Status Use alpha or beta as appropriate
Project Upload Files Check all the relevant files to add to the release

Once we've settled on an RC for release, we will (at a later point in time), edit that release and check the Released box and change the Development Status to Mature.

Other Formats

During the release process, many RC versions are made before the “final” one. Alternative formats are not bothered with until the “final” RC has been decided (to avoid wasting time and space).

RPM Repos

A few repositories are maintained so RPM package managers can automatically download/upgrade their toolchains without having to download files and running `rpm` manually. These are updated on the server with the build-toolchain-rpm-repo script. The repo exists at /var/lib/gforge/filesystem/distros/rpm/.

Tarballs

The tarballs are created from the RPMs using the rpm2targz package. It can be found here. But you shouldn't need to worry about that as typically the RPMs are turned into tarballs on the server.

We do something like:

cd /var/ftp/toolchain
find /var/lib/gforge/filesystem/frsrelease/ -name '*-11r1-1.*.rpm' -exec rpm2tarbz2 {} +

Now these files can be added via the FRS admin page.

Debs

The .debs too are created on the server. This is because they are not downloaded directly by people, but rather used via the apt package system.

These file are created with the build-toolchain-debs script before being posted to the repo in /var/lib/gforge/filesystem/distros/debian/.

Windows

See the dedicated Windows page.

OS X

See the dedicated OS X page.