SRC_URI variable: Source code and patches

All recipies need to contain a definition of SRC_URI. It determines what files and source code is needed and where that source code should be obtained from. This includes patches to be applied and basic files that are shipped as part of the meta-data for the package.

A typical SRC_URI contains a list of URL's, patches and files as shown in this example from quagga:

SRC_URI = "http://www.quagga.net/download/quagga-${PV}.tar.gz \
           file://ospfd-no-opaque-lsa-fix.patch;patch=1 \
           file://fix-for-lib-inpath.patch;patch=1 \
           file://quagga.init \
           file://quagga.default \
           file://watchquagga.init \
           file://watchquagga.default"

All source code and files will be placed into the work directory, ${WORKDIR}, for the package. All patches will be placed into a patches subdirectory of the package source directory, ${S}, and then automatically applied to the source.

Before downloading from a remote URI a check will be made to see if what is to be retrieved is already present in the download source directory, ${DL_DIR}, along with an associated md5 sum. If the source is present in the downloaded sources directory and the md5 sum matches that listed in the associated md5 sum file, then that version will be used in preference to retrieving a new version . Any source that is retrieved from a remote URI will be stored in the download source directory and an appropriate md5 sum generated and stored alongside it.

Each URI supports a set of additional options. These options are tag/value pairs of the form "a=b" and are semi-colon separated from each other and from the URI. The follow examples shows two options being included, the patch and pnum options:

file://ospfd-no-opaque-lsa-fix.patch;patch=1;pnum=2

The supported methods for fetching source and files are:

http, https, ftps

Used to download files and source code via the specified URL. These are fetched from the specified location using wget.

file

Used for files that are included locally in the meta-data. These may be plain files, such as init scripts to be added to the final package, or they may be patch files to be applied to other source.

cvs

Used to download from a CVS repository.

svn

Used to download from a subversion repository.

git

Used to download from a git repository.

When source code is specified as a part of SRC_URI it is unpacked into the work directory, ${WORKDIR}. The unpacker recognises several archive and compression types and for these it will decompress any compressed files and extract all of the files from archives into the work directory. The supported types are:

.tar

Tar archives which will be extracted with "tar x --no-same-owner -f <srcfile>".

.tgz, .tar.gz, tar.Z

Gzip compressed tar archives which will be extracted with "tar xz --no-same-owner -f <srcfile>".

.tbz, .tar.bz2

Bzip2 compressed tar archives which will be extracted with "bzip2 -dc <srcfile> | tar x --no-same-owner -f -".

.gz, .Z, .z

Gzip compressed files which will be decompressed with "gzip -dc <srcfile> > <dstfile>".

.bz2

Bzip2 compressed files which will be decompressed with "bzip2 -dc <srcfile> > <dstfile>".

.zip

Zip archives which will be extracted with "unzip -q <srcfile>".

The downloading of the source files occurs in the fetch task, the unpacking and copying to the work directory occurs in the unpack task and the applying of patches occurs in the patch task.

http/https/ftp (wget)

The wget fetcher handles http, https and ftp URLs.

http://www.quagga.net/download/quagga-${PV}.tar.gz

Supported options:

md5sum

If an md5sum is provided then the downloaded files will only be considered valid if the md5sum of the downloaded file matches the md5sum option provided.

Related variables:

MIRRORS

Mirrors define alternative locations to download source files from. See the mirror section below for more information.

DL_DIR

The downloaded files will be placed in this directory with the name exactly as supplied via the URI.

file: for patches and additional files

The file URI's are used to copy files, included as part of the package meta data, into the work directory to be used when building the package. Typical use of the file URI's is to specify patches that be applied to the source and to provide additional files, such as init scripts, to be included in the final package.

The following example shows the specification of a patch file:

file://ospfd-no-opaque-lsa-fix.patch;patch=1

Patch files are be copied to the patches subdirectory of the source directory, ${S}/patches, and then applied from the source directory. The patches are searched for along the path specified via the file path variable, ${FILESPATH}, and if not found the directory specified by the file directory variable, ${FILEDIR}, is also checked.

The following example shows the specification of a non-patch file. In this case it's an init script:

file://quagga.init

Non-patch files are copied to the work directory, ${WORKDIR}. You can access these files from within a recipe by referring to them relative to the work directory. The following example, from the quagga recipe, shows the above init script being included in the package by copying it during the install task:

do_install () {
    # Install init script and default settings
    install -m 0755 -d ${D}${sysconfdir}/default ${D}${sysconfdir}/init.d ${D}${sysconfdir}/quagga
    install -m 0644 ${WORKDIR}/quagga.init ${D}${sysconfdir}/init.d/quagga
...

Supported options:

patch

Used as "patch=1" to define this file as a patch file. Patch files will be copied to ${S}/patches and then applied to source from within the source directory, ${S}.

pnum

By default patches are applied with the "-p 1" parameter, which strips off the first directory of the pathname in the patches. This option is used to explicitly control the value passed to "-p". The most typical use is when the patches are relative to the source directory already and need to be applied using "-p 0", in which case the "pnum=0" option is supplied.

cvs

The cvs fetcher is used to retrieve files from a CVS repository.

    cvs://anonymous@cvs.sourceforge.net/cvsroot/linuxsh;module=linux;date=20051111

A cvs URI will retrieve the source from a cvs repository. Note that use of the date= to specify a checkout for specified date. It is preferable to use either a date= or a tag= option to select a specific date and/or tag from cvs rather than leave the checkout floating at the head revision.

Supported options:

module

The name of a module to retrieve. This is a required parameter and there is no default value.

tag

The name of a cvs tag to retrieve. Releases are often tagged with a specific name to allow easy access. Either a tag or a date can be specified, but not both.

date

The date to retrieve. This requests that files as of the specified date, rather then the current code or a tagged release. If no date or tag options are specified, then the date is set to the current date. The date is of any form accepted by cvs with the most common format being "YYYYMMDD".

method

The method used to access the repository. Common options are "pserver" and "ext" (for cvs over rsh or ssh). The default is "pserver".

rsh

The rsh command to use with the "ext" method. Common options are "rsh" or "ssh". The default is "rsh".

Related variables:

CVS_TARBALL_STASH

Used to specifies a location to search for pre-generated tar archives to use instead of accessing cvs directly.

CVSDIR

The directory in which the cvs checkouts will be performed. The default is ${DL_DIR}/cvs.

DL_DIR

A compressed tar archive of the retrieved files will be placed in this directory. The archive name will be of the form: "<module>_<host>_<tag>_<date>.tar.gz". Path separators in module will be replaced with full stops.

svn

The svn fetcher is used to retrieve files from a subversion repository.

    svn://svn.xiph.org/trunk;module=Tremor;rev=4573;proto=http

Supported options:

module

The name of a module to retrieve. This is a required parameter and there is no default value.

rev

The revision to retrieve. Revisions in subversion are integer values.

proto

The method to use to access the repository. Common options are "svn", "svn+ssh", "http" and "https". The default is "svn".

rsh

The rsh command to use with using the "svn+ssh" method. Common options are "rsh" or "ssh". The default is "ssh".

Related variables:

SVNDIR

The directory in which the svn checkouts will be performed.. The default is ${DL_DIR}/svn.

DL_DIR

A compressed tar archive of the retrieved files will be placed in this directory. The archive name will be of the form: "<module>_<host>_<path>_<revn>_<date>.tar.gz". Path separators in path and module will be replaced with full stops.

git

The git fetcher is used to retrieve files from a git repository.

  SRC_URI = "git://www.denx.de/git/u-boot.git;protocol=git;tag=${TAG}"

Supported options:

tag

The tag to retrieve. The default is "master".

protocol

The method to use to access the repository. Common options are "git" and "rsync". The default is "rsync".

Related variables

GITDIR

The directory in which the git checkouts will be performed. The default is ${DL_DIR}/git.

DL_DIR

A compressed tar archive of the retrieved files will be placed in this directory. The archive name will be of the form: "git_<host><mpath>_<tag>.tar.gz". Path separators in host will be replaced with full stops.

Mirrors

The support for mirror sites enables spreading the load over sites and allows for downloads to occur even when one of the mirror sites are unavailable.

Default mirrors, along with their primary URL, include:

GNU_MIRROR

ftp://ftp.gnu.org/gnu

DEBIAN_MIRROR

ftp://ftp.debian.org/debian/pool

SOURCEFORGE_MIRROR

http://heanet.dl.sourceforge.net/sourceforge

GPE_MIRROR

http://handhelds.org/pub/projects/gpe/source

XLIBS_MIRROR

http://xlibs.freedesktop.org/release

XORG_MIRROR

http://xorg.freedesktop.org/releases

GNOME_MIRROR

http://ftp.gnome.org/pub/GNOME/sources

FREEBSD_MIRROR

ftp://ftp.freebsd.org/pub/FreeBSD

GENTOO_MIRROR

http://distro.ibiblio.org/pub/linux/distributions/gentoo/distfiles

APACHE_MIRROR

http://www.apache.org/dist

When creating new recipes this mirrors should be used when you wish to use one of the above sites by referring to the name of the mirror in the URI, as show in this example from flex:

SRC_URI = "${SOURCEFORGE_MIRROR}/lex/flex-2.5.31.tar.bz2

You can manually define your mirrors if you wish to force the use of a specific mirror by exporting the appropriate mirrors in local.conf with them set to the local mirror:

export GNU_MIRROR = "http://www.planetmirror.com/pub/gnu"
export DEBIAN_MIRROR = "http://mirror.optusnet.com.au/debian/pool"
export SOURCEFORGE_MIRROR = "http://optusnet.dl.sourceforge.net/sourceforge"

Mirrors can be extended in individual recipes via the use of MIRRORS_prepend or MIRRORS_append. Each entry in the list contains the mirror name on the left-hand side and the URI of the mirror on the right-hand side. The following example from libffi shows the addition of two URI for the "${GNU_MIRROR}/gcc/" URI:

MIRRORS_prepend () {
  ${GNU_MIRROR}/gcc/  http://gcc.get-software.com/releases/
  ${GNU_MIRROR}/gcc/  http://mirrors.rcn.net/pub/sourceware/gcc/releases/
}

Manipulating SRC_URI

Sometimes it is desirable to only include patches for a specific architecture and/or to include different files based on the architecture. This can be done via the SRC_URI_append and/or SRC_URI_prepend methods for adding additional URI's based on the architecture or machine name.

In this example from glibc, the patch creates a configuration file for glibc, which should only be used or the sh4 architecture. Therefore this patch is appended to the SRC_URI, but only for the sh4 architecture. For other architectures it is ignored:

# Build fails on sh4 unless no-z-defs is defined
SRC_URI_append_sh4 = " file://no-z-defs.patch;patch=1"

Source distribution (src_distribute_local)

In order to obtain a set of source files for a build you can use the src_distribute_local class. This will result in all the files that were actually used during a build being made available in a seperate directory and therefore they can be distributed with the binaries.

Enabling this option is as simple as activating the functionality by including the required class in one of your configuration files:

SRC_DIST_LOCAL = "copy"
INHERIT += "src_distribute_local"

Now during a build each recipe which has a LICENSE that mandates source availability, like the GPL, will be placed into the source distribution directory, ${SRC_DISTRIBUTEDIR}, after building.

There are some options available to effect the option

SRC_DIST_LOCAL

Specifies if the source files should be copied, symlinked or moved and symlinked back. The default is "move+symlink".

SRC_DISTRIBUTEDIR

Specifies the source distribution directory - this is why the source files that was used for the build are placed. The default is "${DEPLOY_DIR}/sources".

The valid values for SRC_DIST_LOCAL are:

copy

Copies the files to the downloaded sources directory into the distribution directory.

symlink

Symlinks the files from the downloaded sources directory into the distribution directory.

move+symlink

Moves the files from the downloaded sources directory into the distribution directory. Then creates a symlink in the download sources directory to the moved files.