Yocto: Part 7 – Writing recipes for tarballs (local and remote)

Welcome to part 7 of this series of blog posts on Yocto. We have come a long way and if this is your first time here, we would highly recommend to go through the entire series linked below.

Yocto: Part 1 – A Definitive Introduction

Yocto: Part 2 – Setting up Ubuntu host

Yocto: Part 3 – Build & run your first ever image!

Yocto: Part 4 – Building a basic image for Raspberry Pi

Yocto: Part 5 – Creating & adding a new layer to your image

Yocto: Part 6 – Understanding and creating your first custom recipe

In this post, we talk about using source trees that are compressed as tarballs.

Why even use tarballs?

You will find that a lot of modern-day bitbake recipes use SCM (eg: git). However, the tarballs are not going anywhere. An overwhelming majority of recipes found today in poky and open-embedded have their source trees compressed as tarballs.

Tarballs are pretty easy to deal with as they consume a small storage space before they are expanded, the compression algorithm is universally used and bitbake is smart enough to deal with versioning with tarballs as a mature filename parser is already in place. In fact, a lot of source trees based on git generate releases in the form of tarballs that are then used in bitbake recipes.

Where do I place tarballs?

Bitbake can fetch tarballs from a remote URL that you provide to it. If that is not your thing, you can always place the tarball inside a folder named files located in the same directory as the recipe.

Both of these approaches are discussed below.

Structure of a recipe for tarballs

The structure of a recipe that uses tarballs is very similar to the recipe we wrote in the previous post where we wrote our very first recipe. The only minor difference is that the SRC_URI now points to the local tarball.

In the examples below, instead of adding a compilation step, we will use cmake to do the build for us. We talked about cmake in this post – do check it out if you are new to cmake and are looking for a crisp introduction to cmake.

Creating a recipe for a local tarball

We will create our recipe in three steps.

Step 1: Creating the recipe placeholder

In the previous post, we created a recipe named hwlocal inside our yocto layer named meta-ke that looks something like below.

• meta-ke
  • conf/
  • COPYING.MIT
  • README
  • recipes-example/
  • recipes-ke/
    • hwlocal/
      • files/
        • hello-world-local.c
      • hwlocal_0.1.bb

Now, let us create our local tarball recipe named hwtarlocal and create other resources required as highlighted below.

• meta-ke
  • conf/
  • COPYING.MIT
  • README
  • recipes-example/
  • recipes-ke/
    • hwlocal/
      • files/
        • hello-world-local.c
      • hwlocal_0.1.bb
    • hwtarlocal/
      • files/
        • <empty as of now>
      • hwtarlocal0.1.bb

Step 2: Preparing the archive

Let us now fetch the source code that we will use for this example. This source code is located here: https://github.com/sckulkarni246/yocto-test-apps

On any location on your PC, clone the source tree at this link by executing the below command.

$ git clone https://github.com/sckulkarni246/yocto-test-apps hwtarlocal-0.1

Now, let us create our tarball from the source code that we fetched.

$ tar -czvf hwtarlocal_0.1.tar.gz hwtarlocal-0.1

The purpose of appending the version (0.1) to the tarball (and the cloned repo) is to allow for easy versioning and reusability.

NOTE: If you plan to use any other name for the folder, the same should be explicitly provided in the bitbake recipe by specifying the value of the environment variable S like we did in the last post. More on this below where we talk about remote tarball! For now, let us specify the version and the names correctly!

To use the latest version of your tarball, all you need to to do is create a new recipe or simply change the name of an existing recipe. For example, let us say you have version 0.5 of your tarball. Simply renaming the recipe to hwtarlocal_0.5.bb will select this latest version without needing to change the content of the recipe.

Now, copy the hwtarlocal_0.1.tar.gz to the files/ folder of your recipe such that the meta-ke layer now looks something like this.

• meta-ke
  • conf/
  • COPYING.MIT
  • README
  • recipes-example/
  • recipes-ke/
    • hwlocal/
      • files/
        • hello-world-local.c
      • hwlocal_0.1.bb
    • hwtarlocal/
      • files/
        • hwtarlocal_0.1.tar.gz
      • hwtarlocal0.1.bb

Step 3: Writing the recipe

Let us now quickly write the recipe. As we mentioned before, the structure of this recipe is very similar to the hwlocal recipe we wrote previously. Also, note the use of cmake in this recipe. Like before, we install the binary output to the /usr/bin directory of the output image.

DESCRIPTION = "This is a simple Hello World recipe - extracts a local tarball and then uses CMake to build it"
HOMEPAGE = "https://kickstartembedded.com"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=45269dcabf49617cca580ad6878cbcd2"

SRC_URI = "file://hwtarlocal-${PV}.tar.gz"

inherit cmake

EXTRA_OECMAKE = ""

do_install() {
	install -d ${D}${bindir}
	install -m 0755 hello-world-git ${D}${bindir}
}

How did we get the md5sum of the license? There are two ways to do this.

You can manually calculate the md5sum of the license file named LICENSE and then paste it in the recipe. Or, you can put a dummy md5sum and then during the first bitbake run, bitbake will itself tell us that there is a mismatch and the actual md5sum should be so-and-so.

To manually calculate the md5sum of the LICENSE file, run the below command and copy the md5sum present in the output into your recipe.

$ md5sum LICENSE
45269dcabf49617cca580ad6878cbcd2  LICENSE

That’s it! Let us now try baking this recipe. If you don’t already have a build environment ready, you can follow the steps outlined in this post to set one up quickly.

NOTE: You will also need to ensure that the meta-ke layer is added into your build configuration. The simplest way to do that is by following the steps outlined in this post.

Now, try to bake your recipe by executing the below command.

$ bitbake hwtarlocal

You should see this recipe being baked successfully. Congratulations! You have created your very own recipe to work with a local tarball!

Let us now see how to do the same for a tarball located at a remote location.

Creating a recipe for a remote tarball

We will create this recipe in two steps as we don’t need the preparation step like the case where we used a local tarball.

Step 1: Creating the recipe placeholder

We will add to our meta-ke layer like before and create the resources needed for our new recipe named hwtarfetch.

• meta-ke
  • conf/
  • COPYING.MIT
  • README
  • recipes-example/
  • recipes-ke/
    • hwlocal/
      • files/
        • hello-world-local.c
      • hwlocal_0.1.bb
    • hwtarlocal/
      • files/
        • hwtarlocal_0.1.tar.gz
      • hwtarlocal0.1.bb
    • hwtarfetch/
      • hwtarfetch_0.1.bb

Step 2: Writing the recipe

Like before, we will use the same GitHub repo for our source code. This time, we will download one of the releases on this repo named test_release. The asset that we will download is named hwtarfetch_0.1.tar.gz.

Wait what?! We see two deviations from our standard case where the recipe is named something like recipe_name-recipe_version and the tarball and when the tarball is expanded, the folder name is also the same.

In our case:

1. The tarball’s name is hwtarfetch_0.1.tar.gz – notice the separator is _ and not -
2. The repo is named yocto-test-apps. This tells us that when the tarball expands, the folder name containing the source code would be yocto-test-apps and not hwtarfetch-0.1.

Let us address both of these issues in our recipe below.

DESCRIPTION = "This is a simple Hello World recipe - downloads a tarball and then uses CMake to build it"
HOMEPAGE = "https://kickstartembedded.com"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://LICENSE;md5=45269dcabf49617cca580ad6878cbcd2"

SRC_URI = "https://github.com/sckulkarni246/yocto-test-apps/archive/refs/tags/hwtarfetch_${PV}.tar.gz"
SRC_URI[sha256sum] = "4a6f509a2921d15f33d6117910b2edee86fd91bde9e1b56c74a6b718ef841c1b"

S = "${WORKDIR}/yocto-test-apps-${BPN}_${PV}"

inherit cmake

EXTRA_OECMAKE = ""

do_install() {
	install -d ${D}${bindir}
	install -m 0755 hello-world-git ${D}${bindir}
}

Notice that the SRC_URI is the address of the tarball. Also since we are deviating from the expected format of the tarball name, we will explicitly tell bitbake to use the provided value for S instead of using the default value.

In the previous post, we saw that the environment variable S is the location in the build directory where unpacked recipe source code resides. For any recipe, bitbake pre-populates this variable as WORKDIR. However, since we are using a different name, we explicitly tell bitbake about the path to be used. Also, we have specified the separator as an _ instead of a - in the same place.

There is one more configuration here that requires our attention. We have populated a variable called SRC_URI[sha256sum]. There are two ways to populate this – you can manually calculate the tarball’s SHA256 and add it to this variable or you can populate a dummy value here and then rely on bitbake to tell you the value you should put here.

To calculate the SHA256 sum of this tarball, simply download the tarball from here and execute the below command.

$ sha256sum yocto-test-apps-hwtarfetch_0.1.tar.gz
4a6f509a2921d15f33d6117910b2edee86fd91bde9e1b56c74a6b718ef841c1b  yocto-test-apps-hwtarfetch_0.1.tar.gz

Now, try baking this recipe exactly like we did above. You should see this go successfully. Congratulations! You have successfully written a recipe for fetching a remote tarball!

What’s next?

In the next post, we talk about writing recipes for fetching source tree from a git repository.

See you soon!