We all have used linux-powered systems at some point in our lives. In fact, most of us use such systems on a daily basis and some of us use them almost all the time!
The linux kernel is used in a lot of gadgets around us – our smartphones, tablet computers, personal computers, medical instruments, car infotainment systems, GPS navigation systems (replaced mostly by smartphones today), set-top boxes, Wi-Fi routers and so on. A lot of items that we don’t use directly but are used in systems that we may interact daily also use linux kernels like industrial gateway systems, telecommunication equipments, a lot of data servers across the globe, etc. But all these systems don’t use the same processors! Some of these are built using extremely powerful multi-core processors, some use a simpler processor at their heart.
The sheer volume of systems around us that are linux-powered tells us that there must be something that makes it easy for the manufacturers and designers of these systems to use it, isn’t it? Think of it – if it wasn’t easy to use it, someone else somewhere in the world would have designed an alternative system. That’s just how the tech world works!
One of the key steps in being easily able to deploy linux-based systems is to be able to make system images that use linux kernel – easily. Considering that such systems are complex and support a wide variety of processors, making it easy sounds like a humongous task… enter the Yocto Project!
What the Yocto Project is not…
Before we see what the Yocto Project is, let us know what it is not.
The Yocto Project is not an SDK that you can use for your hardware machine – it can be used to build one!
The Yocto Project is not a system binary image that you can deploy on your hardware machine – it can be used to build one!
The Yocto Project is not a linux distribution that you can use to install to your hardware machine – it can be used to build an extremely tailored one for your resource-constrained hardware machine.
What the Yocto Project is…
The Yocto Project is an open-source collaboration project that helps developers create custom Linux-based systems for embedded products, regardless of the hardware architecture. The project provides a flexible set of tools and a space where embedded developers worldwide can share technologies, software stacks, configurations and best practices which can be used to create tailored Linux images for embedded devices.
The Yocto Project combines, maintains and validates three key development elements.
- A set of integrated tools to make working with embedded Linux successful, including tools for automated building and testing, processes for board support and license compliance, and component information for custom Linux-based embedded operating systems
- A reference embedded distribution (called Poky)
- The OpenEmbedded build system, co-maintained with the OpenEmbedded Project
But what… what is Poky?
Poky (pronounced Pock-ee) is a reference embedded distribution and a reference test configuration created to:
- Provide a base level functional distro which can be used to illustrate how to customize a distribution
- To test the Yocto Project components, Poky is used to validate Yocto Project
- As a vehicle for users to download Yocto Project. Poky is not a product level distro, but a good starting point for customization. Poky is an integration layer on top of oe-core.
But what… what is oe-core?
oe-core or OpenEmbedded-Core is meta-data comprised of foundation recipes, classes and associated files that are meant to be common among many different OpenEmbedded-derived systems, including the Yocto Project. It is a curated subset of an original repository developed by the OpenEmbedded community which has been pared down into a smaller, core set of continuously validated recipes resulting in a tightly controlled and a quality-assured core set of recipes.
That was quite a mouthful, no? Simply put… oe-core is a quality assured core foundational recipes that Poky uses to be able to generate a good and working base linux image.
We just said that oe-core is a set of recipes.
But what… what is a recipe?
Recipe is the most common form of metadata. A recipe will contain a list of settings and tasks (instructions) for building packages which are then used to build the binary image. A recipe describes where you get source code and which patches to apply. Recipes describe dependencies for libraries or for other recipes, as well as configuration and compilation options.
They are stored in layers. In fact, the layered nature of Poky is what makes it extremely scalable, versatile and easy to adapt to a variety of systems. For example, you will have all recipes pertaining to networking in one layer, all recipes related to your application in another, a dedicated layer for your graphics subsystems, and so on!
Similarly, there are files called configuration files. These are files which hold global definitions of variables, user defined variables and hardware configuration information. They tell the build system what to build and put into the image to support a particular platform.
Configuration Files and Recipes are often referred to as the metadata in the Poky build system. Other than these, commands and data that are used to build the image using the recipes and configuration files also constitute metadata.
We have oe-core which contains the validated metadata for a good image. But, what do you do with these recipes? Enter BitBake… (by the way, these cooking reference are almost funny.. wait till you hear of Toaster!)
But what… what is Bitbake?
Bitbake is a scheduler and execution engine which parses instructions (recipes) and configuration data. It then creates a dependency tree to order the compilation, schedules the compilation of the included code, and finally, executes the building of the specified, custom Linux image (distribution). BitBake is a make-like build tool. BitBake recipes specify how a particular package is built. They include all the package dependencies, source code locations, configuration, compilation, build, install and remove instructions.
During the build process dependencies are tracked and native or cross-compilation of the package is performed. As a first step in a cross-build setup, the framework will attempt to create a cross-compiler toolchain suited for the target platform.
Remember we talked about layers above? The BitBake parser is the one that ensures that a layer at the top can override settings in the lower layer thus avoiding any conflicts as the parser moves down the layers!
The BitBake engine uses the metadata to create a build output called packages. These packages come together to give what is the ultimate goal – the final image.
Below is a good high-level representation of the the Yocto Project components:
Yocto – Development Workflow
So, we get it. There are a lot of components that play a significant role in the creation of the image? But what does the overall workflow look like? Let’s find out.
- Step 1: It all starts with … yes you guessed it..getting the source code? No! First, the developer has to decide the various high-level configurations like what machine the image is to be built for, any special configuration data needed for the build, types of images to be built, etc.
- Step 2: Once the configuration is ready, next comes the source code. The source code can be in the form of tarballs, fetched from project releases or from Git or SVN or even be locally added to the workspace.
- Step 3: Often, there may be a need to apply patches to the source code to tailor the software component to the desired machine or this specific image. These patches are applied next.
- Step 4: What do you do with source code? You build! That indeed is the next step. Most common build tools like autotools, cmake, etc. are supported. Bitbake takes care of doing the necessary configuration and compilation.
- Step 5: The outputs of this build process are then placed into a temporary staging area where the packaging is done like .deb, .rpm, .ipk, etc. This packaging info is one of the configurations specified at the very beginning before we obtained the source code.
- Step 6: Next comes the QA process. Although it is incorrect to say that this is the first stage where the QA happens – it is more correct to say that QA activities happen throughout the process – especially during the BitBake engine’s operation.
- Step 7: Once the binary packages are ready, all that is necessary is to create a package feed that is suitable for the image that is requested.
- Step 8: This package feed culminates with the creation of the final image! Easy-peasy! By the way, a linux image is not all that the build process can create. The build process can also optionally be used to generate an SDK that can be used to develop and build applications for the machine that is running the very image you generate and load on that machine. This is extremely handy and takes away tons of complications that may result from not using a tailored SDK – especially in a diverse team of developers!
This was a brief introduction to the world of Yocto – hope this was useful.
In the next post, we see how to prepare a host computer to use the Yocto Project. See you, next time!
14 thoughts on “Yocto: Part 1 – A Definitive Introduction”