7. Advanced Topics

7.1. Security

The RAUC bundle format consists of a squashfs image containing the images and the manifest, which is followed by a public key signature over the full image. This signature is stored in the CMS (Cryptographic Message Syntax, see RFC5652) format. Before installation, the signature is verified against the keyring already stored on the system.

We selected the CMS to avoid designing and implementing our own custom security mechanism (which often results in vulnerabilities). CMS is well proven in S/MIME and has widely available implementations, while supporting simple and as well as complex PKI use-cases (certificate expiry, intermediate CAs, revocation, algorithm selection, hardware security modules…) without additional complexity in RAUC itself.

RAUC uses OpenSSL as a library for signing and verification of bundles. An PKI with intermediate CAs for the unit tests is generated by the test/openssh-ca.sh shell script, which may also be useful as an example for creating your own PKI.

In the following sections, general CA configuration, some use-cases and corresponding PKI setups are described.

7.1.1. CA Configuration

OpenSSL uses a openssl.cnf file to define paths to use for signing, default parameters for certificates and additional parameters to be stored during signing. Configuring a CA correctly (and securely) is a complex topic and obviously exceeds the scope of this documentation. As a starting point, the OpenSSL manual pages (espcially ca, req, x509, cms, verify and config) and Stefan H. Holek’s pki-tutorial are useful.

7.1.2. Single Key

You can use openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes to create a key and a self-signed certificate. While you can use RAUC with these, you can’t:

  • replace expired certificates without updating the keyring
  • distinguish between development versions and releases
  • revoke a compromised key

7.1.3. Simple CA

By using the (self-signed) root CA only for signing other keys, which are used for bundle signing, you can:

  • create one key per developer, with limited validity periods
  • revoke keys and ship the CRL (Certificate Revocation List) with an update

With this setup, you can reduce the impact of a compromised developer key.

7.1.4. Separate Development and Release CAs

By creating a complete separate CA and bundle signing keys, you can give only specific persons (or roles) the keys necessary to sign final releases. Each device only has one of the two CAs in its keyring, allowing only installation of the corresponding updates.

While using signing also during development may seem unnecessary, the additional testing of the whole update system (RAUC, bootloader, migration code, …) allows finding problems much earlier.

7.2. Data Storage and Migration

Most systems require a location for storing configuration data such as passwords, ssh keys or application data. When performing an update, you have to ensure that the updated system takes over or can access the data of the old system.

7.2.1. Storing Data in The Root File System

In case of a writeable root file system, it often contains additional data, for example cryptographic material specific to the machine, or configuration files modified by the user. When performing the update, you have to ensure that the files you need to preserve are copied to the target slot after having written the system data to it.

RAUC provides support for executing hooks from different slot installation stages. For migrating data from your old rootfs to your updated rootfs, simply specify a slot post-install hook. Read the Hooks chapter on how to create one.

7.2.2. Using Data Partions

Often, there are a couple of reasons why you don’t want to or cannot store your data inside the root file system:

  • You want to keep you rootfs read-only to reduce probability of corrupting it.
  • You have a non-writable rootfs such as SquashFS.
  • You want to keep your data separated from the rootfs to ease setup, reset or recovery.

In this case you need a separate storage location for your data on a different partition, volume or device.

If the update concept uses full redundant root file systems, there are also good reasons for using a redundant data storage, too. Read below about the possible impact on data migration.

To let your system access the separate storage location, it has to be mounted into your rootfs. Note that if you intend to store configurable system information on your data partition, you have to map the default Linux paths (such as /etc/passwd) to your data storage. You can do this by using:

  • symbolic links
  • bind mounts
  • an overlay file system

It depends on the amount and type of data you want to handle which option you should choose.

7.2.3. Application Data Migration

_images/data_migration.svg

Both a single and a redundant data storage have their advantages and disadvantages. Note when storing data inside your rootfs you will have a redundant setup by design and cannot choose.

The decision about how to set up a configuration storage and how to handle it depends on several aspects:

  • May configuration format change over different application versions?
  • Can a new application read (and convert) old data?
  • Does your infrastructure allow working on possibly obsolete data?
  • Enough storage to store data redundantly?
  • …

The basic advantages and disadvantages a single or a redundant setup implicate are listed below:

  Single Data Redundant Data
Setup easy assure using correct one
Migration no backup by default copy on update, migrate
Fallback tricky (reconvert data?) easy (old data!)

7.3. Updating the Bootloader

Updating the bootloader is a special case, as it is a single point of failure on most systems: The selection of which redundant system images should be booted cannot itself be implemented in a redundant component (otherwise there would need to be an even earlier selection component).

Some SoCs contain a fixed firmware or ROM code which already supports redundant bootloaders, possibly integrated with a HW watchdog or boot counter. On these platforms, it is possible to have the selection point before the bootloader, allowing it to be stored redundantly and updated as any other component.

If redundant bootloaders with fallback is not possible (or too inflexible) on your platform, you may instead be able to ensure that the bootloader update is atomic. This doesn’t support recovering from a buggy bootloader, but will prevent a non-bootable system caused by an error or power-loss during the update.

Whether atomic bootloader updates can be implemented depends on your SoC/firmware and storage medium. For example eMMCs have two dedicated boot partitions (see the JEDEC standard JESD84-B51 for details), one of which can be enabled atomically via configuration registers in the eMMC.

As a further example, the NXP i.MX6 supports up to four bootloader copies when booting from NAND flash. The ROM code will try each copy in turn until it finds one which is readable without uncorrectable ECC errors and has a correct header. By using the trait of NAND flash that interrupted writes cause ECC errors and writing the first page (containing the header) last, the bootloader images can be replaced one after the other, while ensuring that the system will boot even in case of a crash or power failure.

Currently, independent of whether you are able to update your bootloader with fallback, atomically or with some risk of an unbootable system, our suggestion is to handle updates for it outside of RAUC. The main reason is to avoid booting an old system with a new bootloader, as this combination is usually not tested during development, increasing the risk of problems appearing only in the field.

One possible approach to this is:

  • Store a copy of the bootloader in the rootfs.
  • Use RAUC only to update the rootfs. The combinations to test can be reduced by limiting which old versions are supported by an update.
  • Reboot into the new system.
  • On boot, before starting the application, check that the current slot is ‘sane’. Then check if the installed bootloader is older than the version shipped in the (new) rootfs. In that case:
    • Disable the old rootfs slot and update the bootloader.
    • Reboot
  • Start the application.

This way you still have fallback support for the rootfs upgrade and need to test only:

  • The sanity check functionality and the bootloader installation when started from old bootloader and new rootfs
  • Normal operation when started from new bootloader and new rootfs

The case of new bootloader with old rootfs can never happen, because you disable the old one from the new before installing a new bootloader.

If you need to ensure that you can fall back to the secondary slot even after performing the bootloader update, you should check that the “other” slot contains the same bootloader version as the currently running one during the sanity check. This means that you need to update both slots in turn before the bootloader is updated.

7.4. Updating Sub-Devices

Besides the internal storage, some systems have external components or sub-devices which can be updated. For example:

  • Firmware for micro-controllers on modular boards
  • Firmware for a system management controller
  • FPGA bitstreams (stored in a separate flash)
  • Other Linux-based systems in the same enclosure
  • Software for third-party hardware components

In many cases, these components have some custom interface to query the currently installed version and to upload an update. They may or may not have internal redundancy or recovery mechanisms as well.

Although it is possible to configure RAUC slots for these and let it call a script to perform the installation, there are some disadvantages to this approach:

  • After a fallback to an older version in an A/B scenario, the sub-devices may be running an incompatible (newer) version.
  • A modular sub-device may be replaced and still has an old firmware version installed.
  • The number of sub-devices may not be fixed, so each device would need a different slot configuration.

Instead, a more robust approach is to store the sub-device firmware in the rootfs and (if needed) update them to the current versions during boot. This ensures that the sub-devices are always running the correct set of versions corresponding to the version of the main application.

If the bootloader falls back to the previous version on the main system, the same mechanism will downgrade the sub-devices as needed. During a downgrade, sub-devices which are running Linux with RAUC in an A/B scenario will detect that the image to be installed already matches the one in the other slot and avoid unnecessary installations.

7.5. Migrating to an Updated Bundle Version

As RAUC undergoes constant development, it might be extended and new features or enhancements will make their way into RAUC. Thus, also the sections and options contained in the bundle manifest may be extended over time.

To assure a well-defined and controlled update procedure, RAUC is rather strict in parsing the manifest and will reject bundles containing unknown configuration options.

But, this does not prevent you from being able to use those new RAUC features on your current sytem. All you have to do is to perform an intermediate update:

  • Create a bundle containing a rootfs with the recent RAUC version, but not containing the new RAUC features in its manifest.
  • Update your system and reboot
  • Now you have a system with a recent RAUC version which is able to interpretate and appropriately handle a bundle with the latest options

7.6. Software Deployment

When designing your update infrastructure, you must think about how to deploy the updates to your device(s). In general, you have two major options: Deployment via storage media such as USB sticks or network-based deployment.

As RAUC uses signed bundles instead of e.g. trusted connections to enable update author verification, RAUC fully supports both methods with the same technique and you may also use both of them in parallel.

Some influential factors on the method to used can be:

  • Do you have network access on the device?
  • How many devices have to be updated?
  • Who will perform the update?

7.6.1. Deployment via Storage Media

This method is mainly used for decentralized updates of devices without network access (either due to missing infrastructure or because of security concerns).

To handle deployment via storage media, you need a component that detects the plugged-in storage media and calls RAUC to trigger the actual installation.

When using systemd, you could use automount units for detecting plugged-in media and trigger an installation.

7.6.2. Deployment via Deployment Server

Deployment over a network is especially useful when having a larger set of devices to update or direct access to these devices is tricky.

As RAUC focuses on update handling on the target side, it does not provide a deployment server out of the box. But if you do not already have a deployment infrastructure, there a few Open Source deployment server implementations available in the wilderness.

One of it worth being mentioned is hawkBit from the Eclipse IoT project, which also provides some strategies for rollout management for larger-scale device farms.

7.6.2.1. The RAUC hawkBit client

As a separate project, the RAUC development team provides a Python-based example application that acts as a hawkBit client via its REST DDI-API while controlling RAUC via D-Bus.

For more information and testing it, visit it on GitHub:

https://github.com/rauc/rauc-hawkbit

It is also available via pypi:

https://pypi.python.org/pypi/rauc-hawkbit/