Hi all,

I've been looking at making product(s) which can do OTA updates, and how to structure the overall project.
A basic aim is to be able to checkout and build (assuming toolchain and environment is setup) with minimal commands, and also minimal manual 'configuration' changes required for a default build.
I'm not really feeling like I'm seeing a nice solution, so was wondering if anyone else has overcome a similar situation, or anyone has any ideas to progress.

Following, I've tried to summarize the problem & design as I see it to make sure I'm on the right track, and then I've got a few example commands to further illustrate where the problem is for me.

Any and all suggestions welcome.

Cheers,
Marc

-----

Overview:
The product acheives OTA updates, at a general level, by:
 - use mcuboot as the bootloader
 - app implements SMP service (eg mcumgr/smp_svr example)
 - initially program the product with a combination of the mcuboot + app
 - further updates can be done OTA, the app handles transmission and copying of the new firmware image to a partition on flash, then resets. the bootloader checks the new firmware image does some partition 'accounting' and then boots the new image.

Typical project elements/dependencies:
 - app for the product/device
  - board files are defined in the 'app' repo
 - common libs (eg for our GATT services which most our devices have)
 - zephyr framework
  - dependent modules (eg nordic_hal, segger)
 - mcuboot
  - (references the app board file)

This resembles either the T2 or T3 of west's documented tolopogies. [https://docs.zephyrproject.org/latest/guides/west/repo-tool.html#topologies-supported]

Build:
 - We want 2 primary build artifacts:
   - the app (configured as launched from a bootloader)
   - bootloader mcuboot
 - And then from this with can run a variety of commands to merge and/or sign the build artifacts.
 - As a bonus, perhaps another app configuration to make dev/debug easier where the bootloader is not used.

-----

West seems like the ideal tool to use for this, it boils down to a manifest repo (for project & dependencies checkout), and then a series of commands to build the various artifacts and merge & sign them. For example:

west init -m git@...:SomeManifestRepo.git
west build --board=theboard -d build-mcuboot -s mcuboot/boot/zephyr -- -DBOARD_ROOT=/abs/path/to/app
west build --board=theboard -d build-app -s app -- -DBOARD_ROOT=/abs/path/to/app
west sign ...

However, it would be nice to be able to glue this all together with some script etc to simplify the extra elements of the west commands ('theboard' 'BOARD_ROOT') etc. And here is where I become unsure of how this glue script/(c)makefile/etc fits into the topology of the project.. The natural place for the script to run from is the west installation folder itself (ie, the parent folder of all the subprojects) however the natural place for it to 'live' is in the 'manifest-repo'.

Alternatives:
 - copy 'glue' script into west installation folder ?
 - west commands ? in the manifest repo, or maybe another common lib etc.

Hi, I see you are using nrf.

Signing, building, and hex merging of multiple images in a single west command is supported
out-of-the box with the nRF Connect SDK, which uses a patched Zephyr.

So if not using vanilla Zephyr is an option then see:

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/samples/nrf9160/http_application_update/README.html#http-application-update-sample
https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/ug_multi_image.html

________________________________________
From: devel@... <devel@...> on behalf of Marc Reilly via Lists.Zephyrproject.Org <marc.reilly=gmail.com@...>
Sent: Monday, December 9, 2019 4:38 PM
To: devel@...
Cc: devel@...
Subject: [Zephyr-devel] Managing/structuring multi image/binary projects for (OTA updatable products etc)

Hi all,

I've been looking at making product(s) which can do OTA updates, and how to structure the overall project.
A basic aim is to be able to checkout and build (assuming toolchain and environment is setup) with minimal commands, and also minimal manual 'configuration' changes required for a default build.
I'm not really feeling like I'm seeing a nice solution, so was wondering if anyone else has overcome a similar situation, or anyone has any ideas to progress.

Following, I've tried to summarize the problem & design as I see it to make sure I'm on the right track, and then I've got a few example commands to further illustrate where the problem is for me.

Any and all suggestions welcome.

Cheers,
Marc

-----

Overview:
The product acheives OTA updates, at a general level, by:
- use mcuboot as the bootloader
- app implements SMP service (eg mcumgr/smp_svr example)
- initially program the product with a combination of the mcuboot + app
- further updates can be done OTA, the app handles transmission and copying of the new firmware image to a partition on flash, then resets. the bootloader checks the new firmware image does some partition 'accounting' and then boots the new image.

Typical project elements/dependencies:
- app for the product/device
- board files are defined in the 'app' repo
- common libs (eg for our GATT services which most our devices have)
- zephyr framework
- dependent modules (eg nordic_hal, segger)
- mcuboot
- (references the app board file)

This resembles either the T2 or T3 of west's documented tolopogies. [https://docs.zephyrproject.org/latest/guides/west/repo-tool.html#topologies-supported]

Build:
- We want 2 primary build artifacts:
- the app (configured as launched from a bootloader)
- bootloader mcuboot
- And then from this with can run a variety of commands to merge and/or sign the build artifacts.
- As a bonus, perhaps another app configuration to make dev/debug easier where the bootloader is not used.

-----

West seems like the ideal tool to use for this, it boils down to a manifest repo (for project & dependencies checkout), and then a series of commands to build the various artifacts and merge & sign them. For example:

west init -m git@...:SomeManifestRepo.git
west build --board=theboard -d build-mcuboot -s mcuboot/boot/zephyr -- -DBOARD_ROOT=/abs/path/to/app
west build --board=theboard -d build-app -s app -- -DBOARD_ROOT=/abs/path/to/app
west sign ...

However, it would be nice to be able to glue this all together with some script etc to simplify the extra elements of the west commands ('theboard' 'BOARD_ROOT') etc. And here is where I become unsure of how this glue script/(c)makefile/etc fits into the topology of the project.. The natural place for the script to run from is the west installation folder itself (ie, the parent folder of all the subprojects) however the natural place for it to 'live' is in the 'manifest-repo'.

Alternatives:
- copy 'glue' script into west installation folder ?
- west commands ? in the manifest repo, or maybe another common lib etc.

Hello Marc,

"Sebastian Boe via Lists.Zephyrproject.Org"
<Sebastian.Boe=nordicsemi.no@...> writes:

Hi, I see you are using nrf.

Signing, building, and hex merging of multiple images in a single west command is supported
out-of-the box with the nRF Connect SDK, which uses a patched Zephyr.

So if not using vanilla Zephyr is an option then see:

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/samples/nrf9160/http_application_update/README.html#http-application-update-sample
https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/ug_multi_image.html

I second Sebastian's suggestion to look at nRF Connect SDK (NCS).
It has useful features for building OTA binaries for nRF devices.

________________________________________
From: devel@... <devel@...> on behalf of Marc Reilly via Lists.Zephyrproject.Org <marc.reilly=gmail.com@...>
Sent: Monday, December 9, 2019 4:38 PM
To: devel@...
Cc: devel@...
Subject: [Zephyr-devel] Managing/structuring multi image/binary projects for (OTA updatable products etc)

Hi all,

I've been looking at making product(s) which can do OTA updates, and how to structure the overall project.
A basic aim is to be able to checkout and build (assuming toolchain and environment is setup) with minimal commands, and also minimal manual 'configuration' changes required for a default build.
I'm not really feeling like I'm seeing a nice solution, so was wondering if anyone else has overcome a similar situation, or anyone has any ideas to progress.

Following, I've tried to summarize the problem & design as I see it to make sure I'm on the right track, and then I've got a few example commands to further illustrate where the problem is for me.

Any and all suggestions welcome.

Cheers,
Marc

-----

Overview:
The product acheives OTA updates, at a general level, by:
- use mcuboot as the bootloader
- app implements SMP service (eg mcumgr/smp_svr example)
- initially program the product with a combination of the mcuboot + app
- further updates can be done OTA, the app handles transmission and copying of the new firmware image to a partition on flash, then resets. the bootloader checks the new firmware image does some partition 'accounting' and then boots the new image.

Typical project elements/dependencies:
- app for the product/device
- board files are defined in the 'app' repo
- common libs (eg for our GATT services which most our devices have)
- zephyr framework
- dependent modules (eg nordic_hal, segger)
- mcuboot
- (references the app board file)

This resembles either the T2 or T3 of west's documented tolopogies. [https://docs.zephyrproject.org/latest/guides/west/repo-tool.html#topologies-supported]

Build:
- We want 2 primary build artifacts:
- the app (configured as launched from a bootloader)
- bootloader mcuboot
- And then from this with can run a variety of commands to merge and/or sign the build artifacts.
- As a bonus, perhaps another app configuration to make dev/debug easier where the bootloader is not used.

-----

West seems like the ideal tool to use for this, it boils down to a manifest repo (for project & dependencies checkout), and then a series of commands to build the various artifacts and merge & sign them. For example:

west init -m git@...:SomeManifestRepo.git
west build --board=theboard -d build-mcuboot -s mcuboot/boot/zephyr -- -DBOARD_ROOT=/abs/path/to/app
west build --board=theboard -d build-app -s app -- -DBOARD_ROOT=/abs/path/to/app
west sign ...

However, it would be nice to be able to glue this all together with
some script etc to simplify the extra elements of the west commands
('theboard' 'BOARD_ROOT') etc.

No matter what you choose to use (mainline or NCS), though, you can
always tell 'west build' your default board like this:

west config build.board theboard

https://docs.zephyrproject.org/latest/guides/west/build-flash-debug.html#configuration-options

The build.board value will be used if you don't provide --board
(provided BOARD is also unset in the calling environment).

I thought about your BOARD_ROOT question. I've gotten other similar
requests recently, and I agree it would be useful, so I've proposed a
new "build.cmake-args" config option:

https://github.com/zephyrproject-rtos/zephyr/pull/21251

With the patch from #21251, you can save your BOARD_ROOT like this:

west config build.cmake-args -- -DBOARD_ROOT=/abs/path/to/app

Your one-time setup work would then be:

west init -m git@.../SomeManifestRepo.git
west update
west config build.board theboard
west config build.cmake-args -- -DBOARD_ROOT=/abs/path/to/app

Note that plain 'west config var val' sets configuration options locally
(just for your west installation). See the --global and --system options
in the 'west config' documentation for alternatives.

# build and flash
west build -d build-mcuboot -s mcuboot/boot/zephyr
west build -d build-app -s app
west sign ...

And here is where I become unsure of how this glue script/(c)makefile/etc fits into the topology of the project.. The natural place for the script to run from is the west installation folder itself (ie, the parent folder of all the subprojects) however the natural place for it to 'live' is in the 'manifest-repo'.

Alternatives:
- copy 'glue' script into west installation folder ?
- west commands ? in the manifest repo, or maybe another common lib etc.

I hope the above is what you're looking for.

Note 'west build' can only compile one Zephyr application at a time.
The NCS can build application and mcuboot binaries at once out of the
box, for reference.

Please test and comment in the PR if you get a chance.

Thanks,
Martí

Sorry for the double-post, but I realized that the latter half of my
inline responses might be easy to ignore, so I'm going to hang a lantern
on them with this email:

"Bolivar, Marti via Lists.Zephyrproject.Org"
<marti.bolivar=nordicsemi.no@...> writes:

From: devel@... <devel@...> on behalf of Marc Reilly via Lists.Zephyrproject.Org <marc.reilly=gmail.com@...>
However, it would be nice to be able to glue this all together with
some script etc to simplify the extra elements of the west commands
('theboard' 'BOARD_ROOT') etc.

No matter what you choose to use (mainline or NCS), though, you can
always tell 'west build' your default board like this:

west config build.board theboard

https://docs.zephyrproject.org/latest/guides/west/build-flash-debug.html#configuration-options

The build.board value will be used if you don't provide --board
(provided BOARD is also unset in the calling environment).

I thought about your BOARD_ROOT question. I've gotten other similar
requests recently, and I agree it would be useful, so I've proposed a
new "build.cmake-args" config option:

https://github.com/zephyrproject-rtos/zephyr/pull/21251

With the patch from #21251, you can save your BOARD_ROOT like this:

west config build.cmake-args -- -DBOARD_ROOT=/abs/path/to/app

Your one-time setup work would then be:

west init -m git@.../SomeManifestRepo.git
west update
west config build.board theboard
west config build.cmake-args -- -DBOARD_ROOT=/abs/path/to/app

Note that plain 'west config var val' sets configuration options locally
(just for your west installation). See the --global and --system options
in the 'west config' documentation for alternatives.

# build and flash
west build -d build-mcuboot -s mcuboot/boot/zephyr
west build -d build-app -s app
west sign ...

Please make sure you see this part ^^.

Martí

Hi Marc,

On Mon, Dec 09, 2019 at 07:38:53AM -0800, Marc Reilly wrote:

Hi all,

I've been looking at making product(s) which can do OTA updates, and
how to structure the overall project.
A basic aim is to be able to checkout and build (assuming toolchain
and environment is setup) with minimal commands, and also minimal
manual 'configuration' changes required for a default build.
I'm not really feeling like I'm seeing a nice solution, so was
wondering if anyone else has overcome a similar situation, or anyone
has any ideas to progress.

Following, I've tried to summarize the problem & design as I see it to
make sure I'm on the right track, and then I've got a few example
commands to further illustrate where the problem is for me.

Any and all suggestions welcome.

We faced similar problems when we had to make an IVI system that uses
bare metal + FreeRTOS + Linux in one SoC and requires a consistent build
for all of them. In the end we created our own meta-tool:

https://bobbuildtool.dev/

Now, Zephyr already came up with the west tool which solves some of the
problems and drives many Zephyr use cases better than Bob could at the
moment. Nonetheless I created some demo recipes a couple of weeks ago to
show how a multi-application-build could look like:

https://github.com/jkloetzke/zephyr-example-recipes

Now, I'm perfectly aware that it creates another layer of complexity.
The recipes effectively replace west to drive the build. But Bob is
meant to solve many other problems too that are constantly coming up
with such more-than-one-image embedded systems: variants management,
reliable incremental building, reproducibility, toolchain setup, Jenkins
integration and automatic artifact caching. Also building and running
unit tests from the same sources on the host in one command was a
requirement for us. The goal is to always execute just one build
command:

$ git clone <project>
$ cd <project>
$ bob build <image>

and start working from there on:

... hack on the source ...
bob build <image>
bob build <unit-tests>
git commit/push/pull
... repeat ...

We're currently expanding the public-visible recipes. There is another
example which shows how FreeRTOS and Linux share some code are built
together as AMP system:

https://github.com/BobBuildTool/bob-example-embedded

If your product stays in the nRF+mcuboot+zephyr world then sticking to
the nRF Connect SDK may be easier. Otherwise Bob might be worth a look
IMHO.

Regards,
Jan

Cheers,
Marc

-----

Overview:
The product acheives OTA updates, at a general level, by:
- use mcuboot as the bootloader
- app implements SMP service (eg mcumgr/smp_svr example)
- initially program the product with a combination of the mcuboot + app
- further updates can be done OTA, the app handles transmission and copying of the new firmware image to a partition on flash, then resets. the bootloader checks the new firmware image does some partition 'accounting' and then boots the new image.

Typical project elements/dependencies:
- app for the product/device
- board files are defined in the 'app' repo
- common libs (eg for our GATT services which most our devices have)
- zephyr framework
- dependent modules (eg nordic_hal, segger)
- mcuboot
- (references the app board file)

This resembles either the T2 or T3 of west's documented tolopogies. [https://docs.zephyrproject.org/latest/guides/west/repo-tool.html#topologies-supported]

Build:
- We want 2 primary build artifacts:
- the app (configured as launched from a bootloader)
- bootloader mcuboot
- And then from this with can run a variety of commands to merge and/or sign the build artifacts.
- As a bonus, perhaps another app configuration to make dev/debug easier where the bootloader is not used.

-----

West seems like the ideal tool to use for this, it boils down to a manifest repo (for project & dependencies checkout), and then a series of commands to build the various artifacts and merge & sign them. For example:

west init -m git@...:SomeManifestRepo.git
west build --board=theboard -d build-mcuboot -s mcuboot/boot/zephyr -- -DBOARD_ROOT=/abs/path/to/app
west build --board=theboard -d build-app -s app -- -DBOARD_ROOT=/abs/path/to/app
west sign ...

However, it would be nice to be able to glue this all together with some script etc to simplify the extra elements of the west commands ('theboard' 'BOARD_ROOT') etc. And here is where I become unsure of how this glue script/(c)makefile/etc fits into the topology of the project.. The natural place for the script to run from is the west installation folder itself (ie, the parent folder of all the subprojects) however the natural place for it to 'live' is in the 'manifest-repo'.

Alternatives:
- copy 'glue' script into west installation folder ?
- west commands ? in the manifest repo, or maybe another common lib etc.

The multi-image approach in NCS will not make it's way into mainline Zephyr because
it was not accepted upstream unfortunately.

________________________________________
From: Marc Reilly <marc.reilly@...>
Sent: Tuesday, December 10, 2019 4:25 PM
To: Bolivar, Marti
Cc: Bøe, Sebastian; devel@...
Subject: Re: [Zephyr-devel] Managing/structuring multi image/binary projects for (OTA updatable products etc)

Hi Marti, Sebastian, list

Thanks for the ideas, my replies are inline below.

Hi, I see you are using nrf.

Signing, building, and hex merging of multiple images in a single west command is supported
out-of-the box with the nRF Connect SDK, which uses a patched Zephyr.

This led me to find the "multiple images" documentation for the NCS Zephyr (https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/zephyr/application/index.html#building-and-configuring-multiple-images) ...

So if not using vanilla Zephyr is an option then see:

https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/samples/nrf9160/http_application_update/README.html#http-application-update-sample
https://developer.nordicsemi.com/nRF_Connect_SDK/doc/latest/nrf/ug_multi_image.html

I second Sebastian's suggestion to look at nRF Connect SDK (NCS).
It has useful features for building OTA binaries for nRF devices.


.. it looks like the NCS capabilities will do what I wanted with regard to building a bootloader + app in one go. I haven't had time to look at it properly, but it seems like the strategy is to make the bootloader an extra output of the primary 'app' project (ie extra targets in the app project cmake).
This is a different approach than what I was thinking of. I was envisaging that the bootloader and app were treated as more isolated projects, and were configured and glued together by another higher level _something_.

Traditionally, I tend away from relying too much on vendor 'BSP's for the long term, they've tend to stagnate or diverge too much from mainline as time goes on.

Does anyone know if there are any plans to bring NCS's 'multi-image' functionality into mainline zephyr?

--snip--

-----

West seems like the ideal tool to use for this, it boils down to a manifest repo (for project & dependencies checkout), and then a series of commands to build the various artifacts and merge & sign them. For example:

west init -m git@...:SomeManifestRepo.git
west build --board=theboard -d build-mcuboot -s mcuboot/boot/zephyr -- -DBOARD_ROOT=/abs/path/to/app
west build --board=theboard -d build-app -s app -- -DBOARD_ROOT=/abs/path/to/app
west sign ...

However, it would be nice to be able to glue this all together with
some script etc to simplify the extra elements of the west commands
('theboard' 'BOARD_ROOT') etc.

No matter what you choose to use (mainline or NCS), though, you can
always tell 'west build' your default board like this:

west config build.board theboard

https://docs.zephyrproject.org/latest/guides/west/build-flash-debug.html#configuration-options

The build.board value will be used if you don't provide --board
(provided BOARD is also unset in the calling environment).


This is useful to know, thanks !


I thought about your BOARD_ROOT question. I've gotten other similar
requests recently, and I agree it would be useful, so I've proposed a
new "build.cmake-args" config option:

https://github.com/zephyrproject-rtos/zephyr/pull/21251

With the patch from #21251, you can save your BOARD_ROOT like this:

west config build.cmake-args -- -DBOARD_ROOT=/abs/path/to/app

Your one-time setup work would then be:

west init -m git@.../SomeManifestRepo.git<http://git@gitlab.com/SomeManifestRepo.git>
west update
west config build.board theboard
west config build.cmake-args -- -DBOARD_ROOT=/abs/path/to/app

Note that plain 'west config var val' sets configuration options locally
(just for your west installation). See the --global and --system options
in the 'west config' documentation for alternatives.

# build and flash
west build -d build-mcuboot -s mcuboot/boot/zephyr
west build -d build-app -s app
west sign ...

And here is where I become unsure of how this glue script/(c)makefile/etc fits into the topology of the project.. The natural place for the script to run from is the west installation folder itself (ie, the parent folder of all the subprojects) however the natural place for it to 'live' is in the 'manifest-repo'.

Alternatives:
- copy 'glue' script into west installation folder ?
- west commands ? in the manifest repo, or maybe another common lib etc.

I hope the above is what you're looking for.


I think what I was envisaging was something to glue together the above commands (and keep some separation between the specific 'app' and the bootloader), I thought this would be 'west', but maybe not.. to take this approach would mean having to invoke the build/sign/whatever commands from the 'west installation' folder, and it would only be available from the manifest repo folder.


Note 'west build' can only compile one Zephyr application at a time.
The NCS can build application and mcuboot binaries at once out of the
box, for reference.

Please test and comment in the PR if you get a chance.

Done!, thanks.

Cheers
Marc

See https://github.com/zephyrproject-rtos/zephyr/pull/13672#issuecomment-550203475

________________________________________
From: Marc Reilly <marc.reilly@...>
Sent: Tuesday, December 10, 2019 5:58 PM
To: Bøe, Sebastian
Cc: Bolivar, Marti; devel@...
Subject: Re: [Zephyr-devel] Managing/structuring multi image/binary projects for (OTA updatable products etc)


Hi,

The multi-image approach in NCS will not make it's way into mainline Zephyr because
it was not accepted upstream unfortunately.

Is there some record of the rationale behind this? (eg PR #).
I'm wondering that if its rejected from mainline zephyr for reasons like 'out of scope' etc then maybe there is some existing consensus for alternative ways - or if it was due to lack of demand then maybe that can be reappraised... But that's a lot of "maybes" and "wonderings".

Thanks again!

Cheers
Marc