https://github.com/zephyrproject-rtos/zephyr/issues/19244

More discussions on this issue here: https://github.com/zephyrproject-rtos/zephyr/issues/19244

On 17/10/2019 18.51, Bolivar, Marti wrote:

I'm trying to get a beta in people's hands in early November. This will
be available on PyPI, but you will need to manually install a
pre-release version of west with 'pip3 install --pre west' or so to get
it. It's not in 0.6.3.

I want to give it some "soak time" this way before we release an
official version of west with this feature, because it will be harder to
make big changes based on user feedback after that release happens.

I will send mail to the lists with details at that time.

Cool. Thank you :-)


~Jørn

Jørn Villesen Christensen via Lists.Zephyrproject.Org
<extjvc=trifork.com@lists.zephyrproject.org> writes:

Hi Marti,

Thank you too - and for the links :)

On 16/10/2019 22.43, Bolivar, Marti wrote:
As far as I can tell, the existing proposal described on GitHub
satisfies these requirements:

1. one west.yml
2. it's short
3. you don't have to manually track changes in other west.yml files

I was not aware of this work that you are doing. I only saw the #221
issue (West manifest imports) which (afaIcs) is more about merging
several manifests files rather than the import feature you are
documenting.

I read your documentation (most of it at least :) ) and it indeed
sounds like the thing I was looking for. So thank you. Would you have
a road map for when this feature would be implemented?

(I tested with my west 0.6.3 and it does not seem to recognize the
import keyword.)

I'm trying to get a beta in people's hands in early November. This will
be available on PyPI, but you will need to manually install a
pre-release version of west with 'pip3 install --pre west' or so to get
it. It's not in 0.6.3.

I want to give it some "soak time" this way before we release an
official version of west with this feature, because it will be harder to
make big changes based on user feedback after that release happens.

I will send mail to the lists with details at that time.

I think your proposal is indeed what I am looking for. Yes, there is a
slight functional difference whether it is recursive or not. I just
assumed recursive would be beneficial / logical, but you made a point
that it might not be what everybody wants. I think for our project the
one-level import may be sufficient. At least for now.

OK, great, that's really good to hear. I hope you will try the
pre-release when it's available and let me know what you think.

Thanks,
Marti

Hi Jørn,

Thank you for taking the time to write down your thoughts.

Jørn Villesen Christensen via Lists.Zephyrproject.Org

My vision / idea (a.k.a. recursive parsing):
I envisioned that we had a top level project with a west.yml. Upon running west update, west would check out project dependencies (as today), but then it would examine each of these projects to see if they would contain a west.yml. If so, it would run a west update on those as well, ignoring that it is already in a (parent) west project.

Doing this automatically would be undesirable to at those users who do
not want to automatically include all upstream modules in their
downstream (e.g. if they want to exclude modules that are only relevant
to arches, socs, etc. that they are targeting, for bandwidth
optimization or due to License/IP allergies).

Hm... I just assumed that if you were including a project you would always a) need all dependencies of that project, and b) be ok with their licenses. But ok... point accepted :-)

=== Longer story: ===

The main benefits of recursive parsing (as I see it):
The top level project would only have a single west.yml file to maintain - and it would be short. It would not need to keep track of all dependency dependencies. If a project dependency is to be updated, you would not need to run the merge command again (and resolve possible conflicts from intermittent changes).

As far as I can tell, the existing proposal described on GitHub
satisfies these requirements:

1. one west.yml
2. it's short
3. you don't have to manually track changes in other west.yml files

I was not aware of this work that you are doing. I only saw the #221 issue (West manifest imports) which (afaIcs) is more about merging several manifests files rather than the import feature you are documenting.

I read your documentation (most of it at least :) ) and it indeed sounds like the thing I was looking for. So thank you. Would you have a road map for when this feature would be implemented?

(I tested with my west 0.6.3 and it does not seem to recognize the import keyword.)

If that doesn't work for you, can you explain more about why your
specific proposed mechanism of automatically and recursively pulling in
all west.yml files in all subprojects is better?

I think your proposal is indeed what I am looking for. Yes, there is a slight functional difference whether it is recursive or not. I just assumed recursive would be beneficial / logical, but you made a point that it might not be what everybody wants. I think for our project the one-level import may be sufficient. At least for now.

(Note: I originally thought that these explicit imports would not be
recursive to start out, but I'm going to try to make that work.)

When I read your proposal I was thinking that the recursive-feature could be optional:

• If import is boolean (today), this could be changed to enumeration with following keywords:
  • false, off, none: No import is done - same as false today.
    
  • true, top: Same as true today - yielding an error if any sub projects use import keyword
    
  • top-ignore: Same as true today but without yielding an error is sub projects use import keyword. Just ignore it.
    
  • recursive: Recursive import.
• If import is mapping, one could add a key recursive => {false / false-ignore / true}
  

I am not sure whether there is need for this detailed control; I just wanted to share the idea.

Challenges in build procedure:

I think it's not really a build system problem as much as it is a west
init / west update problem.

The build sytem just uses 'west list' to get the default ZEPHYR_MODULES:

https://docs.zephyrproject.org/latest/guides/modules.html

Since 'west list' will still work the same way after this feature is
implemented, I expect no impact on zephyr's build system.

The meat of the feature is making west init and west update work
properly when there multiple files, some of which may also need to be
cloned from separate repositories.

Thank you for the info (west list and build system). I'll take your word for it that the challenge lies with init and update. :-) 

BR
Jørn

Hi Jørn,

Thank you for taking the time to write down your thoughts.

Jørn Villesen Christensen via Lists.Zephyrproject.Org
<extjvc=trifork.com@lists.zephyrproject.org> writes:

Hi Carles (and others),

On 11/10/2019 21.46, Cufi, Carles wrote:

Hi there,



-----Original Message-----
From: users@lists.zephyrproject.org<mailto:users@lists.zephyrproject.org> <users@lists.zephyrproject.org><mailto:users@lists.zephyrproject.org> On
Behalf Of Jørn Villesen Christensen via Lists.Zephyrproject.Org
Sent: 11 October 2019 21:20
To: users@lists.zephyrproject.org<mailto:users@lists.zephyrproject.org>
Cc: users@lists.zephyrproject.org<mailto:users@lists.zephyrproject.org>
Subject: [Zephyr-users] Recursive checkout with West

Hi Zephyr community,

I am a little bit uncertain whether this is the correct list to ask for
help; if not I would appreciate if you could guide me to the correct
place :-)

TLDR: (Current) main question: Is it possible to make West work with
recursive West-based projects? I.e. have a top level project (managed by
West) that has another dependency (Zephyr) that also have a west.yml
file?

A little more background:

I am new to Zephyr and to West, but I am working on a project where we
are considering to use both :-) We do have other external dependencies
and we also have a variety of products that a) uses the same
dependencies (drivers, controllers, etc.) and b) may or may not use
Zephyr, or different configurations of Zephyr.

So in a learning step, we tried to make a top level project that would
depend on Zephyr and another project. The intention was just to see how
West and the build system would behave. What happens is that West is
able to check out the project and to download the dependencies in the
top level west.yml. However, it does not download the dependencies of
Zephyr.

Is this possible? Or is this really unintended use of West (and Zephyr)?



Not currently, but work to support this is ongoing right now, since this has been requested multiple times already and we realize it is a critical feature to have.

Feel free to track progress and add your requirements here:
https://github.com/zephyrproject-rtos/west/issues/221


Thank you for your response and link to the issue you are working on. But it seems to me that what you are working on / discussing is a slightly different issue - or at least different solution (than I had imagined) to the challenges I face. Let me sketch it out.

My vision / idea (a.k.a. recursive parsing):
I envisioned that we had a top level project with a west.yml. Upon running west update, west would check out project dependencies (as today), but then it would examine each of these projects to see if they would contain a west.yml. If so, it would run a west update on those as well, ignoring that it is already in a (parent) west project.

What I understand you are creating:
You are creating functionality to merge multiple west.yml files into one. If I understand correctly, the idea is to write a top-level west template file (west-top.yml) and then merge this with all west.yml files from the project dependencies (that are west projects). This creates a merged west.yml file that the top level project need to maintain.

Have you (as the Zephyr community) considered the recursive parsing approach?

Below are my thoughts on (implementing) this. If you find it interesting, I would love some feedback. If you already considered this (and decided against it) feel free to skip reading the last part of my mail. But I would like to hear your thoughts on the subject :-) (Perhaps you can link me to a discussion elsewhere :) ).

Doing this automatically would be undesirable to at those users who do
not want to automatically include all upstream modules in their
downstream (e.g. if they want to exclude modules that are only relevant
to arches, socs, etc. that they are targeting, for bandwidth
optimization or due to License/IP allergies).

=== Longer story: ===

The main benefits of recursive parsing (as I see it):
The top level project would only have a single west.yml file to maintain - and it would be short. It would not need to keep track of all dependency dependencies. If a project dependency is to be updated, you would not need to run the merge command again (and resolve possible conflicts from intermittent changes).

As far as I can tell, the existing proposal described on GitHub
satisfies these requirements:

1. one west.yml
2. it's short
3. you don't have to manually track changes in other west.yml files

Let's say you have:

- a Zephyr downstream
- one downstream Git repo, my-repo, with your source code
- you are OK with putting west.yml in my-repo

You can then do that with:

# my-repo/west.yml:
manifest:
remotes:
- name: zephyrproject-rtos
url-base: https://github.com/zephyrproject-rtos
projects:
- name: zephyr
remote: zephyrproject-rtos
import: true
self:
path: my-repo

If you want to fork individual upstream projects, you can just add
entries for them in my-repo/west.yml and they'll override
zephyr/west.yml. If you have multiple repositories in your downstream,
you can add them to the projects list in my-repo/west.yml.

If that doesn't work for you, can you explain more about why your
specific proposed mechanism of automatically and recursively pulling in
all west.yml files in all subprojects is better?

As far as I can tell, that saves typing 'import: true' for each project
in my-repo/west.yml that you also want to import from.

(Note: I originally thought that these explicit imports would not be
recursive to start out, but I'm going to try to make that work.)

Challenges in build procedure:
I am not that familiar with West's build system - except that it uses cmake and ninja in the background - but I do realise that special care need to be taken when building project dependencies. As I see it, there are two situations:

1. If the project is *not* a west-managed project: How are these compiled and linked today? [1] I would expect the procedures to be transferable to the recursive parsing build procedure - with minor tweaks.
2. If the project *is* a west-managed project: Normally this project would be the top project and need to be linked to a final executable. But in the recursive parsing scheme, these projects need to be compiled only, and then only linked in the top level project. I guess such logic would be able to be build into the West build system.

[1]: Not really a question. I don't know it, but it is on my todo list to read up upon. So ignore this question unless it has significance for this discussion.

I think it's not really a build system problem as much as it is a west
init / west update problem.

The build sytem just uses 'west list' to get the default ZEPHYR_MODULES:

https://docs.zephyrproject.org/latest/guides/modules.html

Since 'west list' will still work the same way after this feature is
implemented, I expect no impact on zephyr's build system.

The meat of the feature is making west init and west update work
properly when there multiple files, some of which may also need to be
cloned from separate repositories.

Then there is an issue about duplicated (sub) dependencies:
Say we have a top level project that depends on suba and subb. Each of these two sub projects depends on subc. Here are two situations:

1. suba and subb both depend on the same version of subc.
2. suba and subb depend on different versions of subc.

I think the discussion on how to handle this should be deferred to another place than this (already too long) mail - but I guess this is an issue that would also have to be considered in the merge-yml-files approach that you are working on atm.

This reminds me of the diamond problem in multiple inheritance. It's an
issue whether or not imports are explicit.

I think the more formal description of how this works in the docs tree
I've written specing this out covers how it would work if you're willing
to dig through a bit of notation and use your imagination.

The latest docs are here, fwiw:

https://github.com/mbolivar/zephyr/tree/west-multi-manifest-v4

This is just docs, no code. See "Manifest Import Details".

Thanks,
Marti

If you got this far: Thank you for taking your time to read my thoughts :-)

BR
Jørn

"barry.solomon via Lists.Zephyrproject.Org"
<barry.solomon=dexcom.com@lists.zephyrproject.org> writes:

In case this is useful to anyone. I am doing a recursive checkout currently with a little bit of a hack. So I have a west file that just includes my project and the nordic NCS release. The NCS release has its own west.yml file that specifies the version of zephyr, mcuboot, and many more. To do the recursive part, I made a custom west command that is part of my project. So first I do init. This loads my project and NCS repo that contains the other west.yml file. The I run my west custom_update command instead of west update. My command modifies the config file .west to point to the NCS repo, and then the executes west update, which pulls in all the source specified by the other west.yml file. If you wanted to do multiple levels or recursion you could create multiple scripts or just make one complicated script.

Sounds like a fun hack. I wouldn't have thought of doing it like that!

I tried to use pyyaml includes (https://stackoverflow.com/a/9577670),
but couldn't get it to work.

Marti

Hi Stefan,

"stefan.hristozov via Lists.Zephyrproject.Org"
<stefan.hristozov=aisec.fraunhofer.de@lists.zephyrproject.org> writes:

3) West, Kconfig and Cmake: I want to understand how west uses the Kconfig and Cmake files to build an executable image.

If you want to see what 'west build' is doing, just run it in verbose
mode with -v, like this:

$ west -v build -b reel_board samples/hello_world

This prints every cmake command line before it is run (search for
"Running CMake:" in the command output), and also enables verbose
compiler command lines.

You can also run 'west -v flash', 'west -v debug', etc. to see the
commands they run.

That should be all you want to know and more :). Please let me know if
those commands' output doesn't contain what you're looking for.

Thanks,
Marti

Hi Stefan,

On Tue, 2019-10-15 at 01:26 -0700, stefan.hristozov@aisec.fraunhofer.de
wrote:

Dear Zephyr community,

More specific example: Currently I want to use the Ethernet
peripheral of an esp32 chip (board ESP32-POE [2]). Should I write an
network interface or device driver? How to add support for that?

You need to create a device driver for this chip. By doing that, the
network interface is instantiated automatically when the device runs.

I recommend that you look how other Ethernet device drivers in
drivers/ethernet directory are implemented.

[2]:
https://www.olimex.com/Products/IoT/ESP32/ESP32-POE/open-source-hardware

Cheers,
Jukka

Hi all,

This week we will look at:

- V4Z: Update on the state of the Pull Request, should we merge it?
- https://github.com/zephyrproject-rtos/zephyr/pull/17194

- GPIO: Update on progress
- Look at the PRs with driver conversion (https://github.com/zephyrproject-rtos/zephyr/issues/18530)
- Check the timeline
- Any additional outstanding PRs to topic-gpio

Additional items in the "Triage" column in the GitHub project may be discussed if time permits.
If you want an item included in the meeting, please add it to the GitHub project.

https://github.com/zephyrproject-rtos/zephyr/wiki/Zephyr-Committee-and-Working-Group-Meetings#zephyr-api-discussion
https://github.com/zephyrproject-rtos/zephyr/projects/18
https://docs.google.com/document/d/1lv-8B5QE2m4FjBcvfqAXFIgQfW5oz6306zJ7GIZIWCk/edit

Regards,

Carles

Dear Zephyr community,

I very like the zephyr project but I need more documentation in order to get productive with it.

1) Concepts: For me it looks like Zephyr uses some of the Linux concepts. Several good books exists for Linux discussing the kernel, device drivers etc. Which concepts are borrowed form Linux and why? Generally, what concepts does zephyr implement? How an non Linux expert may use those concepts for building high quality embedded software?   

2) Porting to custom boards: The provided documentation [1] is very unclear to me. In the next version of the documentation pages it will be very helpful for me to have a step-by-step tutorial where the migration to a new board is shown, new drivers are included etc.
More specific example: Currently I want to use the Ethernet peripheral of an esp32 chip (board ESP32-POE [2]). Should I write an network interface or device driver? How to add support for that?


3) West, Kconfig and Cmake: I want to understand how west uses the Kconfig and Cmake files to build an executable image.
 

Thank you!
Stefan


[1]: https://docs.zephyrproject.org/latest/guides/porting/board_porting.html
[2]: https://www.olimex.com/Products/IoT/ESP32/ESP32-POE/open-source-hardware

Hi all,

I'm trying to optimise the mcumgr SMP DFU transfer speed over Bluetooth LE.

I'm getting speeds up to 5.12KibB/s (with different combinations of devices) but this Punchthrough blog post claims up to ~50KB/s is possible:
https://punchthrough.com/maximizing-ble-throughput-part-3-data-length-extension-dle-2/

The Bluetooth code is all running on a nRf52832 module.
I'm using Zephyr 1.14.0
I've got scanning turned off

In case this is useful to anyone. I am doing a recursive checkout currently with a little bit of a hack. So I have a west file that just includes my project and the nordic NCS release. The NCS release has its own west.yml file that specifies the version of zephyr, mcuboot, and many more. To do the recursive part, I made a custom west command that is part of my project. So first I do init. This loads my project and NCS repo that contains the other west.yml file. The I run my west custom_update command instead of west update. My command modifies the config file .west to point to the NCS repo, and then the executes west update, which pulls in all the source specified by the other west.yml file. If you wanted to do multiple levels or recursion you could create multiple scripts or just make one complicated script.

Hello,

I'm new to Zephyr, so pardon me if this is a trivial question.

At the moment I'm trying Zephyr on my MIMXRT1060_RT and so far I'm satisfied with the experience, everything works as expected.
Except that I also have the display connected, and while I'm able to play with LVGL for the GUI, I don't see, or understand, how to make the touch screen works.

Any help appreciated.

Olivier

Hi there,

-----Original Message-----
From: users@... <users@...> On
Behalf Of Jørn Villesen Christensen via Lists.Zephyrproject.Org
Sent: 11 October 2019 21:20
To: users@...
Cc: users@...
Subject: [Zephyr-users] Recursive checkout with West

Hi Zephyr community,

I am a little bit uncertain whether this is the correct list to ask for
help; if not I would appreciate if you could guide me to the correct
place :-)

TLDR: (Current) main question: Is it possible to make West work with
recursive West-based projects? I.e. have a top level project (managed by
West) that has another dependency (Zephyr) that also have a west.yml
file?

A little more background:

I am new to Zephyr and to West, but I am working on a project where we
are considering to use both :-) We do have other external dependencies
and we also have a variety of products that a) uses the same
dependencies (drivers, controllers, etc.) and b) may or may not use
Zephyr, or different configurations of Zephyr.

So in a learning step, we tried to make a top level project that would
depend on Zephyr and another project. The intention was just to see how
West and the build system would behave. What happens is that West is
able to check out the project and to download the dependencies in the
top level west.yml. However, it does not download the dependencies of
Zephyr.

Is this possible? Or is this really unintended use of West (and Zephyr)?

Not currently, but work to support this is ongoing right now, since this has been requested multiple times already and we realize it is a critical feature to have.

Feel free to track progress and add your requirements here:
https://github.com/zephyrproject-rtos/west/issues/221

Thank you for your response and link to the issue you are working on. But it seems to me that what you are working on / discussing is a slightly different issue - or at least different solution (than I had imagined) to the challenges I face. Let me sketch it out.

My vision / idea (a.k.a. recursive parsing):
I envisioned that we had a top level project with a west.yml. Upon running west update, west would check out project dependencies (as today), but then it would examine each of these projects to see if they would contain a west.yml. If so, it would run a west update on those as well, ignoring that it is already in a (parent) west project.

What I understand you are creating:
You are creating functionality to merge multiple west.yml files into one. If I understand correctly, the idea is to write a top-level west template file (west-top.yml) and then merge this with all west.yml files from the project dependencies (that are west projects). This creates a merged west.yml file that the top level project need to maintain.

Have you (as the Zephyr community) considered the recursive parsing approach?

Below are my thoughts on (implementing) this. If you find it interesting, I would love some feedback. If you already considered this (and decided against it) feel free to skip reading the last part of my mail. But I would like to hear your thoughts on the subject :-) (Perhaps you can link me to a discussion elsewhere :) ).

=== Longer story: ===

The main benefits of recursive parsing (as I see it):
The top level project would only have a single west.yml file to maintain - and it would be short. It would not need to keep track of all dependency dependencies. If a project dependency is to be updated, you would not need to run the merge command again (and resolve possible conflicts from intermittent changes).

Challenges in build procedure:
I am not that familiar with West's build system - except that it uses cmake and ninja in the background - but I do realise that special care need to be taken when building project dependencies. As I see it, there are two situations:

1. If the project is *not* a west-managed project: How are these compiled and linked today? [1]   I would expect the procedures to be transferable to the recursive parsing build procedure - with minor tweaks.
2. If the project *is* a west-managed project: Normally this project would be the top project and need to be linked to a final executable. But in the recursive parsing scheme, these projects need to be compiled only, and then only linked in the top level project. I guess such logic would be able to be build into the West build system.

[1]: Not really a question. I don't know it, but it is on my todo list to read up upon. So ignore this question unless it has significance for this discussion.

Then there is an issue about duplicated (sub) dependencies:
Say we have a top level project that depends on suba and subb. Each of these two sub projects depends on subc. Here are two situations:

1. suba and subb both depend on the same version of subc.
2. suba and subb depend on different versions of subc.

I think the discussion on how to handle this should be deferred to another place than this (already too long) mail - but I guess this is an issue that would also have to be considered in the merge-yml-files approach that you are working on atm.

If you got this far: Thank you for taking your time to read my thoughts :-)

BR
Jørn

Hi all,

As you all know we are currently transitioning to a new GPIO driver API.
With this transition comes the need to port all existing GPIO drivers to the new API, and work for this is currently ongoing.

We have, however, identified a driver that currently has no one assigned to for its port to the new API: SAM0.

We therefore ask for help to all users that either can do the porting work or can test the work on real hardware.
If we cannot find anyone soon we might be forced to drop support for this platform, since we cannot wok on a driver that we do not know well nor have hardware for.

Progress is documented in this issue:
https://github.com/zephyrproject-rtos/zephyr/issues/18530

Thanks in advance,

Carles