Structure for external libraries, HAL


Nashif, Anas
 

Hi,
As you are probably already aware, we have a few changes in review that add external components to Zephyr, especially the CMSIS headers needed for porting more Cortex-M MCUs and board.
Zephyr already has some external components. Some will need to move to the new proposed location outlined below.

Proposal:

create a top level directory for all external components and headers with the following structure (slightly modified):

ext/
hal/
cmsis
qmsi
ksdk
...
lib/
tinycrypt
matt-bar
lwm2m-foo
foo-tls
….

we should be able to add more 2nd level categories under ext/, we might consider having drivers for example for drivers from vendor SDKs that are well tested and verified.

The advantages of having all the code in one place:
- easy to update and maintain
- files of the same license and from the same source all in one location
- can be excluded from zephyr style checks easily (to make CI happy)
- no contamination with original zephyr code

disadvantages:
- the code will be in a location different from where it is being used and referenced
- need to create cross references across the tree
- …

We plan to make this final by next week. If you have any concerns or other suggestions please raise them now.


Anas


Carles Cufi
 

Hi Anas,

-----Original Message-----
From: Nashif, Anas [mailto:anas.nashif(a)intel.com]
Sent: Thursday, May 19, 2016 18:48
To: devel(a)lists.zephyrproject.org
Subject: [devel] Structure for external libraries, HAL

Hi,
As you are probably already aware, we have a few changes in review that
add external components to Zephyr, especially the CMSIS headers needed
for porting more Cortex-M MCUs and board.
Zephyr already has some external components. Some will need to move to
the new proposed location outlined below.

Proposal:

create a top level directory for all external components and headers
with the following structure (slightly modified):

ext/
hal/
cmsis
qmsi
ksdk
...
lib/
tinycrypt
matt-bar
lwm2m-foo
foo-tls
….

we should be able to add more 2nd level categories under ext/, we might
consider having drivers for example for drivers from vendor SDKs that
are well tested and verified.

The advantages of having all the code in one place:
- easy to update and maintain
- files of the same license and from the same source all in one location
- can be excluded from zephyr style checks easily (to make CI happy)
- no contamination with original zephyr code

disadvantages:
- the code will be in a location different from where it is being used
and referenced
- need to create cross references across the tree
- …
Would that replace the current lib/ folder, or would it live side-by-side? In any case +1 for the ext/ suggestion, we already do the same in some internal projects and it's proven to be practical.
Also worth mentioning that although CMSIS-CORE is made up of headers only, there are some components such as CMSIS-DSP [1] which are full-blown libraries and might or might not be considered strictly "hal".

Carles

[1] https://github.com/ARM-software/CMSIS/tree/master/CMSIS/DSP_Lib


Nashif, Anas
 

On 20 May 2016, at 13:11, Cufi, Carles <Carles.Cufi(a)nordicsemi.no> wrote:

Hi Anas,

-----Original Message-----
From: Nashif, Anas [mailto:anas.nashif(a)intel.com]
Sent: Thursday, May 19, 2016 18:48
To: devel(a)lists.zephyrproject.org
Subject: [devel] Structure for external libraries, HAL

Hi,
As you are probably already aware, we have a few changes in review that
add external components to Zephyr, especially the CMSIS headers needed
for porting more Cortex-M MCUs and board.
Zephyr already has some external components. Some will need to move to
the new proposed location outlined below.

Proposal:

create a top level directory for all external components and headers
with the following structure (slightly modified):

ext/
hal/
cmsis
qmsi
ksdk
...
lib/
tinycrypt
matt-bar
lwm2m-foo
foo-tls
….

we should be able to add more 2nd level categories under ext/, we might
consider having drivers for example for drivers from vendor SDKs that
are well tested and verified.

The advantages of having all the code in one place:
- easy to update and maintain
- files of the same license and from the same source all in one location
- can be excluded from zephyr style checks easily (to make CI happy)
- no contamination with original zephyr code

disadvantages:
- the code will be in a location different from where it is being used
and referenced
- need to create cross references across the tree
- …
Would that replace the current lib/ folder, or would it live side-by-side? In any case +1 for the ext/ suggestion, we already do the same in some internal projects and it's proven to be practical.
it will not replace lib/. Lib currently has both the minimal c library implementation and tiny crypt. We might need to move tiny-crypt to ext/lib though.


Also worth mentioning that although CMSIS-CORE is made up of headers only, there are some components such as CMSIS-DSP [1] which are full-blown libraries and might or might not be considered strictly "hal”.
Well, those can go under ext/lib for example :-)

Anas

Carles

[1] https://github.com/ARM-software/CMSIS/tree/master/CMSIS/DSP_Lib


Nashif, Anas
 

hi,
A followup to this: I just submitted a patch to exclude ext/hal from checkpatch.
The primary reason was how CI works. CI aborts if it finds any style issues and it does not complete the unit testing and other checks, so to be able to get full coverage also on external code being submitted, checkpatch will not skip files that are part of this directory.

Anas

On 19 May 2016, at 12:48, Nashif, Anas <anas.nashif(a)intel.com> wrote:

Hi,
As you are probably already aware, we have a few changes in review that add external components to Zephyr, especially the CMSIS headers needed for porting more Cortex-M MCUs and board.
Zephyr already has some external components. Some will need to move to the new proposed location outlined below.

Proposal:

create a top level directory for all external components and headers with the following structure (slightly modified):

ext/
hal/
cmsis
qmsi
ksdk
...
lib/
tinycrypt
matt-bar
lwm2m-foo
foo-tls
….

we should be able to add more 2nd level categories under ext/, we might consider having drivers for example for drivers from vendor SDKs that are well tested and verified.

The advantages of having all the code in one place:
- easy to update and maintain
- files of the same license and from the same source all in one location
- can be excluded from zephyr style checks easily (to make CI happy)
- no contamination with original zephyr code

disadvantages:
- the code will be in a location different from where it is being used and referenced
- need to create cross references across the tree
- …

We plan to make this final by next week. If you have any concerns or other suggestions please raise them now.


Anas




Kumar Gala
 

On May 20, 2016, at 8:03 PM, Nashif, Anas <anas.nashif(a)intel.com> wrote:

hi,
A followup to this: I just submitted a patch to exclude ext/hal from checkpatch.
The primary reason was how CI works. CI aborts if it finds any style issues and it does not complete the unit testing and other checks, so to be able to get full coverage also on external code being submitted, checkpatch will not skip files that are part of this directory.

Anas
Shouldn’t all of ext/ be excluded from checkpatch? Seems like ext/lib will have same issues.

- k


Kumar Gala
 

On May 19, 2016, at 11:48 AM, Nashif, Anas <anas.nashif(a)intel.com> wrote:

Hi,
As you are probably already aware, we have a few changes in review that add external components to Zephyr, especially the CMSIS headers needed for porting more Cortex-M MCUs and board.
Zephyr already has some external components. Some will need to move to the new proposed location outlined below.

Proposal:

create a top level directory for all external components and headers with the following structure (slightly modified):

ext/
hal/
cmsis
qmsi
ksdk
...
lib/
tinycrypt
matt-bar
lwm2m-foo
foo-tls
….

we should be able to add more 2nd level categories under ext/, we might consider having drivers for example for drivers from vendor SDKs that are well tested and verified.

The advantages of having all the code in one place:
- easy to update and maintain
- files of the same license and from the same source all in one location
- can be excluded from zephyr style checks easily (to make CI happy)
- no contamination with original zephyr code

disadvantages:
- the code will be in a location different from where it is being used and referenced
- need to create cross references across the tree
- …

We plan to make this final by next week. If you have any concerns or other suggestions please raise them now.


Anas
We should probably document rules on import in ext/README. Mostly around documenting to origins of the code import (i.e., what should be in a commit message, what meta file that documents the source of the code, what version or SHA/tag, etc).

Also, probably having a READMEs in ext/hal & ext/lib that has a one liner or two about what things are:

kext - NXP Kinetis SDK drivers
qmsi - Intel Quark Microcontroller Software Interface
cmsis - ARM Cortex Microcontroller Software Interface Standard

(etc)

The other option would be a file kinda structured like MAINTAINERS in top level ext/hal & ext/lib that gets updated on imports:

ext/hal/README:

D: directory
T: (source tree repo)
V: (repo version/tag info)

CMSIS - ARM Cortex Microcontroller Software Interface Standard
D: cmsis
T: https://github.com/ARM-software/CMSIS
V: v4.5.0

NXP Kinetis SDK drivers
D: kext
T: http://kex.nxp.com/
V: v2

QMSI - Intel Quark Microcontroller Software Interface
D: qmsi
T: https://github.com/01org/qmsi
V: v1.0.1

(Needs some hashing out on the details related to how to handle version info/tags/SHAs)

- k


Nashif, Anas
 

On 21 May 2016, at 10:24, Kumar Gala <kumar.gala(a)linaro.org> wrote:


On May 20, 2016, at 8:03 PM, Nashif, Anas <anas.nashif(a)intel.com> wrote:

hi,
A followup to this: I just submitted a patch to exclude ext/hal from checkpatch.
The primary reason was how CI works. CI aborts if it finds any style issues and it does not complete the unit testing and other checks, so to be able to get full coverage also on external code being submitted, checkpatch will not skip files that are part of this directory.

Anas
Shouldn’t all of ext/ be excluded from checkpatch? Seems like ext/lib will have same issues.
Right, changed to ^ext/

Anas


- k


Nashif, Anas
 

On 21 May 2016, at 10:53, Kumar Gala <kumar.gala(a)linaro.org> wrote:


On May 19, 2016, at 11:48 AM, Nashif, Anas <anas.nashif(a)intel.com> wrote:

Hi,
As you are probably already aware, we have a few changes in review that add external components to Zephyr, especially the CMSIS headers needed for porting more Cortex-M MCUs and board.
Zephyr already has some external components. Some will need to move to the new proposed location outlined below.

Proposal:

create a top level directory for all external components and headers with the following structure (slightly modified):

ext/
hal/
cmsis
qmsi
ksdk
...
lib/
tinycrypt
matt-bar
lwm2m-foo
foo-tls
….

we should be able to add more 2nd level categories under ext/, we might consider having drivers for example for drivers from vendor SDKs that are well tested and verified.

The advantages of having all the code in one place:
- easy to update and maintain
- files of the same license and from the same source all in one location
- can be excluded from zephyr style checks easily (to make CI happy)
- no contamination with original zephyr code

disadvantages:
- the code will be in a location different from where it is being used and referenced
- need to create cross references across the tree
- …

We plan to make this final by next week. If you have any concerns or other suggestions please raise them now.


Anas
We should probably document rules on import in ext/README. Mostly around documenting to origins of the code import (i.e., what should be in a commit message, what meta file that documents the source of the code, what version or SHA/tag, etc).

Also, probably having a READMEs in ext/hal & ext/lib that has a one liner or two about what things are:

kext - NXP Kinetis SDK drivers
qmsi - Intel Quark Microcontroller Software Interface
cmsis - ARM Cortex Microcontroller Software Interface Standard

(etc)
Yeah, README is fine here, I would also add this to the documentation, including usage and policies etc for external modules.



The other option would be a file kinda structured like MAINTAINERS in top level ext/hal & ext/lib that gets updated on imports:

ext/hal/README:

D: directory
T: (source tree repo)
V: (repo version/tag info)

CMSIS - ARM Cortex Microcontroller Software Interface Standard
D: cmsis
T: https://github.com/ARM-software/CMSIS
V: v4.5.0

NXP Kinetis SDK drivers
D: kext
T: http://kex.nxp.com/
V: v2

QMSI - Intel Quark Microcontroller Software Interface
D: qmsi
T: https://github.com/01org/qmsi
V: v1.0.1
I would add those to the top level MAINTAINERS file we are planning to add. I do not thing we need another MAINTAINERS file.

Anas


(Needs some hashing out on the details related to how to handle version info/tags/SHAs)

- k