ztest and code coverage on actual hardware


lairdjm
 

Hi,
We are trying to get twister working to run zephyr tests with code coverage using real hardware (Nordic-based) but seem to be having issues with this:
  1. The nrfjprog reset does not seem to work - the kernel tests halt when they run the oops function and the `nrfjprog --reset` function does not work when the module is in this state, it seems to only work when using --debugreset and it doesn't look like twister can use this (I've opened a support ticket with nordic on this separately)
  2. With code coverage enabled for our modules in the boards file and running twister with it enabled, tests do not seem to run, the start up code gets stuck on https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/testsuite/coverage/coverage.c#L273 where it faults, if we enable code coverage with a normal non-test application e.g. hello_world, we see it run and then output the gcov data, if we run the test without code coverage enabled, it works. It only seems to be when both ztest and code coverage are enabled that this does not work
  3. It mentions in one of the MPU files that userspace is required for code coverage but I don't understand the reason for this, e.g. with the hello_world example it is not using userspace and can still output data (I haven't checked the data to see if it's valid or not). This raises an issue because if we try building one of the kernel tests with code coverage and userspace mode enabled, the required RAM is about 300KB - we are aiming to test on modules with 64-256KB RAM so a 300KB requirement is unworkable
Does anyone have ideas for these issues or do code coverage checks on real hardware?
Thanks,
Jamie


Garrett LoVerde
 

I have no extra insight to provide as I've only ever used Mbed OS's on device automated test infrastructure.

But I will say that I plan to get a device test farm running at some point and will need a solution for Zephyr.

-Garrett LoVerde


lairdjm
 

Hi,
After going back and forth between nRF connect SDK 1.7.1 and current top of zephyr, it seems that something in the test system when running on actual hardware has been majorly broken in the current mainline zephyr version. As of right now I can't get anything working on real hardware and at this point I'm considering using a different test system altogether. @Nasif can you comment on these issues? Things seems to work OK on nRF connect SDK 1.7.1, but on mainline zephyr anything jlink based is broken, anything nordic also seems to be broken.
Thanks,
Jamie

On Fri, 2021-11-19 at 11:32 -0500, Garrett LoVerde via lists.zephyrproject.org wrote:
I have no extra insight to provide as I've only ever used Mbed OS's on device automated test infrastructure.

But I will say that I plan to get a device test farm running at some point and will need a solution for Zephyr.

-Garrett LoVerde


Carles Cufi
 

+ Maciej

 

From: devel@... <devel@...> On Behalf Of lairdjm via lists.zephyrproject.org
Sent: 23 November 2021 10:25
To: devel@...; garrett@...
Subject: Re: [Zephyr-devel] ztest and code coverage on actual hardware

 

Hi,

After going back and forth between nRF connect SDK 1.7.1 and current top of zephyr, it seems that something in the test system when running on actual hardware has been majorly broken in the current mainline zephyr version. As of right now I can't get anything working on real hardware and at this point I'm considering using a different test system altogether. @Nasif can you comment on these issues? Things seems to work OK on nRF connect SDK 1.7.1, but on mainline zephyr anything jlink based is broken, anything nordic also seems to be broken.

Thanks,

Jamie

 

On Fri, 2021-11-19 at 11:32 -0500, Garrett LoVerde via lists.zephyrproject.org wrote:

I have no extra insight to provide as I've only ever used Mbed OS's on device automated test infrastructure.

 

But I will say that I plan to get a device test farm running at some point and will need a solution for Zephyr.

 

-Garrett LoVerde

 


Perkowski, Maciej
 

Hi Jamie,

I am not sure what are you referring to. I am running daily on-target tests on quite a big setup internally at Nordic and I haven’t observed any major breakdown. You can find the on-target results here: https://testing.zephyrproject.org/daily_tests/index.html The latest one with Nordic boards is this one: https://testing.zephyrproject.org/daily_tests/v2.7.99-1421-gaace2da7007d/report/index.html

I cannot access the devzone link you’ve added (waiting to get a proper permission). If we move the discussion to the zephyr’s official discord https://chat.zephyrproject.org/, to the testing channel I will try to provide some more direct help to you.

Regards,

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

Nordic_logo_signature

 

From: Cufi, Carles <Carles.Cufi@...>
Sent: Tuesday, November 23, 2021 10:27 AM
To: jamie.mccrae@...; devel@...; garrett@...; Perkowski, Maciej <Maciej.Perkowski@...>
Subject: RE: [Zephyr-devel] ztest and code coverage on actual hardware

 

+ Maciej

 

From: devel@... <devel@...> On Behalf Of lairdjm via lists.zephyrproject.org
Sent: 23 November 2021 10:25
To: devel@...; garrett@...
Subject: Re: [Zephyr-devel] ztest and code coverage on actual hardware

 

Hi,

After going back and forth between nRF connect SDK 1.7.1 and current top of zephyr, it seems that something in the test system when running on actual hardware has been majorly broken in the current mainline zephyr version. As of right now I can't get anything working on real hardware and at this point I'm considering using a different test system altogether. @Nasif can you comment on these issues? Things seems to work OK on nRF connect SDK 1.7.1, but on mainline zephyr anything jlink based is broken, anything nordic also seems to be broken.

Thanks,

Jamie

 

On Fri, 2021-11-19 at 11:32 -0500, Garrett LoVerde via lists.zephyrproject.org wrote:

I have no extra insight to provide as I've only ever used Mbed OS's on device automated test infrastructure.

 

But I will say that I plan to get a device test farm running at some point and will need a solution for Zephyr.

 

-Garrett LoVerde

 


Greg Leach <greg.leach@...>
 

Hi Maciej – We are looking at a long term goal to integrate z_test as part of our CI process such that our Jenkins build process executes a series of test cases on nRF SoC based physical hardware. We would like to be able to generate a code coverage test report as part of this process and make it visible on Jenkins upon completion of the tests.

 

From our attempts so far, we are encountering consistent issues with the build system, typically being indicated by timeout and RAM limit errors.

 

Would you be able to spend 30 minutes with us to review what we’ve done and what may be causing our issue? Failing that, can you provide a tested example project, on Nordic based hardware (e.g. an nRF52840 or nRF5340 Development Kit would be ideal) that we can use as a reference?

 

We have noticed when enabling code coverage that a linker segment is needed for storage of the coverage data, but this is only actually added to the project linker file when C++ is enabled as a project option.

 

What we’d really like to be sure of is the project and build options needed to facilitate code coverage testing on physical hardware.

 

Many thanks,

 

Greg.

 

From: Perkowski, Maciej <Maciej.Perkowski@...>
Sent: 23 November 2021 13:38
To: Cufi, Carles <Carles.Cufi@...>; Jamie Mccrae <Jamie.Mccrae@...>; devel@...; garrett@...
Subject: RE: [Zephyr-devel] ztest and code coverage on actual hardware

 

EXTERNAL EMAIL: Be careful with attachments and links.

Hi Jamie,

I am not sure what are you referring to. I am running daily on-target tests on quite a big setup internally at Nordic and I haven’t observed any major breakdown. You can find the on-target results here: https://testing.zephyrproject.org/daily_tests/index.html The latest one with Nordic boards is this one: https://testing.zephyrproject.org/daily_tests/v2.7.99-1421-gaace2da7007d/report/index.html

I cannot access the devzone link you’ve added (waiting to get a proper permission). If we move the discussion to the zephyr’s official discord https://chat.zephyrproject.org/, to the testing channel I will try to provide some more direct help to you.

Regards,

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

Nordic_logo_signature

 

From: Cufi, Carles <Carles.Cufi@...>
Sent: Tuesday, November 23, 2021 10:27 AM
To: jamie.mccrae@...; devel@...; garrett@...; Perkowski, Maciej <Maciej.Perkowski@...>
Subject: RE: [Zephyr-devel] ztest and code coverage on actual hardware

 

+ Maciej

 

From: devel@... <devel@...> On Behalf Of lairdjm via lists.zephyrproject.org
Sent: 23 November 2021 10:25
To: devel@...; garrett@...
Subject: Re: [Zephyr-devel] ztest and code coverage on actual hardware

 

Hi,

After going back and forth between nRF connect SDK 1.7.1 and current top of zephyr, it seems that something in the test system when running on actual hardware has been majorly broken in the current mainline zephyr version. As of right now I can't get anything working on real hardware and at this point I'm considering using a different test system altogether. @Nasif can you comment on these issues? Things seems to work OK on nRF connect SDK 1.7.1, but on mainline zephyr anything jlink based is broken, anything nordic also seems to be broken.

Thanks,

Jamie

 

On Fri, 2021-11-19 at 11:32 -0500, Garrett LoVerde via lists.zephyrproject.org wrote:

I have no extra insight to provide as I've only ever used Mbed OS's on device automated test infrastructure.

 

But I will say that I plan to get a device test farm running at some point and will need a solution for Zephyr.

 

-Garrett LoVerde

 


novello
 

Also for me well be important to have an example how to make the test  on nrf52840dk_nrf52840.
Best Regards
Novello G.

Il giorno ven 26 nov 2021 alle ore 16:12 Greg Leach <greg.leach@...> ha scritto:

Hi Maciej – We are looking at a long term goal to integrate z_test as part of our CI process such that our Jenkins build process executes a series of test cases on nRF SoC based physical hardware. We would like to be able to generate a code coverage test report as part of this process and make it visible on Jenkins upon completion of the tests.

 

From our attempts so far, we are encountering consistent issues with the build system, typically being indicated by timeout and RAM limit errors.

 

Would you be able to spend 30 minutes with us to review what we’ve done and what may be causing our issue? Failing that, can you provide a tested example project, on Nordic based hardware (e.g. an nRF52840 or nRF5340 Development Kit would be ideal) that we can use as a reference?

 

We have noticed when enabling code coverage that a linker segment is needed for storage of the coverage data, but this is only actually added to the project linker file when C++ is enabled as a project option.

 

What we’d really like to be sure of is the project and build options needed to facilitate code coverage testing on physical hardware.

 

Many thanks,

 

Greg.

 

From: Perkowski, Maciej <Maciej.Perkowski@...>
Sent: 23 November 2021 13:38
To: Cufi, Carles <Carles.Cufi@...>; Jamie Mccrae <Jamie.Mccrae@...>; devel@...; garrett@...
Subject: RE: [Zephyr-devel] ztest and code coverage on actual hardware

 

EXTERNAL EMAIL: Be careful with attachments and links.

Hi Jamie,

I am not sure what are you referring to. I am running daily on-target tests on quite a big setup internally at Nordic and I haven’t observed any major breakdown. You can find the on-target results here: https://testing.zephyrproject.org/daily_tests/index.html The latest one with Nordic boards is this one: https://testing.zephyrproject.org/daily_tests/v2.7.99-1421-gaace2da7007d/report/index.html

I cannot access the devzone link you’ve added (waiting to get a proper permission). If we move the discussion to the zephyr’s official discord https://chat.zephyrproject.org/, to the testing channel I will try to provide some more direct help to you.

Regards,

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

Nordic_logo_signature

 

From: Cufi, Carles <Carles.Cufi@...>
Sent: Tuesday, November 23, 2021 10:27 AM
To: jamie.mccrae@...; devel@...; garrett@...; Perkowski, Maciej <Maciej.Perkowski@...>
Subject: RE: [Zephyr-devel] ztest and code coverage on actual hardware

 

+ Maciej

 

From: devel@... <devel@...> On Behalf Of lairdjm via lists.zephyrproject.org
Sent: 23 November 2021 10:25
To: devel@...; garrett@...
Subject: Re: [Zephyr-devel] ztest and code coverage on actual hardware

 

Hi,

After going back and forth between nRF connect SDK 1.7.1 and current top of zephyr, it seems that something in the test system when running on actual hardware has been majorly broken in the current mainline zephyr version. As of right now I can't get anything working on real hardware and at this point I'm considering using a different test system altogether. @Nasif can you comment on these issues? Things seems to work OK on nRF connect SDK 1.7.1, but on mainline zephyr anything jlink based is broken, anything nordic also seems to be broken.

Thanks,

Jamie

 

On Fri, 2021-11-19 at 11:32 -0500, Garrett LoVerde via lists.zephyrproject.org wrote:

I have no extra insight to provide as I've only ever used Mbed OS's on device automated test infrastructure.

 

But I will say that I plan to get a device test farm running at some point and will need a solution for Zephyr.

 

-Garrett LoVerde