Hi Maciej – We are looking at a long term goal to integrate z_test as part of our CI process such that our Jenkins build process executes a series of test cases on nRF SoC based physical hardware. We would like to be able to generate a code coverage test report as part of this process and make it visible on Jenkins upon completion of the tests.

 

From our attempts so far, we are encountering consistent issues with the build system, typically being indicated by timeout and RAM limit errors.

 

Would you be able to spend 30 minutes with us to review what we’ve done and what may be causing our issue? Failing that, can you provide a tested example project, on Nordic based hardware (e.g. an nRF52840 or nRF5340 Development Kit would be ideal) that we can use as a reference?

 

We have noticed when enabling code coverage that a linker segment is needed for storage of the coverage data, but this is only actually added to the project linker file when C++ is enabled as a project option.

 

What we’d really like to be sure of is the project and build options needed to facilitate code coverage testing on physical hardware.

 

Many thanks,

 

Greg.

 

From: Perkowski, Maciej <Maciej.Perkowski@...>
Sent: 23 November 2021 13:38
To: Cufi, Carles <Carles.Cufi@...>; Jamie Mccrae <Jamie.Mccrae@...>; devel@...; garrett@...
Subject: RE: [Zephyr-devel] ztest and code coverage on actual hardware

 

EXTERNAL EMAIL: Be careful with attachments and links.

Hi Jamie,

I am not sure what are you referring to. I am running daily on-target tests on quite a big setup internally at Nordic and I haven’t observed any major breakdown. You can find the on-target results here: https://testing.zephyrproject.org/daily_tests/index.html The latest one with Nordic boards is this one: https://testing.zephyrproject.org/daily_tests/v2.7.99-1421-gaace2da7007d/report/index.html

I cannot access the devzone link you’ve added (waiting to get a proper permission). If we move the discussion to the zephyr’s official discord https://chat.zephyrproject.org/, to the testing channel I will try to provide some more direct help to you.

Regards,

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

 

From: Cufi, Carles <Carles.Cufi@...>
Sent: Tuesday, November 23, 2021 10:27 AM
To: jamie.mccrae@...; devel@...; garrett@...; Perkowski, Maciej <Maciej.Perkowski@...>
Subject: RE: [Zephyr-devel] ztest and code coverage on actual hardware

 

+ Maciej

 

From: devel@... <devel@...> On Behalf Of lairdjm via lists.zephyrproject.org
Sent: 23 November 2021 10:25
To: devel@...; garrett@...
Subject: Re: [Zephyr-devel] ztest and code coverage on actual hardware

 

Hi,

After going back and forth between nRF connect SDK 1.7.1 and current top of zephyr, it seems that something in the test system when running on actual hardware has been majorly broken in the current mainline zephyr version. As of right now I can't get anything working on real hardware and at this point I'm considering using a different test system altogether. @Nasif can you comment on these issues? Things seems to work OK on nRF connect SDK 1.7.1, but on mainline zephyr anything jlink based is broken, anything nordic also seems to be broken.

Thanks,

Jamie

 

On Fri, 2021-11-19 at 11:32 -0500, Garrett LoVerde via lists.zephyrproject.org wrote:

I have no extra insight to provide as I've only ever used Mbed OS's on device automated test infrastructure.

 

But I will say that I plan to get a device test farm running at some point and will need a solution for Zephyr.

 

-Garrett LoVerde