Re: Test coverage and its measurements


Nashif, Anas
 

You mean test that runs in an emulator or using the unit test type in ztest, which builds as a host binary?

 

Anas

 

From: Jeremy Bettis <jbettis@...>
Date: Thursday, February 3, 2022 at 3:37 PM
To: Mai, Enjia <enjia.mai@...>
Cc: Nashif, Anas <anas.nashif@...>, Hake Huang <hake.huang@...>, maciej.perkowski@... <maciej.perkowski@...>, testing-wg@... <testing-wg@...>, Wang, Guojun <guojun.wang@...>
Subject: Re: [testing-wg] Test coverage and its measurements

I've noticed there is only coverage for posix and qemu_x86 on https://app.codecov.io/gh/zephyrproject-rtos/zephyr.  Do you have any guides for making a unittest for non posix code? 

 

For example, if I wanted to create a simple unit test for the code in /arch/arm/core where would I start?  I'm fine with having to fake out most of the dependencies.

 

 

 

On Thu, Feb 3, 2022 at 7:51 AM Mai, Enjia <enjia.mai@...> wrote:

Dear all,

 

Thanks for bringing and clarifying that question.

 

And excuse me, Hake and all,  I cannot attend this meeting tonight due to family activities of CNY.

 

Thanks!

Enjia

 

From: testing-wg@... <testing-wg@...> On Behalf Of Nashif, Anas
Sent: Thursday, February 3, 2022 10:32 PM
To: Hake Huang <hake.huang@...>; maciej.perkowski@...; testing-wg@...
Cc: Wang, Guojun <guojun.wang@...>
Subject: Re: [testing-wg] Test coverage and its measurements

 

Hi,

Coverage reports and having test coverage is a test activity, has nothing to do with Safety. Maybe it was not discussed here in this group in the past, but it’s something that belongs here IMO.

 

Anas

 

From: testing-wg@... <testing-wg@...> on behalf of Hake Huang <hake.huang@...>
Date: Thursday, February 3, 2022 at 9:14 AM
To: maciej.perkowski@... <maciej.perkowski@...>, testing-wg@... <testing-wg@...>
Cc: Wang, Guojun <guojun.wang@...>
Subject: Re: [testing-wg] Test coverage and its measurements

Coverage analysis is not conducted in the testing working group instead there is another team who is testing the kernel API coverage. IIRC it is safety team, correct me if I am wrong @Mai.Enjia@...

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Perkowski, Maciej via lists.zephyrproject.org
Sent: 2022
23 17:14
To: testing-wg@...
Cc: Wang, Guojun <guojun.wang@...>
Subject: [EXT] [testing-wg] Test coverage and its measurements

 

Caution: EXT Email

Dear Testers,

I was recently asked by my colleague about quality standards related to Zephyr’s contributions, in particular to testing. A relevant section of our docs states:

“When major new functionality is added, tests for the new functionality MUST be added to the automated test suite. All new APIs MUST be documented and tested and tests MUST cover at least 80% of the added functionality using the code coverage tool and reporting provided by the project.”

https://docs.zephyrproject.org/latest/contribute/index.html#other-commit-expectations
Is there some more detailed explanation of this part? In particular: “tests MUST cover at least 80% of the added functionality using the code coverage tool and reporting provided by the project.”  How this code coverage is evaluated? Is this enforced during a PR process?
Another question is related to the Bluetooth stack. How the overall test coverage of Bluetooth stack in Zephyr is measured? Are BubbleSim tests included in such calculations?
Maybe we can have a short discussion related to test coverage measurements during our today’s wg meeting?
Cheers,

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

Nordic_logo_signature

 


 

--

Jeremy Bettis | ChromeOS FAFT lead | jbettis@... | 303-257-2486

Join testing-wg@lists.zephyrproject.org to automatically receive all group messages.