Changes to testcases and sanitychecks


Nashif, Anas
 

Hi,

We have just merged a series of commits that change the behavior of sanitycheck script and the syntax of the testcase definition file. While the syntax stays mostly the same, it was changed from ini to yaml. Please see the documentation of sanitycheck and the syntax here:

 

https://www.zephyrproject.org/doc/subsystems/test/sanitycheck.html

 

 

This change was done to allow for better and fast filtering of testcases and to allow for increasing the coverage of build and emulation tests we run on every PR. One major issue we had in the past was the fact that we did not test-build some architectures (Cortex-M0) by default which resulted in many issues and broken code in master. In this new environment we have introduced a new board metadata file that helps with filtering test-cases correctly without having to dig into Kconfig, this for example includes definition of RAM/ROM sizes right now and will be expanded to expose additional features that can be tested with testcases that will support those features.

 

If you have some work in progress or a PR already in the tree with new testcase.ini or a modified one, please update it to use the new syntax and file format. There is a rather simple and  straightforward script that can do this for you, for example:

 

./scripts/ini2yaml.py samples/hello_world/testcase.ini sample  # for samples

 

./scripts/ini2yaml.py tests/kernel/pipe/pipe_api/testcase.ini testcase # for tests

 

The difference between samples and tests is just some additional placeholder keywords for samples, the test structure stays the same for both.

 

 

Note that the number of testcases that are now run when you execute sanitucheck has increased by at least 100, which is due to the increased coverage.

 

This topic was covered in the TSC meeting on May 10th.

 

Thanks,

Anas

 

 

Join devel@lists.zephyrproject.org to automatically receive all group messages.