Re: About the testing framework


Perez-Gonzalez, Inaky <inaky.perez-gonzalez@...>
 

Hi Jaakko

A few weeks ago I presented in the TSC an automated testing solution we are now in the process of opensourcing so the project as a whole can use it.

It is the opposite of unit testing, as it focuses on being able to run any kind of code in target hardware in an automated way-however, I am interested in the approach you are taking because we could have this system running those unit tests in target hardware too. Especially would like to see how can be work on CONFIG_ option combination. Javier is doing some research into that.

The breadth of my concern is drivers. If we follow the LK model, most of our code base is going to be drivers and subsystems; I am trying to figure out how we can get all that tested with a reasonable frequency and coverage

From: Hannikainen, Jaakko [mailto:jaakko.hannikainen(a)intel.com]
Sent: Wednesday, August 3, 2016 5:29 AM
To: devel(a)lists.zephyrproject.org
Subject: [devel] About the testing framework

Hello,

Currently, the testing stack consists of an unsorted bunch of integration and end-to-end tests. There's nothing wrong with having integration tests, but currently a lot of the tests could be done at least partially as plain unit tests. This way we could skip over a lot of unneeded time used for both compiling and running tests, as a lot of the testable code is written in portable C. Then the unit tests wouldn't have to do anything except compile the single .c file containing the code and link it with the testing framework. The test would also run a lot faster then the current stack since it can be run natively on the host without dragging in QEMU and the rest of Zephyr OS.

Including a unit testing library (either existing or a new one) would have multiple benefits for the entire system:

1) Unit testing enables testing static methods, rather than having to stick with the public api exposed by the module. This way all functions can be tested, for example edge cases of a static validation function.

2) Writing unit tests is also a lot simpler than writing complex integration tests. This would eventually result as an increase of testing coverage as writing tests would have a lower barrier of entry. This would especially happen if the project would start to enforce some percent of required test coverage for all incoming patches (this would have to exclude eg. arch specific code).

3) Writing native tests enables running valgrind on the tests, exposing bugs like invalid pointer dereferences and uninitialized variables. For certain cases, statically initialized code could be turned to dynamically allocated by mocking the respective functions, which would also allow using valgrind to hunt down memory leaks.

Although C lags a bit behind languages like Java for mocking, GCC's -Wl,--wrap works well for isolating single functions for testing. I tested out a library called CMocka (licensed under Apache 2.0) which seemed to work pretty well. Writing our own testing library isn't really that hard either, an initial usable version can be done in a day or two.

I propose including a unit testing framework in the Zephyr project (1.6.0 as target), and would like some discussion about how it should be implemented. Should we write our own test framework or use an existing one? What kind of adaptations to the existing test frameworks would be needed, if any? Should code coverage metrics be enforced so that we could have actual CI, rather than Jenkins just running some tests?

Jaakko

Join devel@lists.zephyrproject.org to automatically receive all group messages.