major sanitycheck changes merged

Nashif, Anas

A complete overhaul of the sanitycheck script and how we build and run
tests. This new version of sanitycheck uses python for job distribution and
drops use of Make.

In addition to the move to python threading library, the following has
been changed:

  • All handlers now run in parallel, meaning that any simulator will run
    in parallel and when testing on multiple devices (using
    --device-testing) the tests are run in parallel.
  • Lexical filtering (using the filter keyword in yaml files) is now
    evaluated at runtime and is no long being pre-processed. This will allow
    us to immediately start executing tests and skip the wait time that was
    needed for filtering.
  • Device testing now supports multiple devices connected at the same
    time and is managed using a hardware map that needs to be generated and
    maintained for every test environment. (using --generate-hardware-map
  • Reports are not long stored in the Zephyr tree and instead stored in
    the output directory where all build artefacts are generated.
  • Each tested target now has a junit report in the output directory.
  • Recording option for performance data and other metrics is now
    available. This will allow us to record the output from the console and
    store the data for later processing. For example benchmark data can be
    captured and uploaded to a tracking server.
  • Test configurations (or instances) are no longer being sorted, this
    will help with balancing the load when we run sanitycheck on multiple
    hosts (as we do in CI).

And many other cleanups and improvements...


Join to automatically receive all group messages.