On 20 Jan 2023, at 13.52, Nashif, Anas <anas.nashif@...> wrote:

Hi,
To be able to move out of PoC status and move most feature development to the new implementation we agreed to have a checkpoint mid-February where we decide how to move forward with our twister plans.
Consensus was that we need to achieve at least 80% feature parity and meet at least 80% of KPI, ie.. execution performance in large scale operations, especially equivalent to those we run in CI on frequent basis.
The following are *must have* features to be able to compare and evaluate:
• Support for the following handlers
• Native Posix
• Qemu (All architectures)
• Device (Devices from different vendors support variety of flashing/serial connections)
• Simulators (for example renode)
• Support for generating coverage reports
• Support for all filtering features we currently have in twister, ie.. evaluating test yaml files and generating a testplan based on parsed data
• Generation of results in json that is compatible with current implementation.
• Output on the screen need to be compatible and close to what we have right now, it does not need to be 100% perfect, but we should be able to see that it is customizable, and we are able to get to similar output
• We need to have runtime statistics and summary statistics compatible with what twister currently emits.
• Error handling and exceptions shall be test oriented and NOT pytest python stacktraces, inline logging shall be supported, and the interface should be zephyr/twister specific and not coming from pytest. Being able to show that this can be done with the basics working should be enough.
• Any diversions or major changes to original implementation need to be documented and explained
• With all of the above done, we should have a list of gaps and missing features that need to be implemented and worked on to be able to achieve full parity.
Evaluation will be done in different scales, with –all being the full scale test. Difference of results shall be minimal.
Regards,
Anas