Nomenclature used in sanitycheck
Renaming output directory to /home/maciej/zephyrproject2/zephyr/sanity-out.1
INFO - Running only a subset: 2/120
INFO - JOBS: 8
INFO - Selecting all possible platforms per test case
INFO - Building initial testcase list...
INFO - 3 test configurations selected, 8 configurations discarded due to filters.
INFO - Adding tasks to the queue...
INFO - 2 of 2 tests passed (100.00%), 0 failed, 1 skipped with 0 warnings in 3.15 seconds
INFO - In total 2 test cases were executed on 269 out of total 272 platforms (98.90%)
INFO - 0 tests executed on platforms, 2 tests were only built.
INFO - Total complete: 2/ 2 100% skipped: 0, failed: 0
Process finished with exit code 0
I think this logging is still a bit confusing and could be cleaned out.
Another issue is how we count tests that were only built. In the line INFO - In total 2 test cases were executed on 269 out of total 272 platforms (98.90%) these 2 test cases were only built. It is also incoherent with the next line: 0 tests executed on platforms, 2 tests were only built. were build-only tests are subtracted
The last issue is that we use different naming for the same stuff. Test configurations is in fact the same as tests as both of them refers to len(suite.instances) in the code. IMO "test suites" from the testing language corresponds the best to what we are counting there. I guess "test suite" is not used since we have TestSuite class in the code which is even something different (it is a suite of all test suites).
So we have test configurations and tests that correspond to instances in the code and test suites in testing language. Test cases are at least coherent everywhere (but I am not sure the difference between test and test case is obvious) ;) Or did I miss something? I think sorting this nomenclature would be beneficial for anyone who would try to work with sanitycheck.