Re: Zephyr: Testing WG
Could we add one topic about this PR https://github.com/zephyrproject-rtos/zephyr/pull/44976 ? I want to push this further to fix the passrate impact in our daily test report, thanks.
From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang
Sent: Thursday, May 12, 2022 12:29 PM
Subject: Re: [testing-wg] Zephyr: Testing WG
This weekly meeting agenda:
a. The test suite can get consistent results, and we test on important coverage(modules, drivers, platform). How to get coverage improved.
l The test suites are based on API coverage, with additional user scenario + API parameter enhancement.
l We only have line/branch coverage analysis with x86 platforms.
b. Reporting(Grafana) and how we do reporting. Test plan and test cases manager. Find a way to take all the results, align on TMS.
l We use test_result repo and allow any one from community to upload their results. Question: Do we need encourage PR owner to provide a test result link in their board testing result for given PR?
l We provide a Grafana web site with statistic report on test result. (maybe we need feedback on what need add)
l Zephyr does not have TMS system so far(which is mainly for historic result comparison), to utilize a TMS system we need below:
1) Test cases id and name shall be mapped consistently, change test name will confuse the test result report over history.
2) Test case mapped with test suite and test plan shall be provide API to update, as well as test result.
3) Any one can generate test report without limitation.
c. Need to create a process to upload test results(json format) for a certain PR on actual HW. (mainly ensure the test is consistent w/wo this PR), select the proper test suites. Also need to isolate real board testing with qemu/cmock way.
1. Current situation:
We have setup a flow based on test_result, which anyone can upload their test result for given commit. Question: do we need provide a PR result, which does not check the version? Also report github ticket based on upload test result? The scripts is ready.
Maintainer.yml help to get more precise cases in PR-CI
Do we need to create more cases on cmock way? which means we need urge API owner to provide test with mocked.
Caution: EXT Email
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#