Hi All,

This weekly meeting agenda

1. Follow up “with what tested and put the summary to the public”

a.       The test suite can get consistent results, and we test on important coverage(modules, drivers, platform). How to get coverage improved.

1. Current situation:

l  The test suites are based on API coverage, with additional user scenario + API parameter enhancement.

l  We only have line/branch coverage analysis with x86 platforms.

b.        Reporting(Grafana) and how we do reporting. Test plan and test cases manager. Find a way to take all the results, align on TMS.

1. Current situation:

l  We use test_result repo and allow any one from community to upload their results. Question: Do we need encourage PR owner to provide a test result link in their board testing result for given PR?

l  We provide a Grafana web site with statistic report on test result. (maybe we need feedback on what need add)

l  Zephyr does not have TMS system so far(which is mainly for historic result comparison), to utilize a TMS system we need below:

1)        Test cases id and name shall be mapped consistently, change test name will confuse the test result report over history.

2)        Test case mapped with test suite and test plan shall be provide API to update, as well as test result.

3)        Any one can generate test report without limitation.

c.       Need to create a process to upload test results(json format) for a certain PR on actual HW. (mainly ensure the test is consistent w/wo this PR), select the proper test suites. Also need to isolate real board testing with qemu/cmock way.

1.       Current situation:

We have setup a flow based on test_result, which anyone can upload their test result for given commit. Question: do we need provide a PR result, which does not check the version? Also report github ticket based on upload test result? The scripts is ready.

Maintainer.yml help to get more precise cases in PR-CI

Do we need to create more cases on cmock way? which means we need urge API owner to provide test with mocked.

2.     PR talk

https://github.com/zephyrproject-rtos/zephyr/pull/45684/files

Regards,

Hake

Hi Hake,

               Could we add one topic about this PR https://github.com/zephyrproject-rtos/zephyr/pull/44976 ? I want to push this further to fix the passrate impact in our daily test report, thanks.

Thanks,

Peng

Updated.

This weekly meeting agenda:

1. PR talk https://github.com/zephyrproject-rtos/zephyr/pull/44976

1. Follow up “with what tested and put the summary to the public”

a.    The test suite can get consistent results, and we test on important coverage(modules, drivers, platform). How to get coverage improved.

1. Current situation:

l  The test suites are based on API coverage, with additional user scenario + API parameter enhancement.

l  We only have line/branch coverage analysis with x86 platforms.

b.     Reporting(Grafana) and how we do reporting. Test plan and test cases manager. Find a way to take all the results, align on TMS.

1. Current situation:

l  We use test_result repo and allow any one from community to upload their results. Question: Do we need encourage PR owner to provide a test result link in their board testing result for given PR?

l  We provide a Grafana web site with statistic report on test result. (maybe we need feedback on what need add)

l  Zephyr does not have TMS system so far(which is mainly for historic result comparison), to utilize a TMS system we need below:

1)     Test cases id and name shall be mapped consistently, change test name will confuse the test result report over history.

2)     Test case mapped with test suite and test plan shall be provide API to update, as well as test result.

3)     Any one can generate test report without limitation.

c.     Need to create a process to upload test results(json format) for a certain PR on actual HW. (mainly ensure the test is consistent w/wo this PR), select the proper test suites. Also need to isolate real board testing with qemu/cmock way.

1.    Current situation:

We have setup a flow based on test_result, which anyone can upload their test result for given commit. Question: do we need provide a PR result, which does not check the version? Also report github ticket based on upload test result? The scripts is ready.

Maintainer.yml help to get more precise cases in PR-CI

Do we need to create more cases on cmock way? which means we need urge API owner to provide test with mocked.

Hi All,

This weekly meeting agenda:

1. Follow up “with what tested and put the summary to the public”

1. The test suite can get consistent results, and we test on important coverage(modules, drivers, platform). How to get coverage improved.
1. Current situation:

l  The test suites are based on API coverage, with additional user scenario + API parameter enhancement.

l  We only have line/branch coverage analysis with x86 platforms.

1.  Reporting(Grafana) and how we do reporting. Test plan and test cases manager. Find a way to take all the results, align on TMS.
1. Current situation:

l  We use test_result repo and allow any one from community to upload their results. Question: Do we need encourage PR owner to provide a test result link in their board testing result for given PR?

l  We provide a Grafana web site with statistic report on test result. (maybe we need feedback on what need add)

l  Zephyr does not have TMS system so far(which is mainly for historic result comparison), to utilize a TMS system we need below:

1)     Test cases id and name shall be mapped consistently, change test name will confuse the test result report over history.

2)     Test case mapped with test suite and test plan shall be provide API to update, as well as test result.

3)     Any one can generate test report without limitation.

1. Need to create a process to upload test results(json format) for a certain PR on actual HW. (mainly ensure the test is consistent w/wo this PR), select the proper test suites. Also need to isolate real board testing with qemu/cmock way.

1.    Current situation:

We have setup a flow based on test_result, which anyone can upload their test result for given commit. Question: do we need provide a PR result, which does not check the version? Also report github ticket based on upload test result? The scripts is ready.

Maintainer.yml help to get more precise cases in PR-CI

Do we need to create more cases on cmock way? which means we need urge API owner to provide test with mocked.

Regards,

Hake

Hi All,

This weekly meeting agenda:

1.     Round table discussion.

1.       Monthly testing report summary.

1)      To export more information to community, do we need to have a monthly testing summary?

2)      If yes, the contents will be include

l  Board test report link

l  Issues that reported

l  new features with testing scripts

Regards,

Hake

Hi All,

This weekly meeting agenda:

1.     Round table discussion

2.     Monthly testing report summary.

1)     To export more information to community, do we need to have a monthly testing summary?

2)     If yes, the contents will be include

l  Board test report link

l  Issues that reported

l  new features with testing scripts

Regards,

Hake

Hi All,

This weekly meeting agenda: 

1.     Round table check

2.     V2.7.2-RC1 status updates.

3.     Code coverage process information gathering.

a)         Background: I get some query from Safety working group on code coverage process and howtos, want to sync current information. 

Regards,

Hake

Hi Hake,

Apologies, I won’t be able to attend this week.

Thank You,

Spoorthy Priya Yerabolu.

Hi All,

No specific topic for this weekly meeting. We can go round table and have a quick sync. Thanks.

Regards,

Hake

Hi All,

This weekly meeting agenda.

1.     Feedback for https://github.com/PerMac/TwisterV2

There is one PR from my side

https://github.com/PerMac/TwisterV2/pull/1

2.     Twister with suite name issue

https://github.com/zephyrproject-rtos/zephyr/issues/44539#issuecomment-1090106223

https://github.com/zephyrproject-rtos/zephyr/issues/44438

3.     Round table

Regards,

Hake

Hi All,

Agenda of this weekly meeting:

1.     Pytest for twister v2 demo.

2.     Round table discussion

Regards,

Hake