Date   

Event: Zephyr: Testing WG weekly call - 05/12/2022 #cal-reminder

Group Notification <noreply@...>
 

Reminder: Zephyr: Testing WG weekly call

When:
05/12/2022
7:00am to 8:00am
(UTC-07:00) America/Los Angeles

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: Zephyr: Testing WG

Chen, Peng1
 

Hi Hake,

 

               Could we add one topic about this PR https://github.com/zephyrproject-rtos/zephyr/pull/44976 ? I want to push this further to fix the passrate impact in our daily test report, thanks.

 

Thanks,

Peng

 

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang
Sent: Thursday, May 12, 2022 12:29 PM
To: testing-wg@...
Subject: Re: [testing-wg] Zephyr: Testing WG

 

Hi All,

 

This weekly meeting agenda:

 

  1. Follow up “with what tested and put the summary to the public”

a.     The test suite can get consistent results, and we test on important coverage(modules, drivers, platform). How to get coverage improved.

      1. Current situation:

l  The test suites are based on API coverage, with additional user scenario + API parameter enhancement.

l  We only have line/branch coverage analysis with x86 platforms.

 

b.      Reporting(Grafana) and how we do reporting. Test plan and test cases manager. Find a way to take all the results, align on TMS.

      1. Current situation:

l  We use test_result repo and allow any one from community to upload their results. Question: Do we need encourage PR owner to provide a test result link in their board testing result for given PR?

l  We provide a Grafana web site with statistic report on test result. (maybe we need feedback on what need add)

l  Zephyr does not have TMS system so far(which is mainly for historic result comparison), to utilize a TMS system we need below:

1)       Test cases id and name shall be mapped consistently, change test name will confuse the test result report over history.

2)       Test case mapped with test suite and test plan shall be provide API to update, as well as test result.

3)       Any one can generate test report without limitation.

     

 

 

c.      Need to create a process to upload test results(json format) for a certain PR on actual HW. (mainly ensure the test is consistent w/wo this PR), select the proper test suites. Also need to isolate real board testing with qemu/cmock way.

1.     Current situation:

We have setup a flow based on test_result, which anyone can upload their test result for given commit. Question: do we need provide a PR result, which does not check the version? Also report github ticket based on upload test result? The scripts is ready.

Maintainer.yml help to get more precise cases in PR-CI

Do we need to create more cases on cmock way? which means we need urge API owner to provide test with mocked.

 

 

 

Regards,

Hake

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...; Mai, Enjia
Subject: [EXT] Zephyr: Testing WG
When: 2022512日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email

 

________________________________________________________________________________

________________________________________________________________________________


Re: Zephyr: Testing WG

Hake Huang
 

Updated.

 

This weekly meeting agenda:

 

  1. PR talk https://github.com/zephyrproject-rtos/zephyr/pull/44976

 

  1. Follow up “with what tested and put the summary to the public”

a.    The test suite can get consistent results, and we test on important coverage(modules, drivers, platform). How to get coverage improved.

      1. Current situation:

l  The test suites are based on API coverage, with additional user scenario + API parameter enhancement.

l  We only have line/branch coverage analysis with x86 platforms.

 

b.     Reporting(Grafana) and how we do reporting. Test plan and test cases manager. Find a way to take all the results, align on TMS.

      1. Current situation:

l  We use test_result repo and allow any one from community to upload their results. Question: Do we need encourage PR owner to provide a test result link in their board testing result for given PR?

l  We provide a Grafana web site with statistic report on test result. (maybe we need feedback on what need add)

l  Zephyr does not have TMS system so far(which is mainly for historic result comparison), to utilize a TMS system we need below:

1)     Test cases id and name shall be mapped consistently, change test name will confuse the test result report over history.

2)     Test case mapped with test suite and test plan shall be provide API to update, as well as test result.

3)     Any one can generate test report without limitation.

     

 

 

c.     Need to create a process to upload test results(json format) for a certain PR on actual HW. (mainly ensure the test is consistent w/wo this PR), select the proper test suites. Also need to isolate real board testing with qemu/cmock way.

1.    Current situation:

We have setup a flow based on test_result, which anyone can upload their test result for given commit. Question: do we need provide a PR result, which does not check the version? Also report github ticket based on upload test result? The scripts is ready.

Maintainer.yml help to get more precise cases in PR-CI

Do we need to create more cases on cmock way? which means we need urge API owner to provide test with mocked.

 

 

From: Chen, Peng1 <peng1.chen@...>
Sent: 2022
512 14:38
To: Hake Huang <hake.huang@...>; testing-wg@...
Subject: [EXT] RE: [testing-wg] Zephyr: Testing WG

 

Caution: EXT Email

Hi Hake,

 

               Could we add one topic about this PR https://github.com/zephyrproject-rtos/zephyr/pull/44976 ? I want to push this further to fix the passrate impact in our daily test report, thanks.

 

Thanks,

Peng

 

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang
Sent: Thursday, May 12, 2022 12:29 PM
To: testing-wg@...
Subject: Re: [testing-wg] Zephyr: Testing WG

 

Hi All,

 

This weekly meeting agenda:

 

  1. Follow up “with what tested and put the summary to the public”
    1. The test suite can get consistent results, and we test on important coverage(modules, drivers, platform). How to get coverage improved.
      1. Current situation:

l  The test suites are based on API coverage, with additional user scenario + API parameter enhancement.

l  We only have line/branch coverage analysis with x86 platforms.

 

    1.  Reporting(Grafana) and how we do reporting. Test plan and test cases manager. Find a way to take all the results, align on TMS.
      1. Current situation:

l  We use test_result repo and allow any one from community to upload their results. Question: Do we need encourage PR owner to provide a test result link in their board testing result for given PR?

l  We provide a Grafana web site with statistic report on test result. (maybe we need feedback on what need add)

l  Zephyr does not have TMS system so far(which is mainly for historic result comparison), to utilize a TMS system we need below:

1)     Test cases id and name shall be mapped consistently, change test name will confuse the test result report over history.

2)     Test case mapped with test suite and test plan shall be provide API to update, as well as test result.

3)     Any one can generate test report without limitation.

     

 

 

    1. Need to create a process to upload test results(json format) for a certain PR on actual HW. (mainly ensure the test is consistent w/wo this PR), select the proper test suites. Also need to isolate real board testing with qemu/cmock way.

1.    Current situation:

We have setup a flow based on test_result, which anyone can upload their test result for given commit. Question: do we need provide a PR result, which does not check the version? Also report github ticket based on upload test result? The scripts is ready.

Maintainer.yml help to get more precise cases in PR-CI

Do we need to create more cases on cmock way? which means we need urge API owner to provide test with mocked.

 

 

 

Regards,

Hake

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021
614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...; Mai, Enjia
Subject: [EXT] Zephyr: Testing WG
When: 2022
512日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email

 

________________________________________________________________________________

________________________________________________________________________________


Re: Zephyr: Testing WG

Hake Huang
 

Hi All,

 

This weekly meeting agenda:

 

  1. Follow up “with what tested and put the summary to the public”
    1. The test suite can get consistent results, and we test on important coverage(modules, drivers, platform). How to get coverage improved.
      1. Current situation:

l  The test suites are based on API coverage, with additional user scenario + API parameter enhancement.

l  We only have line/branch coverage analysis with x86 platforms.

 

    1.  Reporting(Grafana) and how we do reporting. Test plan and test cases manager. Find a way to take all the results, align on TMS.
      1. Current situation:

l  We use test_result repo and allow any one from community to upload their results. Question: Do we need encourage PR owner to provide a test result link in their board testing result for given PR?

l  We provide a Grafana web site with statistic report on test result. (maybe we need feedback on what need add)

l  Zephyr does not have TMS system so far(which is mainly for historic result comparison), to utilize a TMS system we need below:

1)     Test cases id and name shall be mapped consistently, change test name will confuse the test result report over history.

2)     Test case mapped with test suite and test plan shall be provide API to update, as well as test result.

3)     Any one can generate test report without limitation.

     

 

 

    1. Need to create a process to upload test results(json format) for a certain PR on actual HW. (mainly ensure the test is consistent w/wo this PR), select the proper test suites. Also need to isolate real board testing with qemu/cmock way.

1.    Current situation:

We have setup a flow based on test_result, which anyone can upload their test result for given commit. Question: do we need provide a PR result, which does not check the version? Also report github ticket based on upload test result? The scripts is ready.

Maintainer.yml help to get more precise cases in PR-CI

Do we need to create more cases on cmock way? which means we need urge API owner to provide test with mocked.

 

 

Regards,

Hake

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021
614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...; Mai, Enjia
Subject: [EXT] Zephyr: Testing WG
When: 2022
512日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email



________________________________________________________________________________

________________________________________________________________________________


Event: Zephyr: Testing WG weekly call - 05/05/2022 #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <noreply@...>
 

Reminder: Zephyr: Testing WG weekly call

When:
05/05/2022
7:00am to 8:00am
(UTC-07:00) America/Los Angeles

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: Zephyr: Testing WG

Hake Huang
 

Hi All,

 

This weekly meeting agenda:

 

1.     Round table discussion.

 

1.       Monthly testing report summary.

1)      To export more information to community, do we need to have a monthly testing summary?

2)      If yes, the contents will be include

l  Board test report link

l  Issues that reported

l  new features with testing scripts

 

Regards,

Hake

 

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021
614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...; Mai, Enjia
Subject: [EXT] Zephyr: Testing WG
When: 2022
55日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email



________________________________________________________________________________

________________________________________________________________________________


Event: Zephyr: Testing WG weekly call - 04/28/2022 #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <noreply@...>
 

Reminder: Zephyr: Testing WG weekly call

When:
04/28/2022
7:00am to 8:00am
(UTC-07:00) America/Los Angeles

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: Zephyr: Testing WG

Hake Huang
 

Hi All,

 

This weekly meeting agenda:

 

1.     Round table discussion

 

2.     Monthly testing report summary.

1)     To export more information to community, do we need to have a monthly testing summary?

2)     If yes, the contents will be include

l  Board test report link

l  Issues that reported

l  new features with testing scripts

 

 

 

Regards,

Hake

 

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021
614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...; Mai, Enjia
Subject: [EXT] Zephyr: Testing WG
When: 2022
428日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email



________________________________________________________________________________

________________________________________________________________________________


Event: Zephyr: Testing WG weekly call - 04/21/2022 #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <noreply@...>
 

Reminder: Zephyr: Testing WG weekly call

When:
04/21/2022
7:00am to 8:00am
(UTC-07:00) America/Los Angeles

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: Zephyr: Testing WG

Hake Huang
 

Hi All,

 

This weekly meeting agenda: 

 

1.     Round table check

 

2.     V2.7.2-RC1 status updates.

 

3.     Code coverage process information gathering.

 

a)         Background: I get some query from Safety working group on code coverage process and howtos, want to sync current information. 

 

Regards,

Hake

 

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021
614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...
Subject: [EXT] Zephyr: Testing WG
When: 2022
421日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email



________________________________________________________________________________

________________________________________________________________________________


Event: Zephyr: Testing WG weekly call - 04/14/2022 #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <noreply@...>
 

Reminder: Zephyr: Testing WG weekly call

When:
04/14/2022
7:00am to 8:00am
(UTC-07:00) America/Los Angeles

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: Zephyr: Testing WG

Yerabolu, Spoorthy Priya
 

Hi Hake,

 

Apologies, I won’t be able to attend this week.

 

Thank You,

Spoorthy Priya Yerabolu.

 

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang
Sent: Wednesday, April 13, 2022 7:07 PM
To: testing-wg@...
Subject: Re: [testing-wg] Zephyr: Testing WG

 

Hi All,

 

No specific topic for this weekly meeting. We can go round table and have a quick sync. Thanks.

 

 

Regards,

Hake

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...
Subject: [EXT] Zephyr: Testing WG
When: 2022414日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email




________________________________________________________________________________

________________________________________________________________________________


Re: Zephyr: Testing WG

Hake Huang
 

Hi All,

 

No specific topic for this weekly meeting. We can go round table and have a quick sync. Thanks.

 

 

Regards,

Hake

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021
614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...
Subject: [EXT] Zephyr: Testing WG
When: 2022
414日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email



________________________________________________________________________________

________________________________________________________________________________


Event: Zephyr: Testing WG weekly call - 04/07/2022 #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <noreply@...>
 

Reminder: Zephyr: Testing WG weekly call

When:
04/07/2022
7:00am to 8:00am
(UTC-07:00) America/Los Angeles

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: Zephyr: Testing WG

Hake Huang
 

Hi All,

 

This weekly meeting agenda.

 

1.     Feedback for https://github.com/PerMac/TwisterV2

 

There is one PR from my side

https://github.com/PerMac/TwisterV2/pull/1

 

2.     Twister with suite name issue

https://github.com/zephyrproject-rtos/zephyr/issues/44539#issuecomment-1090106223

https://github.com/zephyrproject-rtos/zephyr/issues/44438

 

3.     Round table

Regards,

Hake

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021
614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...
Subject: [EXT] Zephyr: Testing WG
When: 2022
47日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email



________________________________________________________________________________

________________________________________________________________________________


Centralised board farm test results

Henrik Brix Andersen
 

Hi all,

Do we have an established process/interface/API for reporting board test results from the Zephyr community to a centralised overview?

I recently stumbled across https://github.com/zephyrproject-rtos/zephyr/wiki/Board-Testing-Introduction and the linked https://testing.zephyrproject.org/daily_tests/index.html
How are the results presented there propagated/collected?

Regards,
Brix
--
Henrik Brix Andersen


Event: Zephyr: Testing WG weekly call - 03/31/2022 #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <noreply@...>
 

Reminder: Zephyr: Testing WG weekly call

When:
03/31/2022
7:00am to 8:00am
(UTC-07:00) America/Los Angeles

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: Zephyr: Testing WG

Hake Huang
 

Hi All,

 

Agenda of this weekly meeting:

 

1.     Pytest for twister v2 demo.

 

2.     Round table discussion

 

Regards,

Hake

 

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021
614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...
Subject: [EXT] Zephyr: Testing WG
When: 2022
331日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email



________________________________________________________________________________

________________________________________________________________________________


Event: Zephyr: Testing WG weekly call - 03/24/2022 #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <noreply@...>
 

Reminder: Zephyr: Testing WG weekly call

When:
03/24/2022
7:00am to 8:00am
(UTC-07:00) America/Los Angeles

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: Zephyr: Testing WG weekly meeting agenda

Hake Huang
 

Hi All,

 

This weekly meeting agenda:

 

1.     Round table discussion.

 

2.     Git hub issue follows

https://github.com/zephyrproject-rtos/zephyr/pull/44094

 

Regards,

Hake

 

-----Original Appointment-----
From: Brett Preston <bpreston@...>
Sent: 2021
614 9:58
To: Brett Preston
Cc: bpreston@...; kstewart@...; hake.huang@...; anas.nashif@...
Subject: [EXT] Zephyr: Testing WG
When: 2022
324日星期四 7:00-8:00 (UTC-08:00) Pacific Time (US & Canada).
Where:

 

Caution: EXT Email



________________________________________________________________________________

________________________________________________________________________________

101 - 120 of 595