Date   

Re: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/16/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

Hake Huang
 

Hi All,

 

This is this weekly meeting agenda.

 

1.     Call for start weekly testing (All, 10 minuts)

This weekly tag will be created on Monday night US time.

 

2.     Test manager sytem survey and typical process discussion. (Maksim, 15 minutes)

 

3.     Round table disscussion. (all, 20 mintues)

 

Regards,

Hake

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1116 21:50
To: testing-wg@...
Subject: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/16/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

 

Caution: EXT Email

Reminder: Zephyr: Testing WG weekly call

When: Monday, 16 November 2020, 2:00pm to 3:00pm, (GMT+00:00) UTC

Where:Microsoft Teams Meeting

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Allure Report Review by Maksim

Masalski, Maksim
 

Masalski, Maksim has shared a OneDrive for Business file with you. To view it, click the link below.

icon

Hello, I finished my notes about Allure. During next meeting I will demonstrate its capabilities using our Zephyr results. You can also try to run it, easy to use, but have some issues which I mentioned in the document.

 

P.S. Please let me know if you can open Cloud based link of my Document (I’m not sure if outside of Intel you can open it, if not, use .docx file attached)

 

Maksim

 


Re: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

Hake Huang
 

Hi Maciej

 

It is 2:00pm to 3:00pm, (GMT+00:00) UTC

 

Regards,

Hake

 

From: Perkowski, Maciej <Maciej.Perkowski@...>
Sent: 2020
1110 16:21
To: Hake Huang <hake.huang@...>; testing-wg@...
Subject: RE: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

 

Caution: EXT Email

Hi Hake,
Could you please state the time in UTC? Is it as in call for RSVP: 14-15 UTC?

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

Nordic_logo_signature

 

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang via lists.zephyrproject.org
Sent: Tuesday, November 10, 2020 4:47 AM
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

 

Hi All,

 

This is a notice to that weekly call will be 1 hour later for regions that do not use day time saving shift(e.g. China), if you have problem with this time, please propose your favourite time.

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
119 21:50
To: testing-wg@...
Subject: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

 

Caution: EXT Email

Reminder: Zephyr: Testing WG weekly call

When: Monday, 9 November 2020, 2:00pm to 3:00pm, (GMT+00:00) UTC

Where:Microsoft Teams Meeting

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

Perkowski, Maciej
 

Hi Hake,
Could you please state the time in UTC? Is it as in call for RSVP: 14-15 UTC?

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

Nordic_logo_signature

 

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang via lists.zephyrproject.org
Sent: Tuesday, November 10, 2020 4:47 AM
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

 

Hi All,

 

This is a notice to that weekly call will be 1 hour later for regions that do not use day time saving shift(e.g. China), if you have problem with this time, please propose your favourite time.

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
119 21:50
To: testing-wg@...
Subject: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

 

Caution: EXT Email

Reminder: Zephyr: Testing WG weekly call

When: Monday, 9 November 2020, 2:00pm to 3:00pm, (GMT+00:00) UTC

Where:Microsoft Teams Meeting

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

Hake Huang
 

Hi All,

 

This is a notice to that weekly call will be 1 hour later for regions that do not use day time saving shift(e.g. China), if you have problem with this time, please propose your favourite time.

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
119 21:50
To: testing-wg@...
Subject: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

 

Caution: EXT Email

Reminder: Zephyr: Testing WG weekly call

When: Monday, 9 November 2020, 2:00pm to 3:00pm, (GMT+00:00) UTC

Where:Microsoft Teams Meeting

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 9 November 2020, 2:00pm to 3:00pm, (GMT+00:00) UTC

Where:Microsoft Teams Meeting

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/02/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

Hake Huang
 

Hi All,

 

This is this weekly meeting agenda.

 

1.     versions.json with weekly usage case intro. (Anas, 10 minuts)

https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54

 

2.     Feedback on Release metrics document. (Anas, Meciej, 15 mintues)

https://docs.google.com/document/d/1y7mtAhmvl5z1ohazNgXFEXdFg6RO-J38QsFA7twPPW8/edit?pli=1#

 

3.     Test manager sytem survey and typical process discussion. (Maksim, 15 minutes)

 

4.     Round table disscussion. (all, 20 mintues)

 

Regards,

Hake

 

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
119 21:50
To: testing-wg@...
Subject: [EXT] [testing-wg] Zephyr: Testing WG weekly call - Mon, 11/09/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

 

Caution: EXT Email

Reminder: Zephyr: Testing WG weekly call

When: Monday, 9 November 2020, 2:00pm to 3:00pm, (GMT+00:00) UTC

Where:Microsoft Teams Meeting

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Zephyr: Testing WG weekly call - Mon, 11/02/2020 2:00pm-3:00pm, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 2 November 2020, 2:00pm to 3:00pm, (GMT+00:00) UTC

Where:Microsoft Teams Meeting

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Nashif, Anas
 

Hi,

Same here, won’t be able to attend today. Sorry.

 

Anas

 

From: <testing-wg@...> on behalf of "Perkowski, Maciej" <maciej.perkowski@...>
Date: Monday, 2 November 2020 at 02:57
To: "hake.huang@..." <hake.huang@...>, "testing-wg@..." <testing-wg@...>
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Hi,

I will have to skip today’s meeting as well.

Regards,

Maciej

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

Nordic_logo_signature

 

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang via lists.zephyrproject.org
Sent: Monday, November 2, 2020 4:35 AM
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Hi All,

 

This is this weekly meeting agenda.

 

1.  Updates on versions.json update PR. (Anas, 10 minuts)

https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54

 

2.  Feedback on Release metrics document. (Anas, Meciej, 15 mintues)

https://docs.google.com/document/d/1y7mtAhmvl5z1ohazNgXFEXdFg6RO-J38QsFA7twPPW8/edit?pli=1#

 

3.  Seek for help on gPTP testing setup. (Hake, 15 minutes)

Does any one have the test envrion setting on Windows?

https://github.com/zephyrproject-rtos/zephyr/issues/29599

 

4.  Round table disscussion. (all, 20 mintues)

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1026 22:20
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Perkowski, Maciej
 

Hi,

I will have to skip today’s meeting as well.

Regards,

Maciej

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

Nordic_logo_signature

 

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang via lists.zephyrproject.org
Sent: Monday, November 2, 2020 4:35 AM
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Hi All,

 

This is this weekly meeting agenda.

 

1.     Updates on versions.json update PR. (Anas, 10 minuts)

https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54

 

2.     Feedback on Release metrics document. (Anas, Meciej, 15 mintues)

https://docs.google.com/document/d/1y7mtAhmvl5z1ohazNgXFEXdFg6RO-J38QsFA7twPPW8/edit?pli=1#

 

3.     Seek for help on gPTP testing setup. (Hake, 15 minutes)

Does any one have the test envrion setting on Windows?

https://github.com/zephyrproject-rtos/zephyr/issues/29599

 

4.     Round table disscussion. (all, 20 mintues)

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1026 22:20
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Hi All,

 

This is this weekly meeting agenda.

 

1.     Updates on versions.json update PR. (Anas, 10 minuts)

https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54

 

2.     Feedback on Release metrics document. (Anas, Meciej, 15 mintues)

https://docs.google.com/document/d/1y7mtAhmvl5z1ohazNgXFEXdFg6RO-J38QsFA7twPPW8/edit?pli=1#

 

3.     Seek for help on gPTP testing setup. (Hake, 15 minutes)

Does any one have the test envrion setting on Windows?

https://github.com/zephyrproject-rtos/zephyr/issues/29599

4.     Round table disscussion. (all, 20 mintues)

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1026 22:20
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

 

Thanks Anas.

 

My reason to create a new weekly json file.

 

  1. I find it is difficult to create a compatible json file. and I want to keep the versions.json consistent so that the existing parser will not change.
  2. I want to make it easy to select a weekly test once you know the week of year. So I propose a dict format:

{<Week of year> : <version tag>}

 

So it will be very easy to retrieve any given weekly’s hash tag, by

Data = json.load(…)

if week in data:

   version  = data[week]

 

I see your PR at https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54 , and I leave my comments there. Thanks.

 

Regards,
Hake

 

From: Nashif, Anas <anas.nashif@...>
Sent: 2020
1027 19:55
To: Hake Huang <hake.huang@...>; testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

I did not suggest using another file, we should use exactly the same file but expand the data we store there:

  • Add date
  • Add a flag for weekly

 

I do not think creating a new file is needed..

 

Anas

 

From: <testing-wg@...> on behalf of Hake Huang <hake.huang@...>
Date: Tuesday, 27 October 2020 at 02:04
To: "testing-wg@..." <testing-wg@...>
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Hi All,

 

In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.

 

https://github.com/zephyrproject-rtos/test_results/pull/141

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1026 22:20
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Hi All,

 

Anash update this feature in https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54

 

Please check this PR, I have close my former PR.

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang via lists.zephyrproject.org
Sent: 2020
1027 17:00
To: Erwan Gouriou <erwan.gouriou@...>
Cc: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Hi Erwan,

 

Thanks for your comments. Please check my inline comments below(lead by Hake)

 

Regards,

Hake

 

From: Erwan Gouriou <erwan.gouriou@...>
Sent: 2020
1027 16:42
To: Hake Huang <hake.huang@...>
Cc: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Hi Hake,

 

I have few questions indeed:

1. Does it mean that https://testing.zephyrproject.org/daily_tests/versions.json would not be used anymore ?

    I suggest we keep it as weekly frequency will not be sufficient in release periods.

     [Hake] the versions is kept also.

2. Is there a gating protocol in order to avoid publishing a "bad" commit with build issues that would prevent

    having meaningful results? If there's only one test result upload per week, we need to ensure a minimum quality

    is guaranteed

     [Hake] we have some discussion this week, but we need define a scope on what to check, let me creat an issue here.

3. The duration of an automated test campaign on 14 boards is less than 3 hours (when there are no major issues).

    This of course depends on the machine used, but even with twice this duration, this can be done in a night.

    I'm curious about the obstacles to perform such campaigns on a daily basis.

       [Hake] According to my experiences, there are some efforts required, e.g. I meet download issue regularly and some time an application blocks the board due to many things. So it is no a zero cost thing, (qemu board testing is totally different from real board testing). Also the machine performance is an issue. Not everyone has a powerful build machine, for me the configurations are 2 4-core desktop PCs, which take me a whole day to finish 6 boards if there is not issues found, If something stack, I need restart another round of test cycle from beginning.

 

Cheers

Erwan

 

 

 

 

On Tue, 27 Oct 2020 at 07:04, Hake Huang <hake.huang@...> wrote:

Hi All,

 

In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.

 

https://github.com/zephyrproject-rtos/test_results/pull/141

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1026 22:20
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Nashif, Anas
 

I did not suggest using another file, we should use exactly the same file but expand the data we store there:

  • Add date
  • Add a flag for weekly

 

I do not think creating a new file is needed..

 

Anas

 

From: <testing-wg@...> on behalf of Hake Huang <hake.huang@...>
Date: Tuesday, 27 October 2020 at 02:04
To: "testing-wg@..." <testing-wg@...>
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Hi All,

 

In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.

 

https://github.com/zephyrproject-rtos/test_results/pull/141

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1026 22:20
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Erwan Gouriou
 

Thanks Hake for the feedback, it seems I underestimated the impact of test machine performance.


On Tue, 27 Oct 2020 at 09:59, Hake Huang <hake.huang@...> wrote:

Hi Erwan,

 

Thanks for your comments. Please check my inline comments below(lead by Hake)

 

Regards,

Hake

 

From: Erwan Gouriou <erwan.gouriou@...>
Sent: 2020
1027 16:42
To: Hake Huang <hake.huang@...>
Cc: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Hi Hake,

 

I have few questions indeed:

1. Does it mean that https://testing.zephyrproject.org/daily_tests/versions.json would not be used anymore ?

    I suggest we keep it as weekly frequency will not be sufficient in release periods.

     [Hake] the versions is kept also.

2. Is there a gating protocol in order to avoid publishing a "bad" commit with build issues that would prevent

    having meaningful results? If there's only one test result upload per week, we need to ensure a minimum quality

    is guaranteed

     [Hake] we have some discussion this week, but we need define a scope on what to check, let me creat an issue here.

3. The duration of an automated test campaign on 14 boards is less than 3 hours (when there are no major issues).

    This of course depends on the machine used, but even with twice this duration, this can be done in a night.

    I'm curious about the obstacles to perform such campaigns on a daily basis.

       [Hake] According to my experiences, there are some efforts required, e.g. I meet download issue regularly and some time an application blocks the board due to many things. So it is no a zero cost thing, (qemu board testing is totally different from real board testing). Also the machine performance is an issue. Not everyone has a powerful build machine, for me the configurations are 2 4-core desktop PCs, which take me a whole day to finish 6 boards if there is not issues found, If something stack, I need restart another round of test cycle from beginning.

 

Cheers

Erwan

 

 

 

 

On Tue, 27 Oct 2020 at 07:04, Hake Huang <hake.huang@...> wrote:

Hi All,

 

In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.

 

https://github.com/zephyrproject-rtos/test_results/pull/141

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1026 22:20
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Hi Erwan,

 

Thanks for your comments. Please check my inline comments below(lead by Hake)

 

Regards,

Hake

 

From: Erwan Gouriou <erwan.gouriou@...>
Sent: 2020
1027 16:42
To: Hake Huang <hake.huang@...>
Cc: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Hi Hake,

 

I have few questions indeed:

1. Does it mean that https://testing.zephyrproject.org/daily_tests/versions.json would not be used anymore ?

    I suggest we keep it as weekly frequency will not be sufficient in release periods.

     [Hake] the versions is kept also.

2. Is there a gating protocol in order to avoid publishing a "bad" commit with build issues that would prevent

    having meaningful results? If there's only one test result upload per week, we need to ensure a minimum quality

    is guaranteed

     [Hake] we have some discussion this week, but we need define a scope on what to check, let me creat an issue here.

3. The duration of an automated test campaign on 14 boards is less than 3 hours (when there are no major issues).

    This of course depends on the machine used, but even with twice this duration, this can be done in a night.

    I'm curious about the obstacles to perform such campaigns on a daily basis.

       [Hake] According to my experiences, there are some efforts required, e.g. I meet download issue regularly and some time an application blocks the board due to many things. So it is no a zero cost thing, (qemu board testing is totally different from real board testing). Also the machine performance is an issue. Not everyone has a powerful build machine, for me the configurations are 2 4-core desktop PCs, which take me a whole day to finish 6 boards if there is not issues found, If something stack, I need restart another round of test cycle from beginning.

 

Cheers

Erwan

 

 

 

 

On Tue, 27 Oct 2020 at 07:04, Hake Huang <hake.huang@...> wrote:

Hi All,

 

In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.

 

https://github.com/zephyrproject-rtos/test_results/pull/141

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1026 22:20
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Erwan Gouriou
 

Hi Hake,

I have few questions indeed:
1. Does it mean that https://testing.zephyrproject.org/daily_tests/versions.json would not be used anymore ?
    I suggest we keep it as weekly frequency will not be sufficient in release periods.
2. Is there a gating protocol in order to avoid publishing a "bad" commit with build issues that would prevent
    having meaningful results? If there's only one test result upload per week, we need to ensure a minimum quality
    is guaranteed
3. The duration of an automated test campaign on 14 boards is less than 3 hours (when there are no major issues).
    This of course depends on the machine used, but even with twice this duration, this can be done in a night.
    I'm curious about the obstacles to perform such campaigns on a daily basis.

Cheers
Erwan




On Tue, 27 Oct 2020 at 07:04, Hake Huang <hake.huang@...> wrote:

Hi All,

 

In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.

 

https://github.com/zephyrproject-rtos/test_results/pull/141

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1026 22:20
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Hi All,

 

In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.

 

https://github.com/zephyrproject-rtos/test_results/pull/141

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
1026 22:20
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Updated Event: Zephyr: Testing WG weekly call #cal-invite

testing-wg@lists.zephyrproject.org Calendar <noreply@...>
 

Zephyr: Testing WG weekly call

When:
Monday, 2 November 2020
2:00pm to 3:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Zephyr: Testing WG weekly call - Mon, 10/26/2020 1:00pm-2:00pm, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 26 October 2020, 1:00pm to 2:00pm, (GMT+00:00) UTC

Where:Microsoft Teams Meeting

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________

61 - 80 of 331