[EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite
|
Hake Huang
Hi All,
In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.
https://github.com/zephyrproject-rtos/test_results/pull/141
Regards, Hake
From: testing-wg@... <testing-wg@...>
On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020年10月26日 22:20 To: testing-wg@... Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite
Caution: EXT Email Zephyr: Testing WG weekly call When: Where: Organizer: testing-wg@... An RSVP is requested. Click here to RSVP Description: Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing ________________________________________________________________________________ +1 321-558-6518 United States, Orlando (Toll) Conference ID: 831 716 531# Local numbers | Reset PIN | Learn more about Teams | Meeting options
________________________________________________________________________________ |
|
|
|
Erwan Gouriou
Hi Hake, I have few questions indeed: 1. Does it mean that https://testing.zephyrproject.org/daily_tests/versions.json would not be used anymore ? I suggest we keep it as weekly frequency will not be sufficient in release periods. 2. Is there a gating protocol in order to avoid publishing a "bad" commit with build issues that would prevent having meaningful results? If there's only one test result upload per week, we need to ensure a minimum quality is guaranteed 3. The duration of an automated test campaign on 14 boards is less than 3 hours (when there are no major issues). This of course depends on the machine used, but even with twice this duration, this can be done in a night. I'm curious about the obstacles to perform such campaigns on a daily basis. Cheers Erwan On Tue, 27 Oct 2020 at 07:04, Hake Huang <hake.huang@...> wrote:
|
|
|
|
Hake Huang
Hi Erwan,
Thanks for your comments. Please check my inline comments below(lead by Hake)
Regards, Hake
From: Erwan Gouriou <erwan.gouriou@...>
Sent: 2020年10月27日 16:42 To: Hake Huang <hake.huang@...> Cc: testing-wg@... Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite
Caution: EXT Email Hi Hake,
I have few questions indeed: 1. Does it mean that https://testing.zephyrproject.org/daily_tests/versions.json would not be used anymore ? I suggest we keep it as weekly frequency will not be sufficient in release periods. [Hake] the versions is kept also. 2. Is there a gating protocol in order to avoid publishing a "bad" commit with build issues that would prevent having meaningful results? If there's only one test result upload per week, we need to ensure a minimum quality is guaranteed [Hake] we have some discussion this week, but we need define a scope on what to check, let me creat an issue here. 3. The duration of an automated test campaign on 14 boards is less than 3 hours (when there are no major issues). This of course depends on the machine used, but even with twice this duration, this can be done in a night. I'm curious about the obstacles to perform such campaigns on a daily basis. [Hake] According to my experiences, there are some efforts required, e.g. I meet download issue regularly and some time an application blocks the board due to many things. So it is no a zero cost thing, (qemu board testing is totally different from real board testing). Also the machine performance is an issue. Not everyone has a powerful build machine, for me the configurations are 2 4-core desktop PCs, which take me a whole day to finish 6 boards if there is not issues found, If something stack, I need restart another round of test cycle from beginning.
Cheers Erwan
On Tue, 27 Oct 2020 at 07:04, Hake Huang <hake.huang@...> wrote:
|
|
|
|
Erwan Gouriou
Thanks Hake for the feedback, it seems I underestimated the impact of test machine performance. On Tue, 27 Oct 2020 at 09:59, Hake Huang <hake.huang@...> wrote:
|
|
|
|
Nashif, Anas
I did not suggest using another file, we should use exactly the same file but expand the data we store there:
I do not think creating a new file is needed..
Anas
From:
<testing-wg@...> on behalf of Hake Huang <hake.huang@...>
Hi All,
In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.
https://github.com/zephyrproject-rtos/test_results/pull/141
Regards, Hake
From: testing-wg@... <testing-wg@...>
On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020年10月26日 22:20 To: testing-wg@... Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite
Caution: EXT Email Zephyr: Testing WG weekly call When: Where: Organizer: testing-wg@... An RSVP is requested. Click here to RSVP Description: Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing ________________________________________________________________________________ +1 321-558-6518 United States, Orlando (Toll) Conference ID: 831 716 531# Local numbers | Reset PIN | Learn more about Teams | Meeting options
________________________________________________________________________________ |
|
|
|
Hake Huang
Hi All,
Anash update this feature in https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54
Please check this PR, I have close my former PR.
Regards, Hake
From: testing-wg@... <testing-wg@...>
On Behalf Of Hake Huang via lists.zephyrproject.org
Sent: 2020年10月27日 17:00 To: Erwan Gouriou <erwan.gouriou@...> Cc: testing-wg@... Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite
Caution: EXT Email Hi Erwan,
Thanks for your comments. Please check my inline comments below(lead by Hake)
Regards, Hake
From: Erwan Gouriou <erwan.gouriou@...>
Caution: EXT Email Hi Hake,
I have few questions indeed: 1. Does it mean that https://testing.zephyrproject.org/daily_tests/versions.json would not be used anymore ? I suggest we keep it as weekly frequency will not be sufficient in release periods. [Hake] the versions is kept also. 2. Is there a gating protocol in order to avoid publishing a "bad" commit with build issues that would prevent having meaningful results? If there's only one test result upload per week, we need to ensure a minimum quality is guaranteed [Hake] we have some discussion this week, but we need define a scope on what to check, let me creat an issue here. 3. The duration of an automated test campaign on 14 boards is less than 3 hours (when there are no major issues). This of course depends on the machine used, but even with twice this duration, this can be done in a night. I'm curious about the obstacles to perform such campaigns on a daily basis. [Hake] According to my experiences, there are some efforts required, e.g. I meet download issue regularly and some time an application blocks the board due to many things. So it is no a zero cost thing, (qemu board testing is totally different from real board testing). Also the machine performance is an issue. Not everyone has a powerful build machine, for me the configurations are 2 4-core desktop PCs, which take me a whole day to finish 6 boards if there is not issues found, If something stack, I need restart another round of test cycle from beginning.
Cheers Erwan
On Tue, 27 Oct 2020 at 07:04, Hake Huang <hake.huang@...> wrote:
|
|
|
|
Hake Huang
Thanks Anas.
My reason to create a new weekly json file.
{<Week of year> : <version tag>}
So it will be very easy to retrieve any given weekly’s hash tag, by Data = json.load(…) if week in data: version = data[week]
I see your PR at https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54 , and I leave my comments there. Thanks.
Regards,
From: Nashif, Anas <anas.nashif@...>
Sent: 2020年10月27日 19:55 To: Hake Huang <hake.huang@...>; testing-wg@... Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite
Caution: EXT Email I did not suggest using another file, we should use exactly the same file but expand the data we store there:
I do not think creating a new file is needed..
Anas
From:
<testing-wg@...> on behalf of Hake Huang <hake.huang@...>
Hi All,
In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.
https://github.com/zephyrproject-rtos/test_results/pull/141
Regards, Hake
From:
testing-wg@... <testing-wg@...>
On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Caution: EXT Email Zephyr: Testing WG weekly call When: Where: Organizer: testing-wg@... An RSVP is requested. Click here to RSVP Description: Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing ________________________________________________________________________________ +1 321-558-6518 United States, Orlando (Toll) Conference ID: 831 716 531# Local numbers | Reset PIN | Learn more about Teams | Meeting options
________________________________________________________________________________ |
|
|