toggle quoted messageShow quoted text
Thanks Hake for the feedback, it seems I underestimated the impact of test machine performance.
Thanks for your comments. Please check my inline comments below(lead by Hake)
I have few questions indeed:
I suggest we keep it as weekly frequency will not be sufficient in release periods.
[Hake] the versions is kept also.
2. Is there a gating protocol in order to avoid publishing a "bad" commit with build issues that would prevent
having meaningful results? If there's only one test result upload per week, we need to ensure a minimum quality
[Hake] we have some discussion this week, but we need define a scope on what to check, let me creat an issue
3. The duration of an automated test campaign on 14 boards is less than 3 hours (when there are no major issues).
This of course depends on the machine used, but even with twice this duration, this can be done in a night.
I'm curious about the obstacles to perform such campaigns on a daily basis.
[Hake] According to my experiences, there are some efforts required, e.g. I meet download issue regularly and some time an application blocks the board due to many things. So it is no a zero cost thing, (qemu
board testing is totally different from real board testing). Also the machine performance is an issue. Not everyone has a powerful build machine, for me the configurations are 2 4-core desktop PCs, which take me a whole day to finish 6 boards if there is not
issues found, If something stack, I need restart another round of test cycle from beginning.
In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not
afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new
https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag.
Please kindly give you comments.
Zephyr: Testing WG weekly call
Monday, 2 November 2020
2:00pm to 3:00pm
Repeats: Weekly on Monday
Microsoft Teams Meeting
An RSVP is requested.
Click here to RSVP