I have few questions indeed:

1. Does it mean that https://testing.zephyrproject.org/daily_tests/versions.json would not be used anymore ?

    I suggest we keep it as weekly frequency will not be sufficient in release periods.

     [Hake] the versions is kept also.

2. Is there a gating protocol in order to avoid publishing a "bad" commit with build issues that would prevent

    having meaningful results? If there's only one test result upload per week, we need to ensure a minimum quality

    is guaranteed

     [Hake] we have some discussion this week, but we need define a scope on what to check, let me creat an issue here.

3. The duration of an automated test campaign on 14 boards is less than 3 hours (when there are no major issues).

    This of course depends on the machine used, but even with twice this duration, this can be done in a night.

    I'm curious about the obstacles to perform such campaigns on a daily basis.

       [Hake] According to my experiences, there are some efforts required, e.g. I meet download issue regularly and some time an application blocks the board due to many things. So it is no a zero cost thing, (qemu board testing is totally different from real board testing). Also the machine performance is an issue. Not everyone has a powerful build machine, for me the configurations are 2 4-core desktop PCs, which take me a whole day to finish 6 boards if there is not issues found, If something stack, I need restart another round of test cycle from beginning.

Cheers

Erwan

On Tue, 27 Oct 2020 at 07:04, Hake Huang <hake.huang@...> wrote: