Dear All,
I would like to discuss the nomenclature used in sanitycheck reporting during our next meeting. It will relate to the issue: 
https://github.com/zephyrproject-rtos/zephyr/issues/30100#event-4013779249
I already made a comment in few places quite some time ago with my concerns about a confusion (and it seems not only for me) this could cause, so I will just copy it again here [I guess creating a PR next time would be better than comments section/slack]. I will verify to what extent the below is still present. I hope we could agree on using one name for each of items.

/home/maciej/zephyrproject2/zephyr/sc-venv/bin/python3.7 /home/maciej/zephyrproject2/zephyr/scripts/sanitycheck --build-only -T samples/hello_world/ --all --subset 2/120

Renaming output directory to /home/maciej/zephyrproject2/zephyr/sanity-out.1

INFO    - Running only a subset: 2/120

INFO    - JOBS: 8

INFO    - Selecting all possible platforms per test case

INFO    - Building initial testcase list...

INFO    - 3 test configurations selected, 8 configurations discarded due to filters.

INFO    - Adding tasks to the queue...

INFO    - 2 of 2 tests passed (100.00%), 0 failed, 1 skipped with 0 warnings in 3.15 seconds

INFO    - In total 2 test cases were executed on 269 out of total 272 platforms (98.90%)

INFO    - 0 tests executed on platforms, 2 tests were only built.

INFO    - Total complete:    2/   2  100%  skipped:    0, failed:    0

Process finished with exit code 0

I think this logging is still a bit confusing and could be cleaned out.
I find the line: In total 2 test cases were executed on 269 out of total 272 platforms (98.90%) confusing. It is not possible that only 2 test cases are run on 269 platforms. The issue is that 269 is the total number of platform preselected: len(self.selected_platforms) but then we only chose 3 platforms out of them (and 1 is skipped later on).
I think we should be more descriptive adding the info about the further limiting of the platforms. I think this number also gives the wrong impression, it looks like we've tested 98.9% platforms, but vast majority was just skipped in fact.

Another issue is how we count tests that were only built. In the line INFO - In total 2 test cases were executed on 269 out of total 272 platforms (98.90%) these 2 test cases were only built. It is also incoherent with the next line: 0 tests executed on platforms, 2 tests were only built. were build-only tests are subtracted

The last issue is that we use different naming for the same stuff. Test configurations is in fact the same as tests as both of them refers to len(suite.instances) in the code. IMO "test suites" from the testing language corresponds the best to what we are counting there. I guess "test suite" is not used since we have TestSuite class in the code which is even something different (it is a suite of all test suites).

So we have test configurations and tests that correspond to instances in the code and test suites in testing language. Test cases are at least coherent everywhere (but I am not sure the difference between test and test case is obvious) ;) Or did I miss something? I think sorting this nomenclature would be beneficial for anyone who would try to work with sanitycheck.

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

Hello, yesterday I sent a new link to the Release Summary Report.

Please don’t use it and ignore it, because my corporate Gdrive account will be still in use.

Use file link as you used before. No changes.

Thank you

Maksim

Hello, I migrated Release Summary report Document to my private GDrive, because our company team moved from GDrive to OneDrive.

Please refer to that new file link:

https://docs.google.com/document/d/12kEprn7uBkD238EZmQkvD9_hHV9ttcea8Ke-MszLRnc/edit?usp=sharing

Maksim

Hi All,

This is this weekly meeting agenda.

1.     Call for start weekly testing (All, 10 minuts)

This weekly tag will be created on Monday night US time.

2.     Test manager sytem survey and typical process discussion. (Maksim, 15 minutes)

3.     Round table disscussion. (all, 20 mintues)

Regards,

Hake

Hello, I finished my notes about Allure. During next meeting I will demonstrate its capabilities using our Zephyr results. You can also try to run it, easy to use, but have some issues which I mentioned in the document.

P.S. Please let me know if you can open Cloud based link of my Document (I’m not sure if outside of Intel you can open it, if not, use .docx file attached)

Maksim

Hi Maciej

It is 2:00pm to 3:00pm, (GMT+00:00) UTC

Regards,

Hake

Hi Hake,
Could you please state the time in UTC? Is it as in call for RSVP: 14-15 UTC?

Hi All,

This is a notice to that weekly call will be 1 hour later for regions that do not use day time saving shift(e.g. China), if you have problem with this time, please propose your favourite time.

Regards,

Hake

Hi All,

This is this weekly meeting agenda.

1.     versions.json with weekly usage case intro. (Anas, 10 minuts)

https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54

2.     Feedback on Release metrics document. (Anas, Meciej, 15 mintues)

https://docs.google.com/document/d/1y7mtAhmvl5z1ohazNgXFEXdFg6RO-J38QsFA7twPPW8/edit?pli=1#

3.     Test manager sytem survey and typical process discussion. (Maksim, 15 minutes)

4.     Round table disscussion. (all, 20 mintues)

Regards,

Hake

Hi,

Same here, won’t be able to attend today. Sorry.

Anas

Hi,

I will have to skip today’s meeting as well.

Regards,

Maciej

Hi,

I will have to skip today’s meeting as well.

Regards,

Maciej

Hi All,

This is this weekly meeting agenda.

1.     Updates on versions.json update PR. (Anas, 10 minuts)

https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54

2.     Feedback on Release metrics document. (Anas, Meciej, 15 mintues)

https://docs.google.com/document/d/1y7mtAhmvl5z1ohazNgXFEXdFg6RO-J38QsFA7twPPW8/edit?pli=1#

3.     Seek for help on gPTP testing setup. (Hake, 15 minutes)

Does any one have the test envrion setting on Windows?

https://github.com/zephyrproject-rtos/zephyr/issues/29599

4.     Round table disscussion. (all, 20 mintues)

Regards,

Hake

Thanks Anas.

My reason to create a new weekly json file.

1. I find it is difficult to create a compatible json file. and I want to keep the versions.json consistent so that the existing parser will not change.
2. I want to make it easy to select a weekly test once you know the week of year. So I propose a dict format:

{<Week of year> : <version tag>}

So it will be very easy to retrieve any given weekly’s hash tag, by

Data = json.load(…)

if week in data:

   version  = data[week]

I see your PR at https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54 , and I leave my comments there. Thanks.

Regards,
Hake

Hi All,

Anash update this feature in https://github.com/zephyrproject-rtos/zephyr/pull/29583/files/2b962288f2058828282d100c18540928adc539bd..418ab8529567743a30a77ee6e5efdb4f66f7be54

Please check this PR, I have close my former PR.

Regards,

Hake

I did not suggest using another file, we should use exactly the same file but expand the data we store there:

• Add date
• Add a flag for weekly
• …

I do not think creating a new file is needed..

Anas

Hi All,

In this weekly meeting, Anas proposes that we shall create a regular candency to upload our test result. If we can not afford daily result upload, then we shall do a weekly upload in a given version, this PR is to create a new https://testing.zephyrproject.org/daily_tests/weeklys.json which contains the week of year and version tag in dict, so that we can use this json file to sync our efforts. In such way, we can get a full report with as many boards as possible in a given tag. Please kindly give you comments.

https://github.com/zephyrproject-rtos/test_results/pull/141

Regards,

Hake

Hi Erwan,

Thanks for your comments. Please check my inline comments below(lead by Hake)

Regards,

Hake