Dear Zephyr Testers,

I created a python script which automates release readiness `decision` based on number of issues in Zephyr using github’s REST API and the official filters for release issues from Zephyr’s Wiki. The idea is to use it for QA reporting. It can also serve as a base for Zephyr’s issues handling in future (e.g. adding burndown calculations etc). You can have a look at the script here:

https://github.com/PerMac/QA_Zephyr and we can discuss its usage on the upcoming testing wg meeting.

Cheers,

Maciej

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

I can work on The overview and goals sections of the report. I can also check want are the current ready-to-release standards (how many issues by severity can we have and is there anything else used). I could describe there that the # of open issues/bugs is crucial and during the release we evaluate each failing test and create an issue with severity level if not yet existing. Maybe I can also add the number of open issues by milestone and severity somewhere in the begging of the report, since apparently it is the most important information

I can take Performance KPI.

Thanks Maksim,

I can take the first section delivery metrics.

Regards,

Hake

Hello.

As I understand next week we are planning to fill in Summary Report.

Next week we can have initial report. AR [Everyone need to take section and update it for the next week] Link https://docs.google.com/document/d/1y7mtAhmvl5z1ohazNgXFEXdFg6RO-J38QsFA7twPPW8/edit#

Any thoughts what section do you want to take to fill in?

Maksim

Hi All,

Today’s meeting agenda:

1.     Last week AR using power BI to summary a report for release quality metrics. (Maksim, 20 minutes)

2.  Pull request explanation for https://github.com/zephyrproject-rtos/test_results/pull/68 (all, 20 minutes)

3.     Question s to Function working group. (Hake 5 minutes)

a)      I did not find there is a function working group mail list of discussion board, any one knows? How to get coverage plan from them?

4.     Round table discussion. (15 minutes)

Regards,

Hake

Awesome!

Regards,

Hake

Hello, finally I created a pull request, and you can try my script to upload test results. Please read Readme.

https://github.com/zephyrproject-rtos/test_results/pull/68

Maksim

Won't be able to join today.

Anas

Hi All,

This weekly meeting agenda:

1.  Introduction on automation process on uploading board testing report. (30 minutes, Maksim)

2.  Discussion on release quality metrics. (20 minutes, all)

a)   Maksim summary metrics discussion.

3.  Round table discussion. (20 minutes, all)

Regards,

Hake

Hi, I prepared Summary test report proposal, I have things to discuss. I added information from Hake and Maciej to that report, also added new descriptions.

Please review it  https://docs.google.com/document/d/1y7mtAhmvl5z1ohazNgXFEXdFg6RO-J38QsFA7twPPW8/edit?usp=sharing

Hi All,

This weekly meeting agenda:

1.     Introduction on automation process on uploading board testing report. (30 minutes, Maksim)

2.     Discussion on release quality metrics. (20 minutes, all)

a)      Maksim summary metrics discussion.

3.     Round table discussion. (20 minutes, all)

Regards,

Hake

Hi All,

This weekly meeting agenda:

1.     Discussion on release quality metrics. (20 minutes, all)

2.     Introduction on automation process on uploading board testing report. (20 minutes, Maksim)

3.     Round table discussion. (20 mintues, all)

Regards,

Haie

This is the commented version of quality metrics

Discuss on draft release quality metrics

https://drive.google.com/file/d/1DiH1lA4184GhNlt6vHJk7N8vU_q3SmKG/view?usp=sharing

Regards,

Hake

This is my draft version for zephyr release quality metrics.

Regards,

Hake

Most of the questions below can be answered by mail…

1.     Is Github Storage free or not? After Intel started to upload test results, each .xml file can have quite a big size, so please discuss how much storage we have on Github to store .xml test results. Hao, 10 minutes (according to gihub document it is 1G limitation for repo)

We should not be concerned with the storage limit. There is no limit really, why will anyone be uploading a 1G file? If your test results are in GB, then there is something really wrong with the results. Do you have examples of such large files being pushed?

1.     I have one issue, which I want to ask Anas. My script can push test results to my Github repository, but to create a PR from my repository test_results to the upstream test_results repository, I have to manually go to Github and submit a Pull request. Unfortunately Github doesn’t support auto PR submission into another repository. Next explanation is written there https://developer.github.com/v3/pulls/#create-a-pull-request “You cannot submit a pull request to one repository that requests a merge to a base of another repository.” Can I push a new branch to the upstream repository directly? Only that way is possible to create PR automatically. Hao 20 minutes (need discuss a solid solution)

Not sure what you mean here, there should not be a problem creating PRs automatically. See https://pygithub.readthedocs.io/en/latest/examples/PullRequest.html

This works just fine, I just tried it.

See this https://github.com/zephyrproject-rtos/test_results/pull/51, it was created automatically from my fork.

1.     To use my script which I introduced above, necessary for each company to have the same .xml file names for the test results. Necessary to make an agreement about .xml Sanitycheck test results file names. Hao 20 mintues (need hao to introduce the code logic)

This was already mentioned somewhere else, the file names are should not be random, otherwise the scripts wont be able to find them. So, right now it is expected to have the board name as the only thing in the file name, so <board name>.xml.

Anas

Hi All,

This week’s meeting agenda 2020/7/20

1.     Is Github Storage free or not? After Intel started to upload test results, each .xml file can have quite a big size, so please discuss how much storage we have on Github to store .xml test results. Hao, 10 minutes (according to gihub document it is 1G limitation for repo)

2.     I created a script to automatically upload test results and now I’m discussing with Anas the possibility to upstream it.

3.     I have one issue, which I want to ask Anas. My script can push test results to my Github repository, but to create a PR from my repository test_results to the upstream test_results repository, I have to manually go to Github and submit a Pull request. Unfortunately Github doesn’t support auto PR submission into another repository. Next explanation is written there https://developer.github.com/v3/pulls/#create-a-pull-request “You cannot submit a pull request to one repository that requests a merge to a base of another repository.” Can I push a new branch to the upstream repository directly? Only that way is possible to create PR automatically. Hao 20 minutes (need discuss a solid solution)

4.     To use my script which I introduced above, necessary for each company to have the same .xml file names for the test results. Necessary to make an agreement about .xml Sanitycheck test results file names. Hao 20 mintues (need hao to introduce the code logic)

5.     The quality metrics discussion. (10 minutes)

Regards,

Hake

Hi All,

This week’s meeting agenda 2020/7/20

1.     Is Github Storage free or not? After Intel started to upload test results, each .xml file can have quite a big size, so please discuss how much storage we have on Github to store .xml test results. Hao, 10 minutes (according to gihub document it is 1G limitation for repo)

2.     I created a script to automatically upload test results and now I’m discussing with Anas the possibility to upstream it.

3.     I have one issue, which I want to ask Anas. My script can push test results to my Github repository, but to create a PR from my repository test_results to the upstream test_results repository, I have to manually go to Github and submit a Pull request. Unfortunately Github doesn’t support auto PR submission into another repository. Next explanation is written there https://developer.github.com/v3/pulls/#create-a-pull-request “You cannot submit a pull request to one repository that requests a merge to a base of another repository.” Can I push a new branch to the upstream repository directly? Only that way is possible to create PR automatically. Hao 20 minutes (need discuss a solid solution)

4.     To use my script which I introduced above, necessary for each company to have the same .xml file names for the test results. Necessary to make an agreement about .xml Sanitycheck test results file names. Hao 20 mintues (need hao to introduce the code logic)

5.     The quality metrics discussion. (10 minutes)

Regards,

Hake