Thursday 03/30 2023

Attendee: 

• Aaron Massey(Google)   
• Eugeniy Paltsev(Synopsys)
• Hake Huang (NXP)
• Jakub Jatczak(Antmicro)
• Kosycarz Piotr (Nordic)
• Maciej Perkowski (Nordic)
• Nashif Anas(Intel)
• Piotr Gołyźniak(Nordic)

1. T1 board testing plan progress updates
2. Discussion on LTS release(Zephyr v2.7.5)

Please help review:

https://github.com/zephyrproject-rtos/zephyr/pull/55843

> the tests/test_config.yaml will be the community test plan, and its has two parts

1.    For smoke testing

2.    For acceptance testing

       This will be documented

       For test boards, the boards level will be summarized from test result repo statistics

 In the future the LTS test will have T0 and T1 tested, and this will be documented.

Regards,

Hake

Agenda for this weekly meeting

1.     T1 board testing plan progress updates

Please help review:

https://github.com/zephyrproject-rtos/zephyr/pull/55843

2.     Discussion on LTS release(Zephyr v2.7.5)

Do we have solid test strategy for those LTS release?

3.     Round table

Regards,

Hake

Hi all:

Thursday 03/23 2023

Attendee: 

• Hake Huang (NXP)
• Jakub Jatczak(Antmicro)
• Kosycarz Piotr (Nordic)
• Piotr Zierhoffer (Antmicro)
• Piotr Gołyźniak(Nordic)
• Ruffer Dennis(T-Mobile)
• Yves Vandervennet(NXP)

1.   Round table discussion.

 Tools for json result review recommends.

Sublime text + pretty json plugin

https://packagecontrol.io/packages/Pretty%20JSON

2.   PR review reminder

Update test config document, add how to use test config with test level with default test list

https://github.com/zephyrproject-rtos/zephyr/pull/55843

Regards,

Hake

Hi,

I have a conflict today and won’t be able to join the meeting.

Anas

Hi All,

This weekly meeting agenda:

1.       Round table discussion.

2.       PR review reminder

Update test config document, add how to use test config with test level with default test list

https://github.com/zephyrproject-rtos/zephyr/pull/55843

Regards,

Hake

Hi All,

This weekly meeting agenda:

1.     Round table discussion.

2.     PR review reminder

Update test config document, add how to use test config with test level with default test list

https://github.com/zephyrproject-rtos/zephyr/pull/55843

Regards,

Hake

Hi All,

Thursday 03/16 2023

Attendee: 

• Eugeniy Paltsev(Synopsys)
• Hake Huang (NXP)
• Jakub Jatczak(Antmicro)
• Maciej Perkowski (Nordic)
• Nashif Anas(Intel)
• Piotr Gołyźniak(Nordic)
• Richert, Szymon1(Intel)
• Ruffer Dennis(T-Mobile)
• Stephanos Ioannidis(Nodic)

1.     Updates on elasticsearch report status.

General: 

- Titles for all tables. Meaning of some graphs is less obvious for people not familiar with the CI.

- Not clear which graph is for a given version and which is a sum over multiple. Better to have it clearly stated (in a title?)

- Maybe version on X-axis instead of time? If time, maybe a lookup table matching time with a version will be helpful?

Existing graphs:

- Retries table: Name says top 10 but are more than 1. What "other" means? Should "other" be disabled?

- "Coverage" in the title of 2 bottom graphs can be misleading ("does it refer to test coverage?" “Isn’t coverage in %”, etc.). Maybe "Abundance" is better?

- Why there are spikes in testcase counting over time? Is it issue with bucketing (adding multiple entries to one point in time). If so, this probably can be fixed by changing bucket size to sth small, such that two pushes won’t fall to the same bucket and be visualized together (summed).

Ideas for new dashboards:

- table with the most flaky platform: platform with the most retries (for all tests) summed in recent 30 runs

- table the most flaky tests: test with the most retries (for all platforms) summed in recent 30 runs

- table with the most often failing platforms?

- pie chart with a version summary. The graph vs time is already there, but pie charts can be eye grabbing.

- list of heaviest (by memory footprint) configurations.

- table of "other" results. I.e. all non [passed, skipped, filtered] statuses. helps to find unexpected statuses.

- if quarantine will be used: a special table listing what was quarantined in a given version.

 Already asked by Hake:

- ability to compare the same testsuite on multiple platforms (similar to current daily_testing results)

- Memory footprints in time for a given suite

2.     T1 test board and plan status updates

 https://github.com/zephyrproject-rtos/zephyr/pull/55843 

Will change this PR to documents improvements

Regards,

Hake

Hi,

We looked through available dashboards and here are our observations/ideas:

General: 

- Titles for all tables. Meaning of some graphs is less obvious for people not familiar with the CI.

- Not clear which graph is for a given version and which is a sum over multiple. Better to have it clearly stated (in a title?)

- Maybe version on X-axis instead of time? If time, maybe a lookup table matching time with a version will be helpful?

Existing graphs:

- Retries table: Name says top 10 but are more than 1. What "other" means? Should "other" be disabled?

- "Coverage" in the title of 2 bottom graphs can be misleading ("does it refer to test coverage?" “Isn’t coverage in %”, etc.). Maybe "Abundance" is better?

- Why there are spikes in testcase counting over time? Is it issue with bucketing (adding multiple entries to one point in time). If so, this probably can be fixed by changing bucket size to sth small, such that two pushes won’t fall to the same bucket and be visualized together (summed).

Ideas for new dashboards:

- table with the most flaky platform: platform with the most retries (for all tests) summed in recent 30 runs

- table the most flaky tests: test with the most retries (for all platforms) summed in recent 30 runs

- table with the most often failing platforms?

- pie chart with a version summary. The graph vs time is already there, but pie charts can be eye grabbing.

- list of heaviest (by memory footprint) configurations.

- table of "other" results. I.e. all non [passed, skipped, filtered] statuses. helps to find unexpected statuses.

- if quarantine will be used: a special table listing what was quarantined in a given version.

 Already asked by Hake:

- ability to compare the same testsuite on multiple platforms (similar to current daily_testing results)

- Memory footprints in time for a given suite

Cheers,

Maciej Perkowski | Software Test Developer
M +48 728 395 111 | Kraków, Poland
nordicsemi.com | devzone.nordicsemi.com

Hi All,

Remand:

Today the meeting time is moved 1 hour early in none- day time saying zone.

Today’s agenda:

1.     Updates on elasticsearch report status.

2.     T1 test board and plan status updates

3.     Round table

Regards,

Hake

Thursday 03/09 2023

Attendee: 

• Eugeniy Paltsev(Synopsys)
• Hake Huang (NXP)
• Jakub Jatczak(Antmicro)
• Maciej Perkowski (Nordic)
• Nashif Anas(Intel)
• Piotr Gołyźniak(Nordic)
• Ruffer Dennis(T-Mobile)
• Stephanos loannidis(Nodic)
• Yves Vandervennet(NXP)
  
  
  

1.    T1 test case catalog discussion.

a. Need document it test plan first. Anas will do that

b. We can start to push test plan to zephyr mainline.

c. Pushing HW release to Elastic-search.

2.      Follow up last week’s twisterv2 discussion

Maciej will prepare a demo for v2.

3.  Remove the old ztest api, and keep the ztest new.

       Old ztest api still used like blue-tooth.  Need check

Regards,

Hake

Hi All,

Today’s weekly meeting agenda:

1.       T1 test case catalog discussion.

https://github.com/zephyrproject-rtos/zephyr/issues/51286

what’s next step after this PR

https://github.com/zephyrproject-rtos/zephyr/pull/52715

2.     Round table

Regards,

Hake

Thursday 03/02 2023

Attendee: 

• Aaron Massey(Google)   
• Eugeniy Paltsev(Synopsys)
• Hake Huang (NXP)
• Jakub Jatczak(Antmicro)
• Maciej Perkowski (Nordic)
• Ruffer Dennis(T-Mobile)
• Yves Vandervennet(NXP)
  
  
  

1.     Status and plan update for twisterv2.

       Nordic need gather more feedbacks on this project.

              e.g. End to end testing features.

       A better document needed for community to use it.

       A plan of record is required for new features and twisterv1 features readiness

2.     T1 test case catalog discussion.

https://github.com/zephyrproject-rtos/zephyr/issues/51286

And please review this PR

https://github.com/zephyrproject-rtos/zephyr/pull/52715

We have  platform type/

https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/schemas/twister/testsuite-schema.yaml#L307 

3.     Round table.

4.     Short notice:

Too many patches merged after v3.3.0 release, need check quality.

Regards，

hake

Apologies, I will not be able to attend today.

Need this to be reviewed to support test plans and board tiers.

https://github.com/zephyrproject-rtos/zephyr/pull/52715

Anas

Hi All,

Today’s weekly meeting agenda.

1.       Status and plan update for twisterv2.

2.       T1 test case catalog discussion.

https://github.com/zephyrproject-rtos/zephyr/issues/51286

3.       Round table.

4.       Short notice:

Too many patches merged after v3.3.0 release, need check quality.

Regards,

Hake

Hi All,

Today’s weekly meeting agenda.

1.     Status and plan update for twisterv2.

2.     T1 test case catalog discussion.

https://github.com/zephyrproject-rtos/zephyr/issues/51286

3.     Round table.

4.     Short notice:

Too many patches merged after v3.3.0 release, need check quality.

Regards,

Hake

Hi All,

Thursday 02/23 2023

Attendee: 

·         Aaron Massey(Google)    

·         Hake Huang (NXP)

·         Maciej Perkowski (Nordic)

·         Nashif Anas(Intel)

·         Piotr Zierhoffer (Antmicro)

·         Piotr Gołyźniak(Nordic)

·         Ruffer Dennis(T-Mobile)

1. Twister2 feedbacks.

Need keep consistent with twister v1(input/output, user interfaces)

We need a timeline plan on below issues:

- parallel build.

 With -n auto, but only valid for native_posix and qemu

- need a step by step guide, the readme seems not so friendly

- still have many features missing https://github.com/zephyrproject-rtos/twister/issues/135

Need be ready before we moves to twister2

- intergrate with github actions status

Maceij need to have a commitment on twister2

Regards,

Hake