Hi Hake,

Regarding reference boards and tiers, I think we need to expand on this and start drafting a proposal on what all of this means and how these fits in the release criteria alongside other items we want to have. Take the process meeting discussion, the discussion we had in the testing WG and create a more detailed proposal of what the goal is, the expectations and how this will translate into execution.

Regarding test scope, a thorough review of our tests and test structure and a way for us to label/tag tests in a structured way are needed. This should also come as a detailed proposal. Maybe a review of all our tests, their types, classes, priorities and how important it is to run certain tests on hardware is going to be necessary to proceed with this.

As for the test management system and where test reports would go, we discussed having a comparison table with the proposed solutions and some criteria for selection, do we have this already? What are the current from runners?

Anas

Hi Anas,

In today’s meeting, we are impressed by your introduce the new test report structure with many thanks, and we notice that you stress the new TSM system as well.

To break down targets into well accepted steps, I’d like to ask your help to list your intended criteria.  There are too many items in your introduction, and I guess most of us can’t get full track.

For T1 boards:

1.       The T1 boards has test commitment

2.       The T1 boards need to be broadly available

3.       The T1 boards need to be debug able

What is the criteria to select test suite for T1 boards?

As the TSM system, when can we use the newer one? Or still pending for community feedback?

Thanks again.

Regards,

Hake

Hi Anas,

Thanks a lot for your reply.

We have discussed the boards criteria see below and test scope(Piotr Gołyźniak(Nordic) will send out his summary table for all testcases).

1.   At least one from each architecture(arm , arm64, x86, upsqure, intel-adsp)

2.   At least one for each sub-platform (e.g. NXP RT series, LPC series, imx8 series)

RT cortex-m7/cortex-mx33, LPC cortex-m4/m0, imx8 arm64/amp(cortex m / a)

3.      At least one for each function group as leading platform(networking, api)

As to the TMS comparison table please see attachment.

Regards,

Hake

Hi All,

Agenda for this weekly meeting

1.     Tire 1 Board testing  list discussion follow up.

https://github.com/zephyrproject-rtos/zephyr/issues/38566#issuecomment-1204232175 

• T1 boards Proposal discussion:

At least one from each architecture(arm , arm64, x86, upsqure, intel-adsp)

At least one for each sub-platform (e.g. NXP RT series, LPC series, imx8 series)

·         RT cortex-m7/cortex-mx33, LPC cortex-m4/m0, imx8 arm64/amp(cortex m / a)

At least one for each function group as leading platform(networking, api)

• Test cases review structure draft: see attachment

2.    TMS system discussion. See attachment

3.    Round table

Regards,

Hake

Hi All,

This weekly meeting agenda.

1.     Tire 1 Board testing  list discussion follow up.

https://github.com/zephyrproject-rtos/zephyr/issues/38566#issuecomment-1204232175 

·         T1 boards Proposal discussion:

·         Test cases review structure draft:

Based on current test cases -> review them (remove bluetooth, networking) review link is below:

https://docs.google.com/spreadsheets/d/1NAC2SscIW4Wmz_9MScDeZ7VnXi0vO8dHkrKnUCT7Bfk/edit?usp=sharing 

process working group background information:

https://docs.google.com/presentation/d/10Q-B0Upv1bBaoQRNM5n9yUEMeXO6NINnFNDkA86RCsM/edit?usp=sharing

2.    TMS system discussion. 

 https://github.com/marketplace/testquality/plan/MDIyOk1hcmtldHBsYWNlTGlzdGluZ1BsYW44MTM=#pricing-and-setup  

3.    Round table

Regards,

Hake

Hi All,

This is this weekly meeting minutes.

Highlights:

We will be continue discussing the T1 boards test suites, you can use the template below to fill your proposed cases.

https://docs.google.com/spreadsheets/d/1NAC2SscIW4Wmz_9MScDeZ7VnXi0vO8dHkrKnUCT7Bfk/edit

Thursday 9/15 2022

Attendee: 

• Aaron Massey(Google)
• AI Semjonvos(Google)
• Alexey Brodkin(Synopsys)
• Artern Panfilov(Synopsys)
• Danny (Oticon)
• David Leach(NXP)
• Enjia Mai(intel)
• Eugeniy Paltsev(Synopsys)
• Hake Huang (NXP)
• Hongbo(Intel)
• Hein Simon(Baumer Electric AG)
• Lixin Guo(Intel)
• Jocelyn Li (Intel)
• Jereme Bettis(Google)
• Jinru wang(Synopsys)
• Kosycarz Piotr (Nordic)
• Maciej Perkowski (Nordic)
• Meng xianglin(Intel)
• Matthew Holenko (Antmicro)
• Nashif Anas(Intel)
• Paul Fagerburg(google)
• Peng Cheng(Intel)
• Piotr Zierhoffer (Antmicro)
• Piotr Gołyźniak(Nordic)
• Qi Chervl(T-Mobile)
• Rob Woolley(Wind River)
• Ruffer Dennis(T-Mobile) 
• Shao ming(Intel)
• Steven Wang Li(Intel)
• Stephanos loannidis(Nodic)
• Rattanakone Raam(T Mobile)
• Yerabolu Spoorthy Priya(Intel) 
• Yves Vandervennet(NXP)

    Tire 1 Board testing  list discussion follow up.

https://github.com/zephyrproject-rtos/zephyr/issues/38566#issuecomment-1204232175 

·         T1 boards Proposal discussion:

·         Test cases review structure draft:

Based on current test cases -> review them (remove bluetooth, networking) review link is below:

https://docs.google.com/spreadsheets/d/1NAC2SscIW4Wmz_9MScDeZ7VnXi0vO8dHkrKnUCT7Bfk/edit?usp=sharing 

<maceij> split the sheet into submdules and ask owner to review them?

1.    General guide line.

a.      How to group the test

b.      Case shall be running on as many boards as possible

c.      Case shall be run without special fixtures/add-on boards

d.      Each module shall has at leat one test(may not true for some peripheral)

e.      Exclude samples

                     Or start from 0 cases and select cases in to T1 bottom up way

We can use the template here(https://docs.google.com/spreadsheets/d/1NAC2SscIW4Wmz_9MScDeZ7VnXi0vO8dHkrKnUCT7Bfk/edit)

process working group background information:

https://docs.google.com/presentation/d/10Q-B0Upv1bBaoQRNM5n9yUEMeXO6NINnFNDkA86RCsM/edit?usp=sharing

Regards,

Hake

Hi All,

This weekly meeting agenda.

l  We have basic list of testcases, please review and feedback, review link is below:

https://docs.google.com/spreadsheets/d/1NAC2SscIW4Wmz_9MScDeZ7VnXi0vO8dHkrKnUCT7Bfk/edit?usp=sharing 

l  round table

Regards,

Hake

HI All,

This weekly meeting minutes. We have finalize the proposed minimal testing suites as below, and we expect to more members to confirm next week.

Thursday 9/29 2022

Attendee: 

• Aaron Massey(Google)
• AI Semjonvos(Google)
• Alexey Brodkin(Synopsys)
• Artern Panfilov(Synopsys)
• Danny (Oticon)
• David Leach(NXP)
• Enjia Mai(intel)
• Eugeniy Paltsev(Synopsys)
• Hake Huang (NXP)
• Hongbo(Intel)
• Hein Simon(Baumer Electric AG)
• Lixin Guo(Intel)
• Jocelyn Li (Intel)
• Jereme Bettis(Google)
• Jinru wang(Synopsys)
• Kosycarz Piotr (Nordic)
• Maciej Perkowski (Nordic)
• Meng xianglin(Intel)
• Matthew Holenko (Antmicro)
• Nashif Anas(Intel)
• Paul Fagerburg(google)
• Peng Cheng(Intel)
• Piotr Zierhoffer (Antmicro)
• Piotr Gołyźniak(Nordic)
• Qi Chervl(T-Mobile)
• Rob Woolley(Wind River)
• Ruffer Dennis(T-Mobile) 
• Shao ming(Intel)
• Steven Wang Li(Intel)
• Stephanos loannidis(Nodic)
• Rattanakone Raam(T Mobile)
• Yerabolu Spoorthy Priya(Intel) 
• Yves Vandervennet(NXP)

• We have basic list of testcases, please review and feedback, review link is below:

https://docs.google.com/spreadsheets/d/1NAC2SscIW4Wmz_9MScDeZ7VnXi0vO8dHkrKnUCT7Bfk/edit?usp=sharing 

Below are the proposed minimal test cases for T1 boards

• l  round table

https://github.com/zephyrproject-rtos/zephyr/issues/50354  Semjonovs will follow up

Hi All,

This weekly meeting agenda.

l  We have basic list of testcases, please review and feedback, review link is below:

• https://docs.google.com/spreadsheets/d/1NAC2SscIW4Wmz_9MScDeZ7VnXi0vO8dHkrKnUCT7Bfk/edit?usp=sharing 
  
  Below are the proposed minimal test cases for T1 boards

l  round table

Regards,

Hake

Hi All,

This weekly meeting agenda

1.     Discussion on T1 test suites

Proposal: add tag to every cases scenario, 

Rules are:

1.       Only some filters can be allowed.

CONFIG_PICOLIBC_SUPPORTED, will create a github issue and Stephan will help to solve (https://github.com/zephyrproject-rtos/zephyr/issues/51043)

2.         No arch dependency

Now the test list for T1 boards are

source

https://docs.google.com/spreadsheets/d/1NAC2SscIW4Wmz_9MScDeZ7VnXi0vO8dHkrKnUCT7Bfk/edit?usp=sharing 

https://docs.google.com/document/d/1Z_9zhmM6EK09N1lQj-rvaScoVPDG_abJtV85JE7c3n4/edit?usp=sharing 

Regards,

Hake

Hi All,

This weekly meeting agenda

1.     T1 board test cases follow up

https://github.com/zephyrproject-rtos/zephyr/pull/51287

https://github.com/zephyrproject-rtos/zephyr/pull/51288

and the overall features request link is here

https://github.com/zephyrproject-rtos/zephyr/issues/51286

2.     Round table

Regards,

Hake

Hi All,

Agenda for this weekly meeting.

1.     Follow up for T1 board test cases.

2.     Round table.

Regards,

Hake

Hi All,

We do not have specific topic for this weekly meeting. But we can quick sync on the status with

1.     Tag for test cases refactor

https://github.com/zephyrproject-rtos/zephyr/issues/51718

2.     Opensearch status if Anash can present.

Just reminder for all the weekly meeting is moved 1 hour later due to Day time saving in winter.  

Regards,

Hake

Hi Hake,
Could you please specify if the original time stays or was it changed: I.e. is the meeting at 7:00-8:00 (UTC-08:00) Pacific Time or at some other time?

Cheers

Hi All,

This weekly meeting agenda:

1.         Follow up: Issue on how to split build and run in twister, by Piotr(Nordic).

2.         Around table discussion.

Regards,

Hake

Hi All,

This weekly meeting agenda:

1.     Round table discussion.

2.     Follow up issues

https://github.com/zephyrproject-rtos/zephyr/issues/51718

add  a default meta named ‘level’ as below

level:

-      unit/integration/acceptance/system/general

-      simulation/hardware/host

yellow  is default one

Regards,

Hake

Hi All,

This weekly meeting agenda:

Today I did not get any special topics from any member, so we can have a short sync, and round table discussion/

1. Round table discussion.

Regards,

Hake