Date   

Zephyr: Testing WG kick-off call

Brett Preston
 

Members of the Zephyr Testing Working Group,

The kick-off call for the group has been moved to next week.

You should have just received a Doodle to capture your availability, but if not, the link is below:

Please complete the Doodle by 12pm PDT on Friday, July 27.

Thank you,


Brett

--
*Brett Preston*
The Linux Foundation
+1 (971) 303-9030
bpreston@...

Google Talk: bpreston@...
Skype: bprestoncf


initial mail for zephyr testing plan

Hake Huang
 

Hi All,

 

I initialize a test plan based on IEEE 829 template. https://github.com/zephyrproject-rtos/qm/pull/2.

 

I propose we can start at defining below things:

 

1.     Testing Items

a)      Hardware platforms

b)      Software components

e.g. the zephyr project can be divided intro

1. kernel

2. boards drivers

       Each company shall have its own test policy on their drivers

3. subsystems

4. samples

5. unit tests

2.     How the existing samples and unit test meet our test requirements.

3.     Long term / short term quality strategy for test items, and standards.

a)      I think for short term, we shall ensure that all features have test coverage, then function coverage, then line/branch coverage(based on requirements)

b)      We shall have user cases definition for subsystem cases, and this need help from TSC.

4.     Issue reporting and tracking

a)      Would current process enough, Reporting on github -> reproduce -> fix -> close.

b)      Current the issue severity is defined by code owner, would this be acceptable?

5.     Test environment definition and tools

a)      Qume simulation environment(already defined)

b)      Real board testing(each company shall provide this)

c)      Simulation such as renode(working group shall define it in the future)

d)      What tools we are recommend to use

                 i.          TestRail report scripts

                ii.          Renode, do we have free support from renode?

1.      I check the source code, I understand they use the cmsis-svd file to get the memory map of SOC, but how could renode know the SOC driver model? In other simulation they need to have a fast model to define the driver behavior.

 

Thanks for your comments!

 

Regards,

Hake


Re: initial mail for zephyr testing plan

Aga, Svein
 

Hi,

 

I’ve proposed some changes/additions below in blue + some comments and questions.  

 

A more general comment is that the Zephyr project can define a top level master test plan that is common for everyone contributing. Each subsystems and drivers (and the like) can have their own test plan, specifying the additions and deviations from the Zephyr master test plan. It could then be possible to have traceability from top level and down to subsystem testing. Each of the subsystems can define their own scope within a v-model with proper (automated) test reports. The point is not to necessarily provide a strict regime on testing, only good practices, but also provide insight to the users so they can properly assess the risk of using the subsystem. This is what is typically asked for in quality audits by big companies.

 

Thanks.

 

Svein Aga

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: 17. september 2018 18:06
To: testing-wg@...
Subject: [testing-wg] initial mail for zephyr testing plan

 

Hi All,

 

I initialize a test plan based on IEEE 829 template. https://github.com/zephyrproject-rtos/qm/pull/2.

 

I propose we can start at defining below things:

 

 

1.     Testing goals:

a)      Let it be known what we want to achieve with Zephyr master test plan.

2.     Testing Items

a)      Hardware platforms

b)      Software components

e.g. the zephyr project can be divided intro

1. kernel

2. boards drivers

       Each company shall have its own test policy on their drivers

3. subsystems

4. samples / acceptance tests

5. unit tests

6. system tests

 

3.     How the existing samples/acceptance tests, system tests and unit test meet our test requirements.

4.     Long term / short term quality strategy for test items, and standards.

a)      I think for short term, we shall ensure that all features have test coverage, then function coverage, then line/branch coverage(based on requirements)

Svein: There are some testing literature that refers to maturity levels. There is an option to think of a few levels and define milestones for each. This will show our current view on where we are and where to go. Of course, the milestones can be redefined as progressing.

b)      We shall have user cases definition for subsystem cases, and this need help from TSC.

Svein: Could this be same as how to get test requirements for acceptance tests for a subsystem?

5.     V-model and test level requirements

a)      Define a Zephyr v-model (maybe as simple as unit, system and acceptance)

b)      Zephyr Master test plan defines the general test requirements within the v-model

c)      Subsystem defines the subsystem test requirements within the v-model

6.     Issue reporting and tracking

a)      Would current process enough, Reporting on github -> reproduce -> fix -> close.

b)      Current the issue severity is defined by code owner, would this be acceptable?

7.     Test environment definition and tools

a)      Qume simulation environment(already defined)

b)      Real board testing(each company shall provide this)

c)      Simulation such as renode(working group shall define it in the future)

d)      What tools we are recommend to use

                 i.          TestRail report scripts

                ii.          Renode, do we have free support from renode?

1.      I check the source code, I understand they use the cmsis-svd file to get the memory map of SOC, but how could renode know the SOC driver model? In other simulation they need to have a fast model to define the driver behavior.

 

Thanks for your comments!

 

Regards,

Hake


Links to TCF presentation

Perez-Gonzalez, Inaky <inaky.perez-gonzalez@...>
 

Hello All

Thanks for attending the presentation last Monday; here are the links:




Re: initial mail for zephyr testing plan

Hake Huang
 

Hi Svein,

 

Thanks a lot. Some feedback in lines leading with [Hake]

 

Regards,

Hake

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Aga, Svein
Sent: Friday, September 28, 2018 4:46 AM
To: testing-wg@...
Subject: Re: [testing-wg] initial mail for zephyr testing plan

 

Hi,

 

I’ve proposed some changes/additions below in blue + some comments and questions.  

 

A more general comment is that the Zephyr project can define a top level master test plan that is common for everyone contributing. Each subsystems and drivers (and the like) can have their own test plan, specifying the additions and deviations from the Zephyr master test plan. It could then be possible to have traceability from top level and down to subsystem testing. Each of the subsystems can define their own scope within a v-model with proper (automated) test reports. The point is not to necessarily provide a strict regime on testing, only good practices, but also provide insight to the users so they can properly assess the risk of using the subsystem. This is what is typically asked for in quality audits by big companies.

 

Thanks.

 

Svein Aga

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: 17. september 2018 18:06
To: testing-wg@...
Subject: [testing-wg] initial mail for zephyr testing plan

 

Hi All,

 

I initialize a test plan based on IEEE 829 template. https://github.com/zephyrproject-rtos/qm/pull/2.

 

I propose we can start at defining below things:

 

 

1.     Testing goals:

a)      Let it be known what we want to achieve with Zephyr master test plan.

[Hake] usually the goals come from requirements, so I prefer to have a requirement document in community to set as goals.

2.     Testing Items

a)      Hardware platforms

b)      Software components

e.g. the zephyr project can be divided intro

1. kernel

2. boards drivers

       Each company shall have its own test policy on their drivers

3. subsystems

4. samples / acceptance tests

[Hake] acceptance tests is test type, not test item. Test item is something we delivery.

5. unit tests

6. system tests

[Hake] The system tests in zephyr delivery are samples. Do we need to have an system tests folder and adding system tests?

 

3.     How the existing samples/acceptance tests, system tests and unit test meet our test requirements.

[Hake] in zephyr delivery does not have acceptance tests, system tests. Only samples, do we need category them?

4.     Long term / short term quality strategy for test items, and standards.

a)      I think for short term, we shall ensure that all features have test coverage, then function coverage, then line/branch coverage(based on requirements)

Svein: There are some testing literature that refers to maturity levels. There is an option to think of a few levels and define milestones for each. This will show our current view on where we are and where to go. Of course, the milestones can be redefined as progressing.

[Hake] can you share these literatures? We may apply the maturity level, once accepted by all

b)      We shall have user cases definition for subsystem cases, and this need help from TSC.

Svein: Could this be same as how to get test requirements for acceptance tests for a subsystem?

[Hake] Yes

5.     V-model and test level requirements

[Hake] V-model is a sw process, I think we need zephyr TSC to apply this model first, currently they are using scrum process.

a)      Define a Zephyr v-model (maybe as simple as unit, system and acceptance)

b)      Zephyr Master test plan defines the general test requirements within the v-model

c)      Subsystem defines the subsystem test requirements within the v-model

6.     Issue reporting and tracking

a)      Would current process enough, Reporting on github -> reproduce -> fix -> close.

b)      Current the issue severity is defined by code owner, would this be acceptable?

7.     Test environment definition and tools

a)      Qume simulation environment(already defined)

b)      Real board testing(each company shall provide this)

c)      Simulation such as renode(working group shall define it in the future)

d)      What tools we are recommend to use

                 i.          TestRail report scripts

                ii.          Renode, do we have free support from renode?

1.      I check the source code, I understand they use the cmsis-svd file to get the memory map of SOC, but how could renode know the SOC driver model? In other simulation they need to have a fast model to define the driver behavior.

 

Thanks for your comments!

 

Regards,

Hake


Minutes, Nov. 5th

Nashif, Anas
 

 

Minutes - Nov 5th

Monday, 5 November 2018

 

 

Attendees: Anas, Hake, Inaky, Svein, Andrei

Opens

                   • Hake: Test plan discussion, where should it happen? Continue offline.

                   • Hake: PR Status? Anas reviewed with comments

Agenda

Anas: submitted scripts to upload junit results from TCF

Inaky: need to runs all tests on the same commit

Svein: AutoPTS is open-source now?.

Andrei: AutoPTS is automated tool for testing Bluetooth. It was only available SIG members. It will be open-source in few weeks.

Svein: Is AutoPTS an automation framework? Andrei: Yes.

Andrei: PTS = Profile Tuning Suite from SIG.

Svein: To respond to Test plan thread above.

 

Action Items

 Inaky: To demonstrate usage on the command line.

Andrei: To present AutoPTS

Svein: To respond to Test plan thread


Re: initial mail for zephyr testing plan

Aga, Svein
 

Hi Hake,

 

Comments inline [Svein].

 

 

Br

Svein Aga

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: 15. oktober 2018 18:23
To: testing-wg@...
Subject: Re: [testing-wg] initial mail for zephyr testing plan

 

Hi Svein,

 

Thanks a lot. Some feedback in lines leading with [Hake]

 

Regards,

Hake

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Aga, Svein
Sent: Friday, September 28, 2018 4:46 AM
To: testing-wg@...
Subject: Re: [testing-wg] initial mail for zephyr testing plan

 

Hi,

 

I’ve proposed some changes/additions below in blue + some comments and questions.  

 

A more general comment is that the Zephyr project can define a top level master test plan that is common for everyone contributing. Each subsystems and drivers (and the like) can have their own test plan, specifying the additions and deviations from the Zephyr master test plan. It could then be possible to have traceability from top level and down to subsystem testing. Each of the subsystems can define their own scope within a v-model with proper (automated) test reports. The point is not to necessarily provide a strict regime on testing, only good practices, but also provide insight to the users so they can properly assess the risk of using the subsystem. This is what is typically asked for in quality audits by big companies.

 

Thanks.

 

Svein Aga

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: 17. september 2018 18:06
To: testing-wg@...
Subject: [testing-wg] initial mail for zephyr testing plan

 

Hi All,

 

I initialize a test plan based on IEEE 829 template. https://github.com/zephyrproject-rtos/qm/pull/2.

 

I propose we can start at defining below things:

 

 

1.     Testing goals:

a)      Let it be known what we want to achieve with Zephyr master test plan.

[Hake] usually the goals come from requirements, so I prefer to have a requirement document in community to set as goals.

2.     Testing Items

a)      Hardware platforms

b)      Software components

e.g. the zephyr project can be divided intro

1. kernel

2. boards drivers

       Each company shall have its own test policy on their drivers

3. subsystems

4. samples / acceptance tests

[Hake] acceptance tests is test type, not test item. Test item is something we delivery.

[Svein]: yes, but you can think of the samples as acceptance tests. If the samples does not work, the test item (or the test) is broken. If the samples can provide a PASS/FAIL status they could be treated as acceptance tests and be part of the CI.

5. unit tests

6. system tests

[Hake] The system tests in zephyr delivery are samples. Do we need to have an system tests folder and adding system tests?

[Svein]: The samples are not really system tests. It’s just a sample on how to use the subsystem, although important to have (see above for acceptance tests). System tests do more than acceptance tests, ie more thorough, like functional valid/invalid, non-functional (performance/throughout, robustness, stress), power measurements, backwards compatibility, conformance, …

 

3.     How the existing samples/acceptance tests, system tests and unit test meet our test requirements.

[Hake] in zephyr delivery does not have acceptance tests, system tests. Only samples, do we need category them?

[Svein]: See above comments on system and acceptance tests. It is useful to categorize them for the long term. You quickly know what kind of test has failed and possibly the impact. After some time with statistics you can extract reports to analyse what kind of failures are more frequent than others and quantify them. You can then also consider means to reduce them. See next comment on test process improvements.

4.     Long term / short term quality strategy for test items, and standards.

a)      I think for short term, we shall ensure that all features have test coverage, then function coverage, then line/branch coverage(based on requirements)

Svein: There are some testing literature that refers to maturity levels. There is an option to think of a few levels and define milestones for each. This will show our current view on where we are and where to go. Of course, the milestones can be redefined as progressing.

[Hake] can you share these literatures? We may apply the maturity level, once accepted by all

[Svein]: A few references:

-          Test Process Improvements (TPI) next - https://improvement.polteq.com/en/tpi-next/

-          https://www.tmmi.org/tmmi-model/ Also see the figure. At which level do you think Zephyr is? IMO at level 1 or 2.

 

b)      We shall have user cases definition for subsystem cases, and this need help from TSC.

It might not be that Zephyr follow one model to full extent, but some parts could still be relevant to follow.

 

Svein: Could this be same as how to get test requirements for acceptance tests for a subsystem?

[Hake] Yes

5.     V-model and test level requirements

[Hake] V-model is a sw process, I think we need zephyr TSC to apply this model first, currently they are using scrum process.

[Svein]: It’s good to involve TSC for the development process, but regardless you can apply the v-model purely for testing. Most typically the left-branch of the v-model will still happen (higher level requirements, system requirements/design, unit/component requirement/design), however with different names. Defining a v-model helps you to know what kind of tests are required for the different development activities.

a)      Define a Zephyr v-model (maybe as simple as unit, system and acceptance)

b)      Zephyr Master test plan defines the general test requirements within the v-model

c)      Subsystem defines the subsystem test requirements within the v-model

6.     Issue reporting and tracking

a)      Would current process enough, Reporting on github -> reproduce -> fix -> close.

b)      Current the issue severity is defined by code owner, would this be acceptable?

7.     Test environment definition and tools

a)      Qume simulation environment(already defined)

b)      Real board testing(each company shall provide this)

c)      Simulation such as renode(working group shall define it in the future)

d)      What tools we are recommend to use

                 i.          TestRail report scripts

                ii.          Renode, do we have free support from renode?

1.      I check the source code, I understand they use the cmsis-svd file to get the memory map of SOC, but how could renode know the SOC driver model? In other simulation they need to have a fast model to define the driver behavior.

 

Thanks for your comments!

 

Regards,

Hake


Re: initial mail for zephyr testing plan

Hake Huang
 

Hi Svein,

 

Thanks for the reply. Basically I agree with you. But there are two major item, I want to raise here:

 

1.     Will testing working group own all the samples and tests? If not how could we category them? Currently we just take the available cases as is.

2.     If we define a new test suite, I have concern whether we have enough resources to develop them. Pre my experiences, a dedicated and efficient team is required, can we afford?

3.     I am not quite familiar with Zephyr development process, @Nashif, Anas can you help to explain the current process? As Svein proposed, there are some equivalents for higher level requirements, system requirements/design, unit/component requirement/design in zephyr development, but I just can’t find them, I know some of them a list in the issues on git hub, but this is typical scrum process. IMHO, the difference of scrum process and V-model is the timing, in v-modelling, everything is defined  well before execution, but in scrum those are likely to change always. So the questions is whether we have enough resource to closely follow the developers.

 

Some more feedbacks in line leading with [Hake-2].

 

Regards,

hake

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Aga, Svein
Sent: Tuesday, November 6, 2018 6:55 PM
To: testing-wg@...
Subject: Re: [testing-wg] initial mail for zephyr testing plan

 

Hi Hake,

 

Comments inline [Svein].

 

 

Br

Svein Aga

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: 15. oktober 2018 18:23
To: testing-wg@...
Subject: Re: [testing-wg] initial mail for zephyr testing plan

 

Hi Svein,

 

Thanks a lot. Some feedback in lines leading with [Hake]

 

Regards,

Hake

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Aga, Svein
Sent: Friday, September 28, 2018 4:46 AM
To: testing-wg@...
Subject: Re: [testing-wg] initial mail for zephyr testing plan

 

Hi,

 

I’ve proposed some changes/additions below in blue + some comments and questions.  

 

A more general comment is that the Zephyr project can define a top level master test plan that is common for everyone contributing. Each subsystems and drivers (and the like) can have their own test plan, specifying the additions and deviations from the Zephyr master test plan. It could then be possible to have traceability from top level and down to subsystem testing. Each of the subsystems can define their own scope within a v-model with proper (automated) test reports. The point is not to necessarily provide a strict regime on testing, only good practices, but also provide insight to the users so they can properly assess the risk of using the subsystem. This is what is typically asked for in quality audits by big companies.

 

Thanks.

 

Svein Aga

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: 17. september 2018 18:06
To: testing-wg@...
Subject: [testing-wg] initial mail for zephyr testing plan

 

Hi All,

 

I initialize a test plan based on IEEE 829 template. https://github.com/zephyrproject-rtos/qm/pull/2.

 

I propose we can start at defining below things:

 

 

1.     Testing goals:

a)      Let it be known what we want to achieve with Zephyr master test plan.

[Hake] usually the goals come from requirements, so I prefer to have a requirement document in community to set as goals.

2.     Testing Items

a)      Hardware platforms

b)      Software components

e.g. the zephyr project can be divided intro

1. kernel

2. boards drivers

       Each company shall have its own test policy on their drivers

3. subsystems

4. samples / acceptance tests

[Hake] acceptance tests is test type, not test item. Test item is something we delivery.

[Svein]: yes, but you can think of the samples as acceptance tests. If the samples does not work, the test item (or the test) is broken. If the samples can provide a PASS/FAIL status they could be treated as acceptance tests and be part of the CI.

[Hake-2] agree. So samples will be category as acceptances and shall be pass in every release. If not, we need blocking the release for given board or not?

5. unit tests

6. system tests

[Hake] The system tests in zephyr delivery are samples. Do we need to have an system tests folder and adding system tests?

[Svein]: The samples are not really system tests. It’s just a sample on how to use the subsystem, although important to have (see above for acceptance tests). System tests do more than acceptance tests, ie more thorough, like functional valid/invalid, non-functional (performance/throughout, robustness, stress), power measurements, backwards compatibility, conformance, …

[Hake-2] Agree. So who will create those system test? The testing working group or community?

 

3.     How the existing samples/acceptance tests, system tests and unit test meet our test requirements.

[Hake] in zephyr delivery does not have acceptance tests, system tests. Only samples, do we need category them?

[Svein]: See above comments on system and acceptance tests. It is useful to categorize them for the long term. You quickly know what kind of test has failed and possibly the impact. After some time with statistics you can extract reports to analyse what kind of failures are more frequent than others and quantify them. You can then also consider means to reduce them. See next comment on test process improvements.

[Hake-2] in this case, testing working group will take care of the samples? Do we have enough resources?

4.     Long term / short term quality strategy for test items, and standards.

a)      I think for short term, we shall ensure that all features have test coverage, then function coverage, then line/branch coverage(based on requirements)

Svein: There are some testing literature that refers to maturity levels. There is an option to think of a few levels and define milestones for each. This will show our current view on where we are and where to go. Of course, the milestones can be redefined as progressing.

[Hake] can you share these literatures? We may apply the maturity level, once accepted by all

[Svein]: A few references:

-          Test Process Improvements (TPI) next - https://improvement.polteq.com/en/tpi-next/

-          https://www.tmmi.org/tmmi-model/ Also see the figure. At which level do you think Zephyr is? IMO at level 1 or 2.

[Hake-2] Thanks, I agree with your judgement. Pre my understanding, if we targeting to this model, huge resource will be required. And low level does not mean low quality.

 

b)      We shall have user cases definition for subsystem cases, and this need help from TSC.

It might not be that Zephyr follow one model to full extent, but some parts could still be relevant to follow.

 

Svein: Could this be same as how to get test requirements for acceptance tests for a subsystem?

[Hake] Yes

5.     V-model and test level requirements

[Hake] V-model is a sw process, I think we need zephyr TSC to apply this model first, currently they are using scrum process.

[Svein]: It’s good to involve TSC for the development process, but regardless you can apply the v-model purely for testing. Most typically the left-branch of the v-model will still happen (higher level requirements, system requirements/design, unit/component requirement/design), however with different names. Defining a v-model helps you to know what kind of tests are required for the different development activities.

[Hake-2] as I replied in the beginning of mail, it is a timing issue, could we have enough resources and commitment to follow the developers.

a)      Define a Zephyr v-model (maybe as simple as unit, system and acceptance)

b)      Zephyr Master test plan defines the general test requirements within the v-model

c)      Subsystem defines the subsystem test requirements within the v-model

6.     Issue reporting and tracking

a)      Would current process enough, Reporting on github -> reproduce -> fix -> close.

b)      Current the issue severity is defined by code owner, would this be acceptable?

7.     Test environment definition and tools

a)      Qume simulation environment(already defined)

b)      Real board testing(each company shall provide this)

c)      Simulation such as renode(working group shall define it in the future)

d)      What tools we are recommend to use

                 i.          TestRail report scripts

                ii.          Renode, do we have free support from renode?

1.      I check the source code, I understand they use the cmsis-svd file to get the memory map of SOC, but how could renode know the SOC driver model? In other simulation they need to have a fast model to define the driver behavior.

 

Thanks for your comments!

 

Regards,

Hake


TCF complex examples with multiple targets

Perez-Gonzalez, Inaky <inaky.perez-gonzalez@...>
 

Hello

As promised on the call on 11/19, here are some examples for complex
testcases with TCF.

As an example, this is testing the USB CDC sample; references for this
in https://intel.github.io/tcf/doc/02-guides.html#creating-testcases,
https://intel.github.io/tcf/doc/02-guides.html#testcase-training and
https://intel.github.io/tcf/doc/07-Architecture.html#the-testcase-finder-and-runner.

Now this is a simplified version of a more complext test case, for
clarity, but it should work to illustrate the point. The most complex
thing here is the HW that helps automating the plugging/unpluggin
::

import os
import time
import re
import tcfl.tc

# We want an interconnect of the ones that connect a USB capable
# device to a USB host; this represents a setup in which the zephyr
# target has been connected as a 'TCF' thing to the linux target
# (that's why they declare they are part of a usb__host__device
# interconnect. We do this with a USBRLY08B relay board
# (https://intel.github.io/tcf/doc/09-api.html?highlight=usbrly08b#ttbl.usbrly08b.plugger,
# see the examples section)
#
# linux target is a Linux device that is part of the interconnect
#
# Zephyr target is a Zephyr device that can run the cdc sample, only
# on boards of type arduino_101 or quark_se_c1000_devboard
#
@tcfl.tc.interconnect('type == "usb__host__device"')
@tcfl.tc.target('linux', name = 'linux')
@tcfl.tc.target(
name = "zephyr",
spec = """zephyr_board in [
'arduino_101', 'quark_se_c1000_devboard'
]""",
app_zephyr = os.path.join(os.environ['ZEPHYR_BASE'],
"samples", "subsys", "usb", "cdc_acm"))
class _test(tcfl.tc.tc_c):

@staticmethod
@tcfl.tc.serially()
def build_20_zephyr(zephyr):
# Set the zephyr configuration before building the App; the
# app is built with a method build_50(zephyr) introduced by
# the app_zephyr plugin above -- see
# https://intel.github.io/tcf/doc/02-guides.html?highlight=overriden#cover-more-bases-on-the-zephyr-echo-server-client
zephyr.zephyr.config_file_write("usb", """
CONFIG_USB_DEVICE_VID=0xDEAD
CONFIG_USB_DEVICE_PID=0xBEEF
""")

def start_50_zephyr(self):
pass

@staticmethod
def start(linux, zephyr):
# start up the test by powering everything off and ensuring
# the zephyr target is unplugged from the linux one; then
# power on the linux board and wait for the shell to come up.
zephyr.power.off()
linux.power.off()
linux.thing_unplug(zephyr.id)

linux.power.on()
linux.shell.up()

def eval_01(self, linux, zephyr):
# Use the methods created by app_zephyr to power up the
# zephyr board (as it knows the best way to do it).
#
# Then plug the zephyr board to the linux host and wait for
# it to enumerate (we just give it 3secs and check
# /dev/ttyACM0 exists)

self.overriden_start_50_zephyr(zephyr)
linux.thing_plug(zephyr.id)
time.sleep(3)
linux.shell.run('udevadm info /dev/ttyACM0',
'ID_SERIAL=ZEPHYR_USB-DEV')


# Now check the Zephyr side, wait till we get the cdc_acm>
# prompt on console, have the Linux side read so it sets DTR
# and start sending stuff back and forth

zephyr.expect("cdc_acm>")
linux.send("cat /dev/ttyACM0 &")
time.sleep(3)
zephyr.send("test_uart_fifo_read_write")
zephyr.expect("DTR set, start test")
linux.send('echo "Send Helloworld to cdc_acm port" > /dev/ttyACM0')
zephyr.expect("Send Helloworld to cdc_acm port")
linux.send('echo "quit" > /dev/ttyACM0')

In our setup, there is a Linux NUC called nuc-02 which is a member of
an interconnect called usb__nuc-02__a101-05 -- that interconnect
basically groups together the nuc-02 and a target called a101-05.

$ tcf list -vv nuc-02
https://srrsotc03.iind.intel.com:5000/ttb-v1/targets/nuc-02
...
interconnects: {
...,
u'usb__nuc-02__a101-05': {},
},
...
things: [u'a101-05']
..
type: nuc-linux-fedora-x86_64

The nuc-02 declares it is part of the usb__nuc-02__a101-05
interconnect, and that there is a thing we can connect to it, the
target called a101-05::

$ tcf list -avv a101-05
https://srrsotc03.iind.intel.com:5000/ttb-v1/targets/a101-05
...
interconnects: {
u'usb__nuc-02__a101-05': {}
}
...

note how both the nuc-02 and a101-05 declare themselves to be a part
of the usb__nuc-02__a101-05 network, which is::

$ tcf list -vav usb__nuc-02__a101-05
https://srrsotc03.iind.intel.com:5000/ttb-v1/targets/usb__nuc-02__a101-05
id: usb__nuc-02__a101-05
...
type: usb__host__device

note how the type is usb__host__device, which is what we match against
in the testcase.


A detailed explanation of a networking example is available in the
documentation in
https://intel.github.io/tcf/doc/02-guides.html?highlight=overriden#two-interconnected-targets,
so I won't go in detail on it -- but it is basically the same concept,
but with IP networking.

(I also must admit I have not tried those examples on a while, so they
might have bitrotten a wee bit -- my apologies if that is the case).


this is a proposal for test issues tracking process

Hake Huang
 

Hi All,

 

This mail is to propose a process for efficient test working group to control the issues that are found in test runs.

 

1.     We need create a periodic summary report e.g.

We can choose the test runs that we need to track, my suggestion would be to use a milestone to gather all required test runs, so that we can also track community test issues, once they know which milestone we are in.

https://zephyrproject.testrail.io/index.php?/reports/view/39

2.     Based on this summary report, we will see a list of defects that are open.

3.     We need define a periodic bug scrum meeting say by monthly? To assign those open issues to individual person, and report to jira tickets, if possible.

4.     We need post the active milestone to community, and enable them to submit test run issues so that we can also track community issues? Any better comments?

 

 

Regards,

Hake


Re: this is a proposal for test issues tracking process

Perez-Gonzalez, Inaky <inaky.perez-gonzalez@...>
 

One thing to bear in mind, from what we have observed running CI on TCF with Zephyr is that there are issues (failures, errors) that will creep randomly and then not appear again for a while.

Thus running once will not catch them -- so I'd advocate for a way to do (1) that catches multiple test runs, not necessarily on the same commit ID.

On 12/2/18 11:22 PM, Hake Huang wrote:

Hi All,

 

This mail is to propose a process for efficient test working group to control the issues that are found in test runs.

 

1.     We need create a periodic summary report e.g.

We can choose the test runs that we need to track, my suggestion would be to use a milestone to gather all required test runs, so that we can also track community test issues, once they know which milestone we are in.

https://zephyrproject.testrail.io/index.php?/reports/view/39

2.     Based on this summary report, we will see a list of defects that are open.

3.     We need define a periodic bug scrum meeting say by monthly? To assign those open issues to individual person, and report to jira tickets, if possible.

4.     We need post the active milestone to community, and enable them to submit test run issues so that we can also track community issues? Any better comments?

 

 

Regards,

Hake


meeting today

Nashif, Anas
 

Sorry folks, wont be able to attend the meeting due to a last minute personal conflict.

 

anas

 


Zephyr: Testing WG weekly call - Mon, 02/04/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 4 February 2019
8:00am to 9:00am
(GMT-08:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP


Zephyr: Testing WG weekly call - Mon, 02/11/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 11 February 2019
8:00am to 9:00am
(GMT-08:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP


no meeting today

Nashif, Anas
 

Due to holidays, no meeting today.

Also note that next week we will be at Embedded World, so no meeting next week as well.

 

 

Regards,

Anas

 


Zephyr: Testing WG weekly call - Mon, 02/18/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 18 February 2019
8:00am to 9:00am
(GMT-08:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP


Zephyr: Testing WG weekly call - Mon, 02/25/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 25 February 2019
8:00am to 9:00am
(GMT-08:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP


Zephyr: Testing WG weekly call - Mon, 03/04/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 4 March 2019
8:00am to 9:00am
(GMT-08:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP


Re: Updated invitation with note: Zephyr: Testing WG weekly call @ Weekly from 8am to 9am on Monday (PDT) (hake.huang@nxp.com)

Nashif, Anas
 

Hi,

And I will have a dentist appointment at this time, so lets cancel.

 

Anas

 

From: Hake Huang <hake.huang@...>
Date: Monday, 11 March 2019 at 07:21
To: Zephyr Public Meetings <linuxfoundation.org_h64khdovjasejob20q2aam7o9s@...>
Cc: Anas Nashif <anas.nashif@...>
Subject: RE: Updated invitation with note: Zephyr: Testing WG weekly call @ Weekly from 8am to 9am on Monday (PDT) (hake.huang@...)

 

I have a meeting conflict today, and I will join half hour later if possible.

 

Regards,

Hake

 

-----Original Appointment-----

From: Zephyr Public Meetings [mailto:linuxfoundation.org_h64khdovjasejob20q2aam7o9s@...]
Sent: Monday, August 13, 2018 9:51 PM
To: Zephyr Public Meetings; Hake Huang
Subject: Updated invitation with note: Zephyr: Testing WG weekly call @ Weekly from 8am to 9am on Monday (PDT) (hake.huang@...)
When: Monday, March 11, 2019 8:00 AM-9:00 AM America/Los_Angeles.
Where: https://zoom.us/j/679527144

 

This event has been changed with this note:
"extending into 2019"

Zephyr: Testing WG weekly call

When

Changed: Weekly from 8am to 9am on Monday Pacific Time - Los Angeles

Where

Calendar

Who

(Guest list has been hidden at organizer's request)


Live agenda/meeting minutes: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing

-----
Hi there,

Zephyr Working Group is inviting you to a scheduled Zoom meeting.

Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144#  or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656  or +1 669 900 6833  or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

Going (hake.huang@...)?   All events in this series:   Yes - Maybe - No    more options »

Invitation from Google Calendar

You are receiving this courtesy email at the account hake.huang@... because you are an attendee of this event.

To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar.

Forwarding this invitation could allow any recipient to modify your RSVP response. Learn More.

 


Zephyr: Testing WG weekly call - Mon, 03/11/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 11 March 2019
8:00am to 9:00am
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

1 - 20 of 361