testing-wg@lists.zephyrproject.org

Cancelled Event: Zephyr: Testing WG weekly call - Monday, 22 April 2019 #cal-cancelled

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#31

Cancelled: Zephyr: Testing WG weekly call

This event has been cancelled.

When:
Monday, 22 April 2019
8:00am to 9:00am
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

Calendar

Cancelled Event: Zephyr: Testing WG weekly call - Monday, 15 April 2019 #cal-cancelled

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#30

Cancelled: Zephyr: Testing WG weekly call

This event has been cancelled.

When:
Monday, 15 April 2019
8:00am to 9:00am
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

Calendar

Re: some idea on providing test services to community

Nashif, Anas

#29

Hi,

No filter to find those, the whitelisting in the tests is happening for various reasons, sometimes just to limit level of execution in CI to a limited set of boards and architectures. It definitely needs to be optimized. Any whitelisting that happens now is required to have a justification the commit message and a comment in the yaml file.

Anas

Show quoted text

From: Hake Huang [mailto:hake.huang@...]
Sent: Monday, April 8, 2019 11:12 PM
To: Nashif, Anas <anas.nashif@...>; testing-wg@...
Cc: Hake Huang <hake.huang@...>
Subject: RE: [testing-wg] some idea on providing test services to community

Hi Anas,

In yesterday’s meeting, you mentioned that the developer will provide justify on the whitelist selection. Do those all listed in the individual pull request? Is there a filter to retrieve them?

Regards,

Hake

Re: some idea on providing test services to community

Hake Huang

#28

Hi Anas,

In yesterday’s meeting, you mentioned that the developer will provide justify on the whitelist selection. Do those all listed in the individual pull request? Is there a filter to retrieve them?

Regards,

Hake

Zephyr: Testing WG weekly call - Mon, 04/08/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#27

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 8 April 2019
8:00am to 9:00am
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

Zephyr: Testing WG weekly call - Mon, 04/01/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#26

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 1 April 2019
8:00am to 9:00am
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

Re: some idea on providing test services to community

Hake Huang

#25

Hi Anas,

Thanks a lot for the reviewing.

I create our own method is just because our board farm limitation. The board farm run machine can’t support Linux host. And Yes, you are right using the sanity check is a good option for most community users. So my Jenkins solution is not a must(#3). What I propose is the whole process, not the way to do automation.

Besides, I want to learn how the filter list is thus created. And so I start to analyze them one by one. To me the list is just a tracking list for features, as in NXP we have not define a clear supporting strategy for all boards in general.

Btw, I am using the zephyr docker which is at https://github.com/zephyrproject-rtos/ci-dockerfiles.git this CI works fine for me.

Regards,

Hake

Show quoted text

From: Nashif, Anas [mailto:anas.nashif@...]
Sent: Thursday, March 28, 2019 3:19 AM
To: Hake Huang <hake.huang@...>; testing-wg@...
Subject: RE: some idea on providing test services to community

Hake,

I looked at the code you referenced below. I am wondering why you do most of this instead of relying on the filter capabilities already available in sanitycheck. Right now you are able to run:

sanitycheck --device-testing --device-serial /dev/ttyACM0 -p frdm_k64f

and get all tests (and samples) run on the device and produce a junit file with the test results without having to maintain your own whitelist/blacklist or any device specific tests. This also will work on tests that are not part of Zephyr. It is available for all boards and works nicely. The results can be then uploaded to testrail for specific PRs.

This can of course all be done from Jenkins, but most developers or testers will be able to do that on their own and directly.

To make sure wall are using the same environment, we can use a docker image available already publicly with the SDK pre-installed.

Anas

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: Monday, March 25, 2019 11:13 PM
To: testing-wg@...
Subject: [testing-wg] some idea on providing test services to community

Hi All,

Pre last night’s discussion, here is the briefing of my idea.

1. We can announce that we accept pull request testing.(binary test can be later support)

a) A pull request ID shall be provide.

b) A configure file shall include in the pull request, which defines:

1. platform to test, default all.

2. Test suites to test, default all.

2. once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteer’s testing branch. This can be done via shippable.

3. The selected volunteers can start testing the pull request by his own condition.

4. Volunteers shall have a mechanism to upload their test report to testrails, with a given report name.

A comments will be added automatically to the pull request, once the report is uploaded.

I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files.

https://jenkins.io/doc/book/blueocean/

and NXP board pipeline files are defined here

https://github.com/hakehuang/zephyr_app_scm

I am working on the #4 for a unified junit report based on console log only.

Regards,

Hake

Re: some idea on providing test services to community

Nashif, Anas

#24

Hake,

I looked at the code you referenced below. I am wondering why you do most of this instead of relying on the filter capabilities already available in sanitycheck. Right now you are able to run:

sanitycheck --device-testing --device-serial /dev/ttyACM0 -p frdm_k64f

and get all tests (and samples) run on the device and produce a junit file with the test results without having to maintain your own whitelist/blacklist or any device specific tests. This also will work on tests that are not part of Zephyr. It is available for all boards and works nicely. The results can be then uploaded to testrail for specific PRs.

This can of course all be done from Jenkins, but most developers or testers will be able to do that on their own and directly.

To make sure wall are using the same environment, we can use a docker image available already publicly with the SDK pre-installed.

Anas

Show quoted text

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: Monday, March 25, 2019 11:13 PM
To: testing-wg@...
Subject: [testing-wg] some idea on providing test services to community

Hi All,

Pre last night’s discussion, here is the briefing of my idea.

1. We can announce that we accept pull request testing.(binary test can be later support)

a) A pull request ID shall be provide.

b) A configure file shall include in the pull request, which defines:

1. platform to test, default all.

2. Test suites to test, default all.

2. once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteer’s testing branch. This can be done via shippable.

3. The selected volunteers can start testing the pull request by his own condition.

4. Volunteers shall have a mechanism to upload their test report to testrails, with a given report name.

A comments will be added automatically to the pull request, once the report is uploaded.

I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files.

https://jenkins.io/doc/book/blueocean/

and NXP board pipeline files are defined here

https://github.com/hakehuang/zephyr_app_scm

I am working on the #4 for a unified junit report based on console log only.

Regards,

Hake

some idea on providing test services to community

Hake Huang

#23

Hi All,

Pre last night’s discussion, here is the briefing of my idea.

1. We can announce that we accept pull request testing.(binary test can be later support)

a) A pull request ID shall be provide.

b) A configure file shall include in the pull request, which defines:

1. platform to test, default all.

2. Test suites to test, default all.

2. once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteer’s testing branch. This can be done via shippable.

3. The selected volunteers can start testing the pull request by his own condition.

4. Volunteers shall have a mechanism to upload their test report to testrails, with a given report name.

A comments will be added automatically to the pull request, once the report is uploaded.

I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files.

https://jenkins.io/doc/book/blueocean/

and NXP board pipeline files are defined here

https://github.com/hakehuang/zephyr_app_scm

I am working on the #4 for a unified junit report based on console log only.

Regards,

Hake

Zephyr: Testing WG weekly call - Mon, 03/25/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#22

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 25 March 2019
8:00am to 9:00am
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

Zephyr: Testing WG weekly call - Mon, 03/18/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#21

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 18 March 2019
8:00am to 9:00am
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

Zephyr: Testing WG weekly call - Mon, 03/11/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#20

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 11 March 2019
8:00am to 9:00am
(GMT-07:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

Re: Updated invitation with note: Zephyr: Testing WG weekly call @ Weekly from 8am to 9am on Monday (PDT) (hake.huang@nxp.com)

Nashif, Anas

#19

Hi,

And I will have a dentist appointment at this time, so lets cancel.

Anas

From: Hake Huang <hake.huang@...>
Date: Monday, 11 March 2019 at 07:21
To: Zephyr Public Meetings <linuxfoundation.org_h64khdovjasejob20q2aam7o9s@...>
Cc: Anas Nashif <anas.nashif@...>
Subject: RE: Updated invitation with note: Zephyr: Testing WG weekly call @ Weekly from 8am to 9am on Monday (PDT) (hake.huang@...)

I have a meeting conflict today, and I will join half hour later if possible.

Regards,

Hake

-----Original Appointment-----

Show quoted text

From: Zephyr Public Meetings [mailto:linuxfoundation.org_h64khdovjasejob20q2aam7o9s@...]
Sent: Monday, August 13, 2018 9:51 PM
To: Zephyr Public Meetings; Hake Huang
Subject: Updated invitation with note: Zephyr: Testing WG weekly call @ Weekly from 8am to 9am on Monday (PDT) (hake.huang@...)
When: Monday, March 11, 2019 8:00 AM-9:00 AM America/Los_Angeles.
Where: https://zoom.us/j/679527144

This event has been changed with this note:
"extending into 2019"

more details »

Zephyr: Testing WG weekly call

When	Changed: Weekly from 8am to 9am on Monday Pacific Time - Los Angeles
Where	https://zoom.us/j/679527144 (map)
Calendar	hake.huang@...
Who	(Guest list has been hidden at organizer's request)

Live agenda/meeting minutes: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing

-----
Hi there,

Zephyr Working Group is inviting you to a scheduled Zoom meeting.

Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

Going (hake.huang@...)? All events in this series: Yes - Maybe - No more options »

Invitation from Google Calendar

You are receiving this courtesy email at the account hake.huang@... because you are an attendee of this event.

To stop receiving future updates for this event, decline this event. Alternatively you can sign up for a Google account at https://www.google.com/calendar/ and control your notification settings for your entire calendar.

Forwarding this invitation could allow any recipient to modify your RSVP response. Learn More.

Zephyr: Testing WG weekly call - Mon, 03/04/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#18

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 4 March 2019
8:00am to 9:00am
(GMT-08:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

Zephyr: Testing WG weekly call - Mon, 02/25/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#17

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 25 February 2019
8:00am to 9:00am
(GMT-08:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

Zephyr: Testing WG weekly call - Mon, 02/18/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#16

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 18 February 2019
8:00am to 9:00am
(GMT-08:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

no meeting today

Nashif, Anas

#15

Due to holidays, no meeting today.

Also note that next week we will be at Embedded World, so no meeting next week as well.

Regards,

Anas

Zephyr: Testing WG weekly call - Mon, 02/11/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#14

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 11 February 2019
8:00am to 9:00am
(GMT-08:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

Zephyr: Testing WG weekly call - Mon, 02/04/2019 8:00am-9:00am #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>

#13

Reminder:
Zephyr: Testing WG weekly call

When:
Monday, 4 February 2019
8:00am to 9:00am
(GMT-08:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer:
testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144#
Or Telephone:
    Dial(for higher quality, dial a number based on your current location):
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

An RSVP is requested. Click here to RSVP

meeting today

Nashif, Anas

#12

Sorry folks, wont be able to attend the meeting due to a last minute personal conflict.

anas

This event has been changed with this note: "extending into 2019"

Zephyr: Testing WG weekly call

This event has been changed with this note:
"extending into 2019"