Date   

Updated Event: Zephyr: Testing WG weekly call #cal-invite

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Zephyr: Testing WG weekly call

When:
Monday, 25 May 2020
1:00pm to 2:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
https://zoom.us/j/679527144

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


SC verdicts follow-up

Perkowski, Maciej
 

Dear Fellow Testers,

As a follow-up to the last meeting, I am sending you a proposal for verdicts used by sanitycheck to mark results. It is based on Hake’s proposal with some clarifications discussed during the meeting and our internal experience:

·       PASS - test was successful

·       FAIL - test assertion(s) failed

·       ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or some other error occurred during the execution

·       NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected

·       IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution

·       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. 

The idea of marking a test as UNSTABLE is more about an extra flag than a verdict. A test can be marked as unstable if it required reruns to pass or tends to fail occasionally. This can serve as a notification to a user that seeing failures in the given test is nothing unexpected and one should try rerunning the test.

I also think that WARNING should be an additional flag, not the verdict per se. The test can be marked as PASS but have an extra notification in WARNING that some warnings appeared (e.g.  produced when test cleanup failed to restore the system)

Best regards,

Maciej

 


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 05/18/2020 6:00am-7:00am, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 18 May 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Re: [EXT] [testing-wg] Upcoming Event: Zephyr: Testing WG weekly call - Mon, 05/18/2020 6:00am-7:00am, Please RSVP #cal-reminder

Hake Huang
 

Hi All,

 

Meeting agenda for this weekly meeting:

 

  1. Come up with ideas for test report.(5 minutes). round table discussion
  2. CI demo on 2.3.0-RC1 trail. (5 mintues) Anash
  3. Discussion on test result type definition. (20 minuts) Anash

https://elinux.org/Test_Result_Codes

proposal is as below:

·         TPASS Passed - (color: green) test was successful

·         TFAIL Failed - (color: red) test assertions failed

·         TSKIP Skipped - (color: yellow) Test was skipped because of missing pre-requisite or configuration

·         TWARN Warning - (color: magenta) test produced warnings - usually produced when test cleanup failed to restore the system

·         TBROK Broken - Broken is usually reported when test setup fails before the test even attempts to test the test assertions

·         TUNSTABLE – (color: cyan) (test was successfully executed but found failures.)

 

6. TestRail test result upload slow follow up. (15 minutes) Hao/Maksim

   

7Define a process to integrate community CI report initial discussion. (10 minutes) round table discussion

 

8. Try to replace Test Rail, think about how to replace it. Come up with ideas. (10 minutes) round table discussion

       Options:

  1. Open source test report with data mining support.

http://allure.qatools.ru/

       .2. Linaro test report

       https://validation.linaro.org/results/chart

 

Regards,

Hake


about bluetooth certification

Hake Huang
 

Hi Maciej,

 

As we discussed in this weekly meeting, thanks in advance for your help on below item:

 

1.     What tool does Nordic use to pass the certification? If it is not AutoPTS.

 

 

Regards,

Hake


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 05/11/2020 6:00am-7:00am, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 11 May 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Re: [EXT] [testing-wg] event: Zephyr: Testing WG weekly call - Monday, 11 May 2020

Hake Huang
 

Hi All,

 

This weekly meeting agenda:

 

1.     Progress on CI updating.

 

2.     Round table.

 

Regards,

Hake


Re: [EXT] [testing-wg] alternatives to junit

Nashif, Anas
 

It is closer to what testrail has than junit for sure, but we do not want to generate something that is testrail specific, it needs to be generic.

 

Anas

 

From: Hake Huang <hake.huang@...>
Date: Monday, 4 May 2020 at 09:20
To: Anas Nashif <anas.nashif@...>, "testing-wg@..." <testing-wg@...>
Subject: RE: [EXT] [testing-wg] alternatives to junit

 

Hi Anas,

 

This looks good, and I have one question, can test rails accommodate all those items? If not, how can we put them into test report generated by test rail?

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Nashif, Anas via lists.zephyrproject.org
Sent: 2020
54 20:23
To: testing-wg@...
Subject: [EXT] [testing-wg] alternatives to junit

 

Caution: EXT Email

Hi,

 

Junit format we use is very limited and has lots of constraints, I think it is time to consider something else for capturing test results, something that can contain more information than junit does, I was looking at different options, for example nunit is promising:

 

https://github.com/nunit/docs/wiki/Test-Result-XML-Format

 

 

Any other options? What do you think?

 

Anas


Testing WG Meeting Monday May 4 Cancelled

Masalski, Maksim
 

Hello, due to public holidays in China and key members have holidays, we decided to cancel that meeting today.

 

Maksim Masalski

Meeting host

 


hw test workflow updates

Nashif, Anas
 

Hi,

I now added an action in the test_results repo that process the output from daily sanitycheck run and post a report to amazon, this is just a proof of concept to show how we deal with results and process them in the action and then upload back to Amazon S3.

 

 

See

 

https://testing.zephyrproject.org/daily_tests/zephyr-v2.2.0-2229-g9b055ecf82/report/index.html

 

https://github.com/zephyrproject-rtos/test_results/blob/master/.github/workflows/process_results.yml

 

 

Hopefully this will help with other steps planned and upload to testrail.

 

Anas


Re: [EXT] [testing-wg] alternatives to junit

Hake Huang
 

Hi Anas,

 

This looks good, and I have one question, can test rails accommodate all those items? If not, how can we put them into test report generated by test rail?

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Nashif, Anas via lists.zephyrproject.org
Sent: 2020
54 20:23
To: testing-wg@...
Subject: [EXT] [testing-wg] alternatives to junit

 

Caution: EXT Email

Hi,

 

Junit format we use is very limited and has lots of constraints, I think it is time to consider something else for capturing test results, something that can contain more information than junit does, I was looking at different options, for example nunit is promising:

 

https://github.com/nunit/docs/wiki/Test-Result-XML-Format

 

 

Any other options? What do you think?

 

Anas


alternatives to junit

Nashif, Anas
 

Hi,

 

Junit format we use is very limited and has lots of constraints, I think it is time to consider something else for capturing test results, something that can contain more information than junit does, I was looking at different options, for example nunit is promising:

 

https://github.com/nunit/docs/wiki/Test-Result-XML-Format

 

 

Any other options? What do you think?

 

Anas


Cancelled Event: Zephyr: Testing WG weekly call - Monday, 4 May 2020 #cal-cancelled

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Cancelled: Zephyr: Testing WG weekly call

This event has been cancelled.

When:
Monday, 4 May 2020
6:00am to 7:00am
(UTC-07:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer: testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 04/27/2020 6:00am-7:00am, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 27 April 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Re: [EXT] [testing-wg] Upcoming Event: Zephyr: Testing WG weekly call - Mon, 04/27/2020 6:00am-7:00am, Please RSVP #cal-reminder

Hake Huang
 

Hi All,

 

The agenda for this weeks meeting:

 

1.     Discussion on CI progress

a)      Anash provides some solution for part1 / 2 implemented. The version file can be access at below

https://testing.zephyrproject.org/daily_tests/versions.json

b)      A trial result repo is https://github.com/hakehuang/zephyr_ci_repo

2.     Round table on Testing issues.

3.     Next weeks topic

 

 

Regards,

Hake

 

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
420 20:50
To: testing-wg@...
Subject: [EXT] [testing-wg] Upcoming Event: Zephyr: Testing WG weekly call - Mon, 04/20/2020 6:00am-7:00am, Please RSVP #cal-reminder

 

Caution: EXT Email

Reminder: Zephyr: Testing WG weekly call

When: Monday, 20 April 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: 
https://zoom.us/u/ed7Ng0QxH


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 04/20/2020 6:00am-7:00am, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 20 April 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Re: [EXT] [testing-wg] Upcoming Event: Zephyr: Testing WG weekly call - Mon, 04/20/2020 6:00am-7:00am, Please RSVP #cal-reminder

Hake Huang
 

Hi All,

 

The agenda for this weeks meeting:

 

1.     Discussion on CI TSC feedbacks.

a)      They expect to see the real implement soon.

2.     BLE test suite AutoPTS status update.

a)      Nordic will feedback and Andrei will host a meeting

3.     Round table on Testing issues.

4.     Next week’s topic

 

 

Regards,

Hake

 

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
413 20:50
To: testing-wg@...
Subject: [EXT] [testing-wg] Upcoming Event: Zephyr: Testing WG weekly call - Mon, 04/13/2020 6:00am-7:00am, Please RSVP #cal-reminder

 

Caution: EXT Email

Reminder: Zephyr: Testing WG weekly call

When: Monday, 13 April 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: 
https://zoom.us/u/ed7Ng0QxH


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 04/13/2020 6:00am-7:00am, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 13 April 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Cancelled Zephyr Testing WG Meeting on 6th of April

Maksim Masalski <maxxliferobot@...>
 

Hello, community members.
Zephyr Testing WG Meeting today on 6th of April will be cancelled due to the holidays in China, and key members can't attend it to discuss previous topics. See you next week as usual.

Best regards,
Maksim Masalski


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 04/06/2020 6:00am-7:00am, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 6 April 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

201 - 220 of 331