Date   

Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Summary all comments: I propose below type definitions:

 

Basic rules is:

1.     5 chars for each type

2.     clear definition

3.     test specific token

 

TYPE Name

Explanation

Actions

TPASS

test was successful

 

TFAIL

test assertion(s) failed

Report github issue

TERRR

test setup fails

Owner shall recheck

TWARN

other error occurred during the execution

Report github issue with low priority

TNEXE

Test was skipped due to some conditions at the specification stage

No actions

TIGNR

Test was skipped due to being marked manually by a user

No actions

TMISS

test was in the specification marked as to be executed, although it was not found in the report

Owner shall recheck

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Maksim Masalski via lists.zephyrproject.org
Sent: 2020
525 20:47
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Today we should agree on how test types will be defined. To my mind each test type definition should have only 1 robust definition. It is better to avoid "or", "some" words in the definition, because it will make unclear during testing why that test type happened. Will mark my comments using cursive font

a)      PASS - test was successful Agree

b)      FAIL - test assertion(s) failed Agree

c)      ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or(!) some other error occurred during the execution. I think necessary to split that. Make ERR1 and ERR_DARK
ERR1 -reported when test setup fails before the test even attempts to test the test assertions. ERR_DARK-some other error occurred during the execution, Nobody knows what exactly error is.

d)      NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected Agree, only if reason will be in msg. Why to make it shorter? NOT_EXEC

e)      IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution. Why to make it shorter? IGNORE

f)       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. Why to make it shorter? MISS


TestRail reply about tests results uploading and updating of test suite

Masalski, Maksim
 

Hi, I received a reply from the Testrail support. I asked how to make faster uploading of the test results

Hello Maksim,

 

Thank you for your email. I am not certain, if there is anything specific in TestRail that would support multithreading.

 

With TestRail Server, It is my understanding that you would hit the rate limit quickly as the concurrent API processes will create thousands of new test cases each day.

 

Technically, In my opinion with TestRail Server, the high number of daily uploads via API wouldn't be supported. Also, it would not be recommended since each new test case creation is processing a lot of actions in the database, and running multiple test case creations simultaneously is going to slow the whole process down.

 

You can find additional information about accessing API by clicking on the link given below:

 

http://docs.gurock.com/testrail-api2/start

 

I hope this helps. Please let me know if you have any further questions or concerns.

Kind regards,

Varuna Rajput

 


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 05/25/2020 1:00pm-2:00pm, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 25 May 2020, 1:00pm to 2:00pm, (GMT+00:00) UTC

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Maksim Masalski <maxxliferobot@...>
 

Today we should agree on how test types will be defined. To my mind each test type definition should have only 1 robust definition. It is better to avoid "or", "some" words in the definition, because it will make unclear during testing why that test type happened. Will mark my comments using cursive font

a)      PASS - test was successful Agree

b)      FAIL - test assertion(s) failed Agree

c)      ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or(!) some other error occurred during the execution. I think necessary to split that. Make ERR1 and ERR_DARK
ERR1 -
reported when test setup fails before the test even attempts to test the test assertions. ERR_DARK-some other error occurred during the execution, Nobody knows what exactly error is.

d)      NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected Agree, only if reason will be in msg. Why to make it shorter? NOT_EXEC

e)      IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution. Why to make it shorter? IGNORE

f)       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. Why to make it shorter? MISS


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Nashif, Anas
 

Apologies won’t be able to attend today.

 

Anas

 

From: <testing-wg@...> on behalf of Hake Huang <hake.huang@...>
Date: Sunday, 24 May 2020 at 22:42
To: "testing-wg@..." <testing-wg@...>
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Hi All,

 

This weekly meeting agenda.

 

1.  Progress on test result repo. (20 Minutes)

2.  Final the test type definition.

a)   PASS - test was successful

b)   FAIL - test assertion(s) failed

c)   ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or some other error occurred during the execution

d)   NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected

e)   IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution

f)   MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. 

 

3.  Round table discussion. (20 minutes)

 

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
521 0:33
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 25 May 2020
1:00pm to 2:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
https://zoom.us/j/679527144

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: 
https://zoom.us/u/ed7Ng0QxH


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Hi All,

 

This weekly meeting agenda.

 

1.     Progress on test result repo. (20 Minutes)

2.     Final the test type definition.

a)      PASS - test was successful

b)      FAIL - test assertion(s) failed

c)      ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or some other error occurred during the execution

d)      NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected

e)      IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution

f)       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. 

 

3.     Round table discussion. (20 minutes)

 

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
521 0:33
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 25 May 2020
1:00pm to 2:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
https://zoom.us/j/679527144

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: 
https://zoom.us/u/ed7Ng0QxH


Updated Event: Zephyr: Testing WG weekly call #cal-invite

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Zephyr: Testing WG weekly call

When:
Monday, 25 May 2020
1:00pm to 2:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
https://zoom.us/j/679527144

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


SC verdicts follow-up

Perkowski, Maciej
 

Dear Fellow Testers,

As a follow-up to the last meeting, I am sending you a proposal for verdicts used by sanitycheck to mark results. It is based on Hake’s proposal with some clarifications discussed during the meeting and our internal experience:

·       PASS - test was successful

·       FAIL - test assertion(s) failed

·       ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or some other error occurred during the execution

·       NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected

·       IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution

·       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. 

The idea of marking a test as UNSTABLE is more about an extra flag than a verdict. A test can be marked as unstable if it required reruns to pass or tends to fail occasionally. This can serve as a notification to a user that seeing failures in the given test is nothing unexpected and one should try rerunning the test.

I also think that WARNING should be an additional flag, not the verdict per se. The test can be marked as PASS but have an extra notification in WARNING that some warnings appeared (e.g.  produced when test cleanup failed to restore the system)

Best regards,

Maciej

 


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 05/18/2020 6:00am-7:00am, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 18 May 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Re: [EXT] [testing-wg] Upcoming Event: Zephyr: Testing WG weekly call - Mon, 05/18/2020 6:00am-7:00am, Please RSVP #cal-reminder

Hake Huang
 

Hi All,

 

Meeting agenda for this weekly meeting:

 

  1. Come up with ideas for test report.(5 minutes). round table discussion
  2. CI demo on 2.3.0-RC1 trail. (5 mintues) Anash
  3. Discussion on test result type definition. (20 minuts) Anash

https://elinux.org/Test_Result_Codes

proposal is as below:

·         TPASS Passed - (color: green) test was successful

·         TFAIL Failed - (color: red) test assertions failed

·         TSKIP Skipped - (color: yellow) Test was skipped because of missing pre-requisite or configuration

·         TWARN Warning - (color: magenta) test produced warnings - usually produced when test cleanup failed to restore the system

·         TBROK Broken - Broken is usually reported when test setup fails before the test even attempts to test the test assertions

·         TUNSTABLE – (color: cyan) (test was successfully executed but found failures.)

 

6. TestRail test result upload slow follow up. (15 minutes) Hao/Maksim

   

7Define a process to integrate community CI report initial discussion. (10 minutes) round table discussion

 

8. Try to replace Test Rail, think about how to replace it. Come up with ideas. (10 minutes) round table discussion

       Options:

  1. Open source test report with data mining support.

http://allure.qatools.ru/

       .2. Linaro test report

       https://validation.linaro.org/results/chart

 

Regards,

Hake


about bluetooth certification

Hake Huang
 

Hi Maciej,

 

As we discussed in this weekly meeting, thanks in advance for your help on below item:

 

1.     What tool does Nordic use to pass the certification? If it is not AutoPTS.

 

 

Regards,

Hake


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 05/11/2020 6:00am-7:00am, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 11 May 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Re: [EXT] [testing-wg] event: Zephyr: Testing WG weekly call - Monday, 11 May 2020

Hake Huang
 

Hi All,

 

This weekly meeting agenda:

 

1.     Progress on CI updating.

 

2.     Round table.

 

Regards,

Hake


Re: [EXT] [testing-wg] alternatives to junit

Nashif, Anas
 

It is closer to what testrail has than junit for sure, but we do not want to generate something that is testrail specific, it needs to be generic.

 

Anas

 

From: Hake Huang <hake.huang@...>
Date: Monday, 4 May 2020 at 09:20
To: Anas Nashif <anas.nashif@...>, "testing-wg@..." <testing-wg@...>
Subject: RE: [EXT] [testing-wg] alternatives to junit

 

Hi Anas,

 

This looks good, and I have one question, can test rails accommodate all those items? If not, how can we put them into test report generated by test rail?

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Nashif, Anas via lists.zephyrproject.org
Sent: 2020
54 20:23
To: testing-wg@...
Subject: [EXT] [testing-wg] alternatives to junit

 

Caution: EXT Email

Hi,

 

Junit format we use is very limited and has lots of constraints, I think it is time to consider something else for capturing test results, something that can contain more information than junit does, I was looking at different options, for example nunit is promising:

 

https://github.com/nunit/docs/wiki/Test-Result-XML-Format

 

 

Any other options? What do you think?

 

Anas


Testing WG Meeting Monday May 4 Cancelled

Masalski, Maksim
 

Hello, due to public holidays in China and key members have holidays, we decided to cancel that meeting today.

 

Maksim Masalski

Meeting host

 


hw test workflow updates

Nashif, Anas
 

Hi,

I now added an action in the test_results repo that process the output from daily sanitycheck run and post a report to amazon, this is just a proof of concept to show how we deal with results and process them in the action and then upload back to Amazon S3.

 

 

See

 

https://testing.zephyrproject.org/daily_tests/zephyr-v2.2.0-2229-g9b055ecf82/report/index.html

 

https://github.com/zephyrproject-rtos/test_results/blob/master/.github/workflows/process_results.yml

 

 

Hopefully this will help with other steps planned and upload to testrail.

 

Anas


Re: [EXT] [testing-wg] alternatives to junit

Hake Huang
 

Hi Anas,

 

This looks good, and I have one question, can test rails accommodate all those items? If not, how can we put them into test report generated by test rail?

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Nashif, Anas via lists.zephyrproject.org
Sent: 2020
54 20:23
To: testing-wg@...
Subject: [EXT] [testing-wg] alternatives to junit

 

Caution: EXT Email

Hi,

 

Junit format we use is very limited and has lots of constraints, I think it is time to consider something else for capturing test results, something that can contain more information than junit does, I was looking at different options, for example nunit is promising:

 

https://github.com/nunit/docs/wiki/Test-Result-XML-Format

 

 

Any other options? What do you think?

 

Anas


alternatives to junit

Nashif, Anas
 

Hi,

 

Junit format we use is very limited and has lots of constraints, I think it is time to consider something else for capturing test results, something that can contain more information than junit does, I was looking at different options, for example nunit is promising:

 

https://github.com/nunit/docs/wiki/Test-Result-XML-Format

 

 

Any other options? What do you think?

 

Anas


Cancelled Event: Zephyr: Testing WG weekly call - Monday, 4 May 2020 #cal-cancelled

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Cancelled: Zephyr: Testing WG weekly call

This event has been cancelled.

When:
Monday, 4 May 2020
6:00am to 7:00am
(UTC-07:00) America/Los Angeles

Where:
https://zoom.us/j/679527144

Organizer: testing-wg@...

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 04/27/2020 6:00am-7:00am, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 27 April 2020, 6:00am to 7:00am, (GMT-07:00) America/Los Angeles

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH

241 - 260 of 377