Date   

TestRail Support

Masalski, Maksim
 

Hi, I received a new reply from the TestRail.

My question:

Hello, due to our open-source project policy our management decided to leave TestRail and move to our own testing platform. I would like to know how many data we have https://zephyrproject.testrail.com/ and how we can dump data from your servers? One more question, if we will stop paying for TestRail will we still have access to the previous Test reports for our project?

 

TestRail answer:

Hello Maksim,

 

Thank you for your email. If you plan to unsubscribe with TestRail, you won't have access to TestRail instance anymore. You might want to export all the reports to a different server or location before you terminate the TestRail server license so you can access them later.

 

I'm sorry to hear that you would like to unsubscribe from TestRail. If you don't mind my asking, may I know the reason why your team is planning to discontinue using TestRail?

 

If your team intends on subscribing to TestRail again in the future, we highly recommend keeping your TestRail instance active with simply 1 user at a low monthly rate of $32 to ensure your test data remains intact to avoid re-configuring the instance. Would this work for you?

 

Alternatively, we can fully close your subscription and you may export a copy of your data from Administration > Subscription within TestRail.

 

Thanks, and I look forward to your response.

 

Maksim

 


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 06/15/2020 1:00pm-2:00pm, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 15 June 2020, 1:00pm to 2:00pm, (GMT+00:00) UTC

Where:Microsoft Teams Meeting

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: [EXT] [testing-wg] Upcoming Event: Zephyr: Testing WG weekly call - Mon, 06/15/2020 1:00pm-2:00pm

Hake Huang
 

Hi All,

 

This weekly meeting agenda.

 

1.     Testrails phaseout schedule discussion(15 min)

 

2.     Test scope definition and release quality statement(30 minutes).

a)      Current coverage status.

b)      Summary on zephyr quality issues

https://github.com/zephyrproject-rtos/zephyr/issues?q=is%3Aissue+bug+is%3Aopen

 

3.     Round table discussion. (15 minutes)

 

 

Regards,

Hake

 


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 06/08/2020 1:00pm-2:00pm, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 8 June 2020, 1:00pm to 2:00pm, (GMT+00:00) UTC

Where:Microsoft Teams Meeting

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

HI All,

 

This weekly meeting agenda:

 

1.     Testrails usage discussion.(30 minutes)

a)      Test rails report discussion

b)      Performance feedback

c)      Veto for testrails

 

2.     Round table discussion. (30 minutes)

 

 

Note: We are switching to teams not using Zoom

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang via lists.zephyrproject.org
Sent: 2020
62 11:08
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite
Importance: High

 

Caution: EXT Email

Hi All,

 

Please be notified that TSC team has decided to switch to MS teams to hold the meeting, so please update your calendar with below invitation, and I attached in this mail also.

 

Teams meeting link

 

We will be using MS teams instead of Zoom next week. Please give a try on teams apps.  In case you have any question,  please contact Brett Preston bpreston@...  over slack or mail. Thanks

 

Regards,

Hake

 

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
62 7:28
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 25 May 2020
1:00pm to 2:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


about LAVA support for zephyr MCU

Hake Huang
 

Hi Gala,

 

As we talked in SLACK, I wonder whether LAVA can start to consider to provide MCU support for zephyr as below:

 

1.     Support python 3.8 and above

2.     Support run cases based on zephyr sanitycheck scripts and west build and download tool.

 

Currently MCU download support is to create a new device type for each board.

 

Thanks in advance.

 

Regards,

Hake


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Hi All,

 

Please be notified that TSC team has decided to switch to MS teams to hold the meeting, so please update your calendar with below invitation, and I attached in this mail also.

 

Teams meeting link

 

We will be using MS teams instead of Zoom next week. Please give a try on teams apps.  In case you have any question,  please contact Brett Preston bpreston@...  over slack or mail. Thanks

 

Regards,

Hake

 

 

From: testing-wg@... <testing-wg@...> On Behalf Of testing-wg@... Calendar via lists.zephyrproject.org
Sent: 2020
62 7:28
To: testing-wg@...
Subject: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Zephyr: Testing WG weekly call

When:
Monday, 25 May 2020
1:00pm to 2:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 831 716 531#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

 

 

________________________________________________________________________________


Updated Event: Zephyr: Testing WG weekly call #cal-invite

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Zephyr: Testing WG weekly call

When:
Monday, 25 May 2020
1:00pm to 2:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
Microsoft Teams Meeting

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:

________________________________________________________________________________
+1 321-558-6518 United States, Orlando (Toll)
Conference ID: 831 716 531#
Local numbers | Reset PIN | Learn more about Teams | Meeting options
 
 
________________________________________________________________________________


Upcoming Event: Zephyr: Testing WG weekly call - Mon, 06/01/2020 1:00pm-2:00pm, Please RSVP #cal-reminder

testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
 

Reminder: Zephyr: Testing WG weekly call

When: Monday, 1 June 2020, 1:00pm to 2:00pm, (GMT+00:00) UTC

Where:https://zoom.us/j/679527144

An RSVP is requested. Click here to RSVP

Organizer: testing-wg@...

Description: Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: https://zoom.us/u/ed7Ng0QxH


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Hi All,

 

This weekly meeting agenda:

 

1.     Result repo updates. (15 minutes).

a)      Auto merger PR

b)      User guider

c)      Patch checker

 

2.     TestRail plan discussion. (15 minutes)

a)       Whether we still apply budget for this tool.

 

3.     Finalize the test type as below: (15 minutes)

TYPE Name

PASS

FAIL

ERR

WARN

NOT_EXEC

IGNR

MISS

 

4.     Round table issues. (15 minutes)

 

Regards,

Hake

Zephyr: Testing WG weekly call

When:
Monday, 1 June 2020
1:00pm to 2:00pm
(UTC+00:00) UTC
Repeats: Weekly on Monday

Where:
https://zoom.us/j/679527144

Organizer: testing-wg@...

An RSVP is requested. Click here to RSVP

Description:
Zephyr Testing WG Meeting
Meeting Agenda: https://docs.google.com/document/d/1Qti_6mFPkctk9v2vnbz-IMe0ZZO2FJEpX72FaNnEfpE/edit?usp=sharing
-----
Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/679527144

Or iPhone one-tap :
    US: +16465588656,,679527144# or +16699006833,,679527144# 
Or Telephone:
    Dial(for higher quality, dial a number based on your current location): 
        US: +1 646 558 8656 or +1 669 900 6833 or +1 855 880 1246 (Toll Free) or +1 877 369 0926 (Toll Free)
    Meeting ID: 679 527 144
    International numbers available: 
https://zoom.us/u/ed7Ng0QxH


Re: TestRail reply about tests results uploading and updating of test suite

Masalski, Maksim
 

Hi, there. I received new reply from TestRail.

My question was next:

Could people check Testrail test reports without a license? We have https://zephyrproject.testrail.com/

We want to add access to our open-source community members, so every person who wants to check our test reports can access https://zephyrproject.testrail.com/

Is it possible to do? Does license have some limits on number of people who can access TestRail page? Problem is that not every community user would like to pay to access TestRail project page. Could you customize license for our project? How much it can cost?

 

Maksim

 

Reply from the TestRail:

Hello Maksim,

 

Thank you for your email. Non-TestRail users can not have access to the portal reports as they don't have an account to log into. TestRail Test Reports can only be accessed by TestRail users (licensed users) within the portal. However, the reports can be shared with non-TestRail users via email as a PDF or HTML attachment.

 

You can generate cases, Defects, Results, Summary, and Users reports by clicking on the Dashboard>Project_name>Reports>Generate any report listed on the right side>Check mark one of the options or both "Email the report as HTML or PDF attachment" > Enter email address per line>Save.

 

You can also share the scheduled reports with an external user by following the same steps. You can learn more about reports by clicking on the link given below:

 

https://www.gurock.com/testrail/videos/reporting-metrics

 

I hope this helps. Please let me know if you have any further questions or concerns.

Kind regards,

Varuna Rajput

 

--

IMPORTANT: If your team uses Jira with TestRail, please see our release announcement for TestRail 6.1.1.1021:

https://discuss.gurock.com/t/testrail-6-1-1-1021-released/

 

 

From: Masalski, Maksim
Sent: 2020
529 14:53
To: Nashif, Anas <anas.nashif@...>; testing-wg@...
Subject: RE: [testing-wg] TestRail reply about tests results uploading and updating of test suite

 

I asked if we can make updating test suite faster (adding new test cases if necessary, add new sections if necessary). Because the fastest way in TestRail to update test suite is to use GUI and directly upload .xml file manually. (attached picture)

I asked them if TestRail has some API to upload results like in GUI using button Import Cases.

 

Maksim

 

 

From: Nashif, Anas <anas.nashif@...>
Sent: 2020
529 14:47
To: Masalski, Maksim <
maksim.masalski@...>; testing-wg@...
Subject: Re: [testing-wg] TestRail reply about tests results uploading and updating of test suite

 

What was the question you asked? I am still do not understand what the problem is, is this related to updating the testsuite or uploading test results?

 

Anas

 

From: <testing-wg@...> on behalf of "Masalski, Maksim" <maksim.masalski@...>
Date: Monday, 25 May 2020 at 08:56
To: "
testing-wg@..." <testing-wg@...>
Subject: [testing-wg] TestRail reply about tests results uploading and updating of test suite

 

Hi, I received a reply from the Testrail support. I asked how to make faster uploading of the test results

Hello Maksim,

 

Thank you for your email. I am not certain, if there is anything specific in TestRail that would support multithreading.

 

With TestRail Server, It is my understanding that you would hit the rate limit quickly as the concurrent API processes will create thousands of new test cases each day.

 

Technically, In my opinion with TestRail Server, the high number of daily uploads via API wouldn't be supported. Also, it would not be recommended since each new test case creation is processing a lot of actions in the database, and running multiple test case creations simultaneously is going to slow the whole process down.

 

You can find additional information about accessing API by clicking on the link given below:

 

http://docs.gurock.com/testrail-api2/start

 

I hope this helps. Please let me know if you have any further questions or concerns.

Kind regards,

Varuna Rajput

 


Re: TestRail reply about tests results uploading and updating of test suite

Masalski, Maksim
 

I asked if we can make updating test suite faster (adding new test cases if necessary, add new sections if necessary). Because the fastest way in TestRail to update test suite is to use GUI and directly upload .xml file manually. (attached picture)

I asked them if TestRail has some API to upload results like in GUI using button Import Cases.

 

Maksim

 

 

From: Nashif, Anas <anas.nashif@...>
Sent: 2020
529 14:47
To: Masalski, Maksim <maksim.masalski@...>; testing-wg@...
Subject: Re: [testing-wg] TestRail reply about tests results uploading and updating of test suite

 

What was the question you asked? I am still do not understand what the problem is, is this related to updating the testsuite or uploading test results?

 

Anas

 

From: <testing-wg@...> on behalf of "Masalski, Maksim" <maksim.masalski@...>
Date: Monday, 25 May 2020 at 08:56
To: "testing-wg@..." <testing-wg@...>
Subject: [testing-wg] TestRail reply about tests results uploading and updating of test suite

 

Hi, I received a reply from the Testrail support. I asked how to make faster uploading of the test results

Hello Maksim,

 

Thank you for your email. I am not certain, if there is anything specific in TestRail that would support multithreading.

 

With TestRail Server, It is my understanding that you would hit the rate limit quickly as the concurrent API processes will create thousands of new test cases each day.

 

Technically, In my opinion with TestRail Server, the high number of daily uploads via API wouldn't be supported. Also, it would not be recommended since each new test case creation is processing a lot of actions in the database, and running multiple test case creations simultaneously is going to slow the whole process down.

 

You can find additional information about accessing API by clicking on the link given below:

 

http://docs.gurock.com/testrail-api2/start

 

I hope this helps. Please let me know if you have any further questions or concerns.

Kind regards,

Varuna Rajput

 


Re: TestRail reply about tests results uploading and updating of test suite

Nashif, Anas
 

What was the question you asked? I am still do not understand what the problem is, is this related to updating the testsuite or uploading test results?

 

Anas

 

From: <testing-wg@...> on behalf of "Masalski, Maksim" <maksim.masalski@...>
Date: Monday, 25 May 2020 at 08:56
To: "testing-wg@..." <testing-wg@...>
Subject: [testing-wg] TestRail reply about tests results uploading and updating of test suite

 

Hi, I received a reply from the Testrail support. I asked how to make faster uploading of the test results

Hello Maksim,

 

Thank you for your email. I am not certain, if there is anything specific in TestRail that would support multithreading.

 

With TestRail Server, It is my understanding that you would hit the rate limit quickly as the concurrent API processes will create thousands of new test cases each day.

 

Technically, In my opinion with TestRail Server, the high number of daily uploads via API wouldn't be supported. Also, it would not be recommended since each new test case creation is processing a lot of actions in the database, and running multiple test case creations simultaneously is going to slow the whole process down.

 

You can find additional information about accessing API by clicking on the link given below:

 

http://docs.gurock.com/testrail-api2/start

 

I hope this helps. Please let me know if you have any further questions or concerns.

Kind regards,

Varuna Rajput

 


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Hi Anas,

 

From recent feedback below types are most supported.

 

TYPE Name

PASS

FAIL

ERR

WARN

NOT_EXEC

IGNR

MISS

 

 

Regards,

Hake

 

From: Nashif, Anas <anas.nashif@...>
Sent: 2020
526 20:51
To: Hake Huang <hake.huang@...>; maxxliferobot@...; testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Didn’t we agree to remove the T?

Where is the 5 char restriction coming from?

 

From: <testing-wg@...> on behalf of Hake Huang <hake.huang@...>
Date: Monday, 25 May 2020 at 23:35
To: "maxxliferobot@..." <maxxliferobot@...>, "testing-wg@..." <testing-wg@...>
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Summary all comments: I propose below type definitions:

 

Basic rules is:

1.  5 chars for each type

2.  clear definition

3.  test specific token

 

TYPE Name

Explanation

Actions

TPASS

test was successful

 

TFAIL

test assertion(s) failed

Report github issue

TERRR

test setup fails

Owner shall recheck

TWARN

other error occurred during the execution

Report github issue with low priority

TNEXE

Test was skipped due to some conditions at the specification stage

No actions

TIGNR

Test was skipped due to being marked manually by a user

No actions

TMISS

test was in the specification marked as to be executed, although it was not found in the report

Owner shall recheck

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Maksim Masalski via lists.zephyrproject.org
Sent: 2020
525 20:47
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Today we should agree on how test types will be defined. To my mind each test type definition should have only 1 robust definition. It is better to avoid "or", "some" words in the definition, because it will make unclear during testing why that test type happened. Will mark my comments using cursive font

a)      PASS - test was successful Agree

b)      FAIL - test assertion(s) failed Agree

c)      ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or(!) some other error occurred during the execution. I think necessary to split that. Make ERR1 and ERR_DARK
ERR1 -reported when test setup fails before the test even attempts to test the test assertions. ERR_DARK-some other error occurred during the execution, Nobody knows what exactly error is.

d)      NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected Agree, only if reason will be in msg. Why to make it shorter? NOT_EXEC

e)      IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution. Why to make it shorter? IGNORE

f)       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. Why to make it shorter? MISS


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Erwan Gouriou
 

Hello, 

I agree removing the T would make things easier to read.
Besides, we're using a poor man's test instability status using test result first letter:

tests/arch/arm/arm_irq_vector_table/arch.interrupt.arm.irq_vector_table, nucleo_f746zg (instable: pfpfffpppffffppfp)

Getting the second letter is of course doable also, but I like this simplicity.
(Also, it's good that each status begins by a different letter).

Erwan


On Tue, 26 May 2020 at 09:02, Masalski, Maksim <maksim.masalski@...> wrote:

Hello, Hake and team. I inspected your abbreviations of test types. I have some comments.

1. I can’t understand why we need that “T”, it makes difficult to read text. I vote for clear and understandable definitions without “T” letter.

2. TERRR, as I understand should be just ERR.

3. TNEXE” maybe better NOT_EXEC? EXE feels like Windows execution file abbreviation.

 

My variants are below.

TYPE Name

PASS

FAIL

ERR

WARN

NOT_EXEC

IGNR

MISS

 

Maksim

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang
Sent: 2020
526 6:35
To: maxxliferobot@...; testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Summary all comments: I propose below type definitions:

 

Basic rules is:

1.      5 chars for each type

2.      clear definition

3.      test specific token

 

TYPE Name

Explanation

Actions

TPASS

test was successful

 

TFAIL

test assertion(s) failed

Report github issue

TERRR

test setup fails

Owner shall recheck

TWARN

other error occurred during the execution

Report github issue with low priority

TNEXE

Test was skipped due to some conditions at the specification stage

No actions

TIGNR

Test was skipped due to being marked manually by a user

No actions

TMISS

test was in the specification marked as to be executed, although it was not found in the report

Owner shall recheck

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Maksim Masalski via lists.zephyrproject.org
Sent: 2020
525 20:47
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Today we should agree on how test types will be defined. To my mind each test type definition should have only 1 robust definition. It is better to avoid "or", "some" words in the definition, because it will make unclear during testing why that test type happened. Will mark my comments using cursive font

a)      PASS - test was successful Agree

b)      FAIL - test assertion(s) failed Agree

c)      ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or(!) some other error occurred during the execution. I think necessary to split that. Make ERR1 and ERR_DARK
ERR1 -reported when test setup fails before the test even attempts to test the test assertions. ERR_DARK-some other error occurred during the execution, Nobody knows what exactly error is.

d)      NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected Agree, only if reason will be in msg. Why to make it shorter? NOT_EXEC

e)      IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution. Why to make it shorter? IGNORE

f)       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. Why to make it shorter? MISS


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Nashif, Anas
 

Didn’t we agree to remove the T?

Where is the 5 char restriction coming from?

 

From: <testing-wg@...> on behalf of Hake Huang <hake.huang@...>
Date: Monday, 25 May 2020 at 23:35
To: "maxxliferobot@..." <maxxliferobot@...>, "testing-wg@..." <testing-wg@...>
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Summary all comments: I propose below type definitions:

 

Basic rules is:

1.  5 chars for each type

2.  clear definition

3.  test specific token

 

TYPE Name

Explanation

Actions

TPASS

test was successful

 

TFAIL

test assertion(s) failed

Report github issue

TERRR

test setup fails

Owner shall recheck

TWARN

other error occurred during the execution

Report github issue with low priority

TNEXE

Test was skipped due to some conditions at the specification stage

No actions

TIGNR

Test was skipped due to being marked manually by a user

No actions

TMISS

test was in the specification marked as to be executed, although it was not found in the report

Owner shall recheck

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Maksim Masalski via lists.zephyrproject.org
Sent: 2020
525 20:47
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Today we should agree on how test types will be defined. To my mind each test type definition should have only 1 robust definition. It is better to avoid "or", "some" words in the definition, because it will make unclear during testing why that test type happened. Will mark my comments using cursive font

a)      PASS - test was successful Agree

b)      FAIL - test assertion(s) failed Agree

c)      ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or(!) some other error occurred during the execution. I think necessary to split that. Make ERR1 and ERR_DARK
ERR1 -reported when test setup fails before the test even attempts to test the test assertions. ERR_DARK-some other error occurred during the execution, Nobody knows what exactly error is.

d)      NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected Agree, only if reason will be in msg. Why to make it shorter? NOT_EXEC

e)      IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution. Why to make it shorter? IGNORE

f)       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. Why to make it shorter? MISS


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Hi Maciej,

 

WARN is something like all test steps pass. But you get an unexpected log. For example, some warning from the kernel.

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Perkowski, Maciej via lists.zephyrproject.org
Sent: 2020
526 15:09
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Hi,

+1 for Maksim suggestions. I am also still wondering, if WARN supposed to be a verdict. According to the proposition, if there was WARN does it mean that test was neither PASS, nor FAIL, nor ERR but still something else?

Cheers,

Maciej

 

From: testing-wg@... <testing-wg@...> On Behalf Of Masalski, Maksim via lists.zephyrproject.org
Sent: Tuesday, May 26, 2020 9:02 AM
To: Hake Huang <hake.huang@...>; maxxliferobot@...; testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Hello, Hake and team. I inspected your abbreviations of test types. I have some comments.

1. I cant understand why we need that T, it makes difficult to read text. I vote for clear and understandable definitions without T letter.

2. TERRR, as I understand should be just ERR.

3. TNEXE maybe better NOT_EXEC? EXE feels like Windows execution file abbreviation.

 

My variants are below.

TYPE Name

PASS

FAIL

ERR

WARN

NOT_EXEC

IGNR

MISS

 

Maksim

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang
Sent: 2020
526 6:35
To: maxxliferobot@...; testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Summary all comments: I propose below type definitions:

 

Basic rules is:

1.     5 chars for each type

2.     clear definition

3.     test specific token

 

TYPE Name

Explanation

Actions

TPASS

test was successful

 

TFAIL

test assertion(s) failed

Report github issue

TERRR

test setup fails

Owner shall recheck

TWARN

other error occurred during the execution

Report github issue with low priority

TNEXE

Test was skipped due to some conditions at the specification stage

No actions

TIGNR

Test was skipped due to being marked manually by a user

No actions

TMISS

test was in the specification marked as to be executed, although it was not found in the report

Owner shall recheck

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Maksim Masalski via lists.zephyrproject.org
Sent: 2020
525 20:47
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Today we should agree on how test types will be defined. To my mind each test type definition should have only 1 robust definition. It is better to avoid "or", "some" words in the definition, because it will make unclear during testing why that test type happened. Will mark my comments using cursive font

a)      PASS - test was successful Agree

b)      FAIL - test assertion(s) failed Agree

c)      ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or(!) some other error occurred during the execution. I think necessary to split that. Make ERR1 and ERR_DARK
ERR1 -reported when test setup fails before the test even attempts to test the test assertions. ERR_DARK-some other error occurred during the execution, Nobody knows what exactly error is.

d)      NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected Agree, only if reason will be in msg. Why to make it shorter? NOT_EXEC

e)      IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution. Why to make it shorter? IGNORE

f)       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. Why to make it shorter? MISS


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Hake Huang
 

Hi Maksim,

 

At least we can get aligned with type catalogs, i.e. 7 types.

 

leading with ‘T’ means this is test result. The NOT_EXEC looks strange to me because other types are using one word abbreviation. But I am OK for any naming, just want to settle this issue.

 

Regards,

Hake

 

From: Masalski, Maksim <maksim.masalski@...>
Sent: 2020
526 15:02
To: Hake Huang <hake.huang@...>; maxxliferobot@...; testing-wg@...
Subject: RE: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Hello, Hake and team. I inspected your abbreviations of test types. I have some comments.

1. I cant understand why we need that T, it makes difficult to read text. I vote for clear and understandable definitions without T letter.

2. TERRR, as I understand should be just ERR.

3. TNEXE maybe better NOT_EXEC? EXE feels like Windows execution file abbreviation.

 

My variants are below.

TYPE Name

PASS

FAIL

ERR

WARN

NOT_EXEC

IGNR

MISS

 

Maksim

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang
Sent: 2020
526 6:35
To: maxxliferobot@...; testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Summary all comments: I propose below type definitions:

 

Basic rules is:

1.     5 chars for each type

2.     clear definition

3.     test specific token

 

TYPE Name

Explanation

Actions

TPASS

test was successful

 

TFAIL

test assertion(s) failed

Report github issue

TERRR

test setup fails

Owner shall recheck

TWARN

other error occurred during the execution

Report github issue with low priority

TNEXE

Test was skipped due to some conditions at the specification stage

No actions

TIGNR

Test was skipped due to being marked manually by a user

No actions

TMISS

test was in the specification marked as to be executed, although it was not found in the report

Owner shall recheck

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Maksim Masalski via lists.zephyrproject.org
Sent: 2020
525 20:47
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Today we should agree on how test types will be defined. To my mind each test type definition should have only 1 robust definition. It is better to avoid "or", "some" words in the definition, because it will make unclear during testing why that test type happened. Will mark my comments using cursive font

a)      PASS - test was successful Agree

b)      FAIL - test assertion(s) failed Agree

c)      ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or(!) some other error occurred during the execution. I think necessary to split that. Make ERR1 and ERR_DARK
ERR1 -reported when test setup fails before the test even attempts to test the test assertions. ERR_DARK-some other error occurred during the execution, Nobody knows what exactly error is.

d)      NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected Agree, only if reason will be in msg. Why to make it shorter? NOT_EXEC

e)      IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution. Why to make it shorter? IGNORE

f)       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. Why to make it shorter? MISS


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Perkowski, Maciej
 

Hi,

+1 for Maksim suggestions. I am also still wondering, if WARN supposed to be a verdict. According to the proposition, if there was WARN does it mean that test was neither PASS, nor FAIL, nor ERR but still something else?

Cheers,

Maciej

 

From: testing-wg@... <testing-wg@...> On Behalf Of Masalski, Maksim via lists.zephyrproject.org
Sent: Tuesday, May 26, 2020 9:02 AM
To: Hake Huang <hake.huang@...>; maxxliferobot@...; testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Hello, Hake and team. I inspected your abbreviations of test types. I have some comments.

1. I cant understand why we need that T, it makes difficult to read text. I vote for clear and understandable definitions without T letter.

2. TERRR, as I understand should be just ERR.

3. TNEXE maybe better NOT_EXEC? EXE feels like Windows execution file abbreviation.

 

My variants are below.

TYPE Name

PASS

FAIL

ERR

WARN

NOT_EXEC

IGNR

MISS

 

Maksim

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang
Sent: 2020
526 6:35
To: maxxliferobot@...; testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Summary all comments: I propose below type definitions:

 

Basic rules is:

1.     5 chars for each type

2.     clear definition

3.     test specific token

 

TYPE Name

Explanation

Actions

TPASS

test was successful

 

TFAIL

test assertion(s) failed

Report github issue

TERRR

test setup fails

Owner shall recheck

TWARN

other error occurred during the execution

Report github issue with low priority

TNEXE

Test was skipped due to some conditions at the specification stage

No actions

TIGNR

Test was skipped due to being marked manually by a user

No actions

TMISS

test was in the specification marked as to be executed, although it was not found in the report

Owner shall recheck

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Maksim Masalski via lists.zephyrproject.org
Sent: 2020
525 20:47
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Today we should agree on how test types will be defined. To my mind each test type definition should have only 1 robust definition. It is better to avoid "or", "some" words in the definition, because it will make unclear during testing why that test type happened. Will mark my comments using cursive font

a)      PASS - test was successful Agree

b)      FAIL - test assertion(s) failed Agree

c)      ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or(!) some other error occurred during the execution. I think necessary to split that. Make ERR1 and ERR_DARK
ERR1 -reported when test setup fails before the test even attempts to test the test assertions. ERR_DARK-some other error occurred during the execution, Nobody knows what exactly error is.

d)      NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected Agree, only if reason will be in msg. Why to make it shorter? NOT_EXEC

e)      IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution. Why to make it shorter? IGNORE

f)       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. Why to make it shorter? MISS


Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

Masalski, Maksim
 

Hello, Hake and team. I inspected your abbreviations of test types. I have some comments.

1. I can’t understand why we need that “T”, it makes difficult to read text. I vote for clear and understandable definitions without “T” letter.

2. TERRR, as I understand should be just ERR.

3. TNEXE” maybe better NOT_EXEC? EXE feels like Windows execution file abbreviation.

 

My variants are below.

TYPE Name

PASS

FAIL

ERR

WARN

NOT_EXEC

IGNR

MISS

 

Maksim

From: testing-wg@... <testing-wg@...> On Behalf Of Hake Huang
Sent: 2020
526 6:35
To: maxxliferobot@...; testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Summary all comments: I propose below type definitions:

 

Basic rules is:

1.      5 chars for each type

2.      clear definition

3.      test specific token

 

TYPE Name

Explanation

Actions

TPASS

test was successful

 

TFAIL

test assertion(s) failed

Report github issue

TERRR

test setup fails

Owner shall recheck

TWARN

other error occurred during the execution

Report github issue with low priority

TNEXE

Test was skipped due to some conditions at the specification stage

No actions

TIGNR

Test was skipped due to being marked manually by a user

No actions

TMISS

test was in the specification marked as to be executed, although it was not found in the report

Owner shall recheck

 

 

Regards,

Hake

 

From: testing-wg@... <testing-wg@...> On Behalf Of Maksim Masalski via lists.zephyrproject.org
Sent: 2020
525 20:47
To: testing-wg@...
Subject: Re: [EXT] [testing-wg] Updated Event: Zephyr: Testing WG weekly call #cal-invite

 

Caution: EXT Email

Today we should agree on how test types will be defined. To my mind each test type definition should have only 1 robust definition. It is better to avoid "or", "some" words in the definition, because it will make unclear during testing why that test type happened. Will mark my comments using cursive font

a)      PASS - test was successful Agree

b)      FAIL - test assertion(s) failed Agree

c)      ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or(!) some other error occurred during the execution. I think necessary to split that. Make ERR1 and ERR_DARK
ERR1 -reported when test setup fails before the test even attempts to test the test assertions. ERR_DARK-some other error occurred during the execution, Nobody knows what exactly error is.

d)      NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected Agree, only if reason will be in msg. Why to make it shorter? NOT_EXEC

e)      IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution. Why to make it shorter? IGNORE

f)       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. Why to make it shorter? MISS

221 - 240 of 377