Dear Fellow Testers,

As a follow-up to the last meeting, I am sending you a proposal for verdicts used by sanitycheck to mark results. It is based on Hake’s proposal with some clarifications discussed during the meeting and our internal experience:

·       PASS - test was successful

·       FAIL - test assertion(s) failed

·       ERROR – is usually reported when test setup fails before the test even attempts to test the test assertions or some other error occurred during the execution

·       NOT_EXECUTED (reason in msg) - Test was skipped due to some conditions at the specification stage (e.g. was on a filtered list). This would indicate that the behavior (not executing) was expected

·       IGNORED – Test was skipped due to being marked manually by a user. E.g. faulty tests could get such flag before they are repaired and be skipped during the execution

·       MISSING – test was in the specification marked as to be executed, although it was not found in the report. It will work only if we decide to take an approach where after tests execution the program runs through the list of tests in the specification (extracted in advance) and looks for the results in the results report. 

The idea of marking a test as UNSTABLE is more about an extra flag than a verdict. A test can be marked as unstable if it required reruns to pass or tends to fail occasionally. This can serve as a notification to a user that seeing failures in the given test is nothing unexpected and one should try rerunning the test.

I also think that WARNING should be an additional flag, not the verdict per se. The test can be marked as PASS but have an extra notification in WARNING that some warnings appeared (e.g.  produced when test cleanup failed to restore the system)

Best regards,

Maciej

Hi All,

Meeting agenda for this weekly meeting:

1. Come up with ideas for test report.(5 minutes). round table discussion
2. CI demo on 2.3.0-RC1 trail. (5 mintues) Anash
3. Discussion on test result type definition. (20 minuts) Anash

https://elinux.org/Test_Result_Codes

proposal is as below:

·         TPASS Passed - (color: green) test was successful

·         TFAIL Failed - (color: red) test assertions failed

·         TSKIP Skipped - (color: yellow) Test was skipped because of missing pre-requisite or configuration

·         TWARN Warning - (color: magenta) test produced warnings - usually produced when test cleanup failed to restore the system

·         TBROK Broken - Broken is usually reported when test setup fails before the test even attempts to test the test assertions

·         TUNSTABLE – (color: cyan) (test was successfully executed but found failures.)

6. TestRail test result upload slow follow up. (15 minutes) Hao/Maksim

7．Define a process to integrate community CI report initial discussion. (10 minutes) round table discussion

8. Try to replace Test Rail, think about how to replace it. Come up with ideas. (10 minutes) round table discussion

       Options:

1. Open source test report with data mining support.

http://allure.qatools.ru/

       .2. Linaro test report

       https://validation.linaro.org/results/chart

Regards,

Hake

Hi Maciej,

As we discussed in this weekly meeting, thanks in advance for your help on below item:

1.     What tool does Nordic use to pass the certification? If it is not AutoPTS.

Regards,

Hake

Hi All,

This weekly meeting agenda:

1.     Progress on CI updating.

2.     Round table.

Regards,

Hake

It is closer to what testrail has than junit for sure, but we do not want to generate something that is testrail specific, it needs to be generic.

Anas

Hi Anas,

This looks good, and I have one question, can test rails accommodate all those items? If not, how can we put them into test report generated by test rail?

Regards,

Hake

Hello, due to public holidays in China and key members have holidays, we decided to cancel that meeting today.

Maksim Masalski

Meeting host

Hi,

I now added an action in the test_results repo that process the output from daily sanitycheck run and post a report to amazon, this is just a proof of concept to show how we deal with results and process them in the action and then upload back to Amazon S3.

See

https://testing.zephyrproject.org/daily_tests/zephyr-v2.2.0-2229-g9b055ecf82/report/index.html

https://github.com/zephyrproject-rtos/test_results/blob/master/.github/workflows/process_results.yml

Hopefully this will help with other steps planned and upload to testrail.

Anas

Hi Anas,

This looks good, and I have one question, can test rails accommodate all those items? If not, how can we put them into test report generated by test rail?

Regards,

Hake

Hi,

Junit format we use is very limited and has lots of constraints, I think it is time to consider something else for capturing test results, something that can contain more information than junit does, I was looking at different options, for example nunit is promising:

https://github.com/nunit/docs/wiki/Test-Result-XML-Format

Any other options? What do you think?

Anas

Hi All,

The agenda for this week’s meeting:

1.     Discussion on CI progress

a)      Anash provides some solution for part1 / 2 implemented. The version file can be access at below

https://testing.zephyrproject.org/daily_tests/versions.json

b)      A trial result repo is https://github.com/hakehuang/zephyr_ci_repo

2.     Round table on Testing issues.

3.     Next week’s topic

Regards,

Hake

Hi All,

The agenda for this week’s meeting:

1.     Discussion on CI TSC feedbacks.

a)      They expect to see the real implement soon.

2.     BLE test suite AutoPTS status update.

a)      Nordic will feedback and Andrei will host a meeting

3.     Round table on Testing issues.

4.     Next week’s topic

Regards,

Hake