some idea on providing test services to community


Hake Huang
 

Hi All,

 

Pre last night’s discussion, here is the briefing of my idea.

 

1.     We can announce that we accept pull request testing.(binary test can be later support)

a)      A pull request ID shall be provide.

b)      A configure file shall include in the pull request, which defines:

1.      platform to test, default all.

2.      Test suites to test, default all.

 

2.     once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteer’s testing branch. This can be done via shippable.

 

3.     The selected volunteers can start testing the pull request by his own condition.

 

4.     Volunteers shall have a mechanism to upload their test report to testrails, with a given report name.

A comments will be added automatically to the pull request, once the report is uploaded.

 

I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files.

https://jenkins.io/doc/book/blueocean/

and NXP board pipeline files are defined here

https://github.com/hakehuang/zephyr_app_scm

 

I am working on the #4 for a unified junit report based on console log only.

 

Regards,

Hake


Nashif, Anas
 

Hake,

 

I looked at the code you referenced below. I am wondering why you do most of this instead of relying on the filter capabilities already available in sanitycheck. Right now you are able to run:

 

sanitycheck  --device-testing --device-serial /dev/ttyACM0 -p frdm_k64f

 

and get all tests (and samples) run on the device and produce a junit file with the test results without having to maintain your own whitelist/blacklist or any device specific tests. This also will work on tests that are not part of Zephyr. It is available for all boards and works nicely. The results can be then uploaded to testrail for specific PRs.

 

This can of course all be done from Jenkins, but most developers or testers will be able to do that on their own and directly.

 

To make sure wall are using the same environment, we can use a docker image available already publicly with the SDK pre-installed.

 

Anas

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: Monday, March 25, 2019 11:13 PM
To: testing-wg@...
Subject: [testing-wg] some idea on providing test services to community

 

Hi All,

 

Pre last night’s discussion, here is the briefing of my idea.

 

1.      We can announce that we accept pull request testing.(binary test can be later support)

a)       A pull request ID shall be provide.

b)       A configure file shall include in the pull request, which defines:

1.       platform to test, default all.

2.       Test suites to test, default all.

 

2.      once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteer’s testing branch. This can be done via shippable.

 

3.      The selected volunteers can start testing the pull request by his own condition.

 

4.      Volunteers shall have a mechanism to upload their test report to testrails, with a given report name.

A comments will be added automatically to the pull request, once the report is uploaded.

 

I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files.

https://jenkins.io/doc/book/blueocean/

and NXP board pipeline files are defined here

https://github.com/hakehuang/zephyr_app_scm

 

I am working on the #4 for a unified junit report based on console log only.

 

Regards,

Hake


Hake Huang
 

Hi Anas,

 

Thanks a lot for the reviewing.

 

I create our own method is just because our board farm limitation. The board farm run machine can’t support Linux host. And Yes, you are right using the sanity check is a good option for most community users. So my Jenkins solution is not a must(#3). What I propose is the whole process, not the way to do automation.

 

Besides, I want to learn how the filter list is thus created. And so I start to analyze them one by one. To me the list is just a tracking list for features, as in NXP we have not define a clear supporting strategy for all boards in general.

 

Btw, I am using the zephyr docker which is at https://github.com/zephyrproject-rtos/ci-dockerfiles.git this CI works fine for me.

 

Regards,

Hake

 

From: Nashif, Anas [mailto:anas.nashif@...]
Sent: Thursday, March 28, 2019 3:19 AM
To: Hake Huang <hake.huang@...>; testing-wg@...
Subject: RE: some idea on providing test services to community

 

Hake,

 

I looked at the code you referenced below. I am wondering why you do most of this instead of relying on the filter capabilities already available in sanitycheck. Right now you are able to run:

 

sanitycheck  --device-testing --device-serial /dev/ttyACM0 -p frdm_k64f

 

and get all tests (and samples) run on the device and produce a junit file with the test results without having to maintain your own whitelist/blacklist or any device specific tests. This also will work on tests that are not part of Zephyr. It is available for all boards and works nicely. The results can be then uploaded to testrail for specific PRs.

 

This can of course all be done from Jenkins, but most developers or testers will be able to do that on their own and directly.

 

To make sure wall are using the same environment, we can use a docker image available already publicly with the SDK pre-installed.

 

Anas

 

From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: Monday, March 25, 2019 11:13 PM
To: testing-wg@...
Subject: [testing-wg] some idea on providing test services to community

 

Hi All,

 

Pre last nights discussion, here is the briefing of my idea.

 

1.     We can announce that we accept pull request testing.(binary test can be later support)

a)      A pull request ID shall be provide.

b)      A configure file shall include in the pull request, which defines:

1.      platform to test, default all.

2.      Test suites to test, default all.

 

2.     once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteers testing branch. This can be done via shippable.

 

3.     The selected volunteers can start testing the pull request by his own condition.

 

4.     Volunteers shall have a mechanism to upload their test report to testrails, with a given report name.

A comments will be added automatically to the pull request, once the report is uploaded.

 

I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files.

https://jenkins.io/doc/book/blueocean/

and NXP board pipeline files are defined here

https://github.com/hakehuang/zephyr_app_scm

 

I am working on the #4 for a unified junit report based on console log only.

 

Regards,

Hake


Hake Huang
 

Hi Anas,

 

In yesterday’s meeting, you mentioned that the developer will provide justify on the whitelist selection. Do those all listed in the individual pull request? Is there a filter to retrieve them?

 

Regards,

Hake


Nashif, Anas
 

Hi,

No filter to find those, the whitelisting in the tests is happening for various reasons, sometimes just to limit level of execution in CI to a limited set of boards and architectures. It definitely needs to be optimized. Any whitelisting that happens now is required to have a justification the commit message and a comment in the yaml file.

 

Anas

 

From: Hake Huang [mailto:hake.huang@...]
Sent: Monday, April 8, 2019 11:12 PM
To: Nashif, Anas <anas.nashif@...>; testing-wg@...
Cc: Hake Huang <hake.huang@...>
Subject: RE: [testing-wg] some idea on providing test services to community

 

Hi Anas,

 

In yesterday’s meeting, you mentioned that the developer will provide justify on the whitelist selection. Do those all listed in the individual pull request? Is there a filter to retrieve them?

 

Regards,

Hake