Re: some idea on providing test services to community

Hake Huang

Hi Anas,


Thanks a lot for the reviewing.


I create our own method is just because our board farm limitation. The board farm run machine can’t support Linux host. And Yes, you are right using the sanity check is a good option for most community users. So my Jenkins solution is not a must(#3). What I propose is the whole process, not the way to do automation.


Besides, I want to learn how the filter list is thus created. And so I start to analyze them one by one. To me the list is just a tracking list for features, as in NXP we have not define a clear supporting strategy for all boards in general.


Btw, I am using the zephyr docker which is at this CI works fine for me.





From: Nashif, Anas [mailto:anas.nashif@...]
Sent: Thursday, March 28, 2019 3:19 AM
To: Hake Huang <hake.huang@...>; testing-wg@...
Subject: RE: some idea on providing test services to community




I looked at the code you referenced below. I am wondering why you do most of this instead of relying on the filter capabilities already available in sanitycheck. Right now you are able to run:


sanitycheck  --device-testing --device-serial /dev/ttyACM0 -p frdm_k64f


and get all tests (and samples) run on the device and produce a junit file with the test results without having to maintain your own whitelist/blacklist or any device specific tests. This also will work on tests that are not part of Zephyr. It is available for all boards and works nicely. The results can be then uploaded to testrail for specific PRs.


This can of course all be done from Jenkins, but most developers or testers will be able to do that on their own and directly.


To make sure wall are using the same environment, we can use a docker image available already publicly with the SDK pre-installed.




From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: Monday, March 25, 2019 11:13 PM
To: testing-wg@...
Subject: [testing-wg] some idea on providing test services to community


Hi All,


Pre last nights discussion, here is the briefing of my idea.


1.     We can announce that we accept pull request testing.(binary test can be later support)

a)      A pull request ID shall be provide.

b)      A configure file shall include in the pull request, which defines:

1.      platform to test, default all.

2.      Test suites to test, default all.


2.     once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteers testing branch. This can be done via shippable.


3.     The selected volunteers can start testing the pull request by his own condition.


4.     Volunteers shall have a mechanism to upload their test report to testrails, with a given report name.

A comments will be added automatically to the pull request, once the report is uploaded.


I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files.

and NXP board pipeline files are defined here


I am working on the #4 for a unified junit report based on console log only.




Join to automatically receive all group messages.