Re: some idea on providing test services to community
I looked at the code you referenced below. I am wondering why you do most of this instead of relying on the filter capabilities already available in sanitycheck. Right now you are able to run:
sanitycheck --device-testing --device-serial /dev/ttyACM0 -p frdm_k64f
and get all tests (and samples) run on the device and produce a junit file with the test results without having to maintain your own whitelist/blacklist or any device specific tests. This also will work on tests that are not part of Zephyr. It is available for all boards and works nicely. The results can be then uploaded to testrail for specific PRs.
This can of course all be done from Jenkins, but most developers or testers will be able to do that on their own and directly.
To make sure wall are using the same environment, we can use a docker image available already publicly with the SDK pre-installed.
From: testing-wg@... [mailto:testing-wg@...] On Behalf Of Hake Huang
Sent: Monday, March 25, 2019 11:13 PM
Subject: [testing-wg] some idea on providing test services to community
Pre last night’s discussion, here is the briefing of my idea.
1. We can announce that we accept pull request testing.(binary test can be later support)
a) A pull request ID shall be provide.
b) A configure file shall include in the pull request, which defines:
1. platform to test, default all.
2. Test suites to test, default all.
2. once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteer’s testing branch. This can be done via shippable.
3. The selected volunteers can start testing the pull request by his own condition.
4. Volunteers shall have a mechanism to upload their test report to testrails, with a given report name.
A comments will be added automatically to the pull request, once the report is uploaded.
I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files.
and NXP board pipeline files are defined here
I am working on the #4 for a unified junit report based on console log only.