|
Zephyr: Testing WG weekly call - Mon, 03/18/2019 8:00am-9:00am
#cal-reminder
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Reminder: When: Where: Organizer: Description: An RSVP is requested. Click here to RSVP |
|
|
|
Zephyr: Testing WG weekly call - Mon, 03/25/2019 8:00am-9:00am
#cal-reminder
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Reminder: When: Where: Organizer: Description: An RSVP is requested. Click here to RSVP |
|
|
|
some idea on providing test services to community
Hake Huang
Hi All,
Pre last night’s discussion, here is the briefing of my idea.
1. We can announce that we accept pull request testing.(binary test can be later support) a) A pull request ID shall be provide. b) A configure file shall include in the pull request, which defines: 1. platform to test, default all. 2. Test suites to test, default all.
2. once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteer’s testing branch. This can be done via shippable.
3. The selected volunteers can start testing the pull request by his own condition.
4. Volunteers shall have a mechanism to upload their test report to testrails, with a given report name. A comments will be added automatically to the pull request, once the report is uploaded.
I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files. https://jenkins.io/doc/book/blueocean/ and NXP board pipeline files are defined here https://github.com/hakehuang/zephyr_app_scm
I am working on the #4 for a unified junit report based on console log only.
Regards, Hake |
|
|
|
Re: some idea on providing test services to community
Nashif, Anas
Hake,
I looked at the code you referenced below. I am wondering why you do most of this instead of relying on the filter capabilities already available in sanitycheck. Right now you are able to run:
sanitycheck --device-testing --device-serial /dev/ttyACM0 -p frdm_k64f
and get all tests (and samples) run on the device and produce a junit file with the test results without having to maintain your own whitelist/blacklist or any device specific tests. This also will work on tests that are not part of Zephyr. It is available for all boards and works nicely. The results can be then uploaded to testrail for specific PRs.
This can of course all be done from Jenkins, but most developers or testers will be able to do that on their own and directly.
To make sure wall are using the same environment, we can use a docker image available already publicly with the SDK pre-installed.
Anas
From: testing-wg@... [mailto:testing-wg@...]
On Behalf Of Hake Huang
Sent: Monday, March 25, 2019 11:13 PM To: testing-wg@... Subject: [testing-wg] some idea on providing test services to community
Hi All,
Pre last night’s discussion, here is the briefing of my idea.
1. We can announce that we accept pull request testing.(binary test can be later support) a) A pull request ID shall be provide. b) A configure file shall include in the pull request, which defines: 1. platform to test, default all. 2. Test suites to test, default all.
2. once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteer’s testing branch. This can be done via shippable.
3. The selected volunteers can start testing the pull request by his own condition.
4. Volunteers shall have a mechanism to upload their test report to testrails, with a given report name. A comments will be added automatically to the pull request, once the report is uploaded.
I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files. https://jenkins.io/doc/book/blueocean/ and NXP board pipeline files are defined here https://github.com/hakehuang/zephyr_app_scm
I am working on the #4 for a unified junit report based on console log only.
Regards, Hake |
|
|
|
Re: some idea on providing test services to community
Hake Huang
Hi Anas,
Thanks a lot for the reviewing.
I create our own method is just because our board farm limitation. The board farm run machine can’t support Linux host. And Yes, you are right using the sanity check is a good option for most community users. So my Jenkins solution is not a must(#3). What I propose is the whole process, not the way to do automation.
Besides, I want to learn how the filter list is thus created. And so I start to analyze them one by one. To me the list is just a tracking list for features, as in NXP we have not define a clear supporting strategy for all boards in general.
Btw, I am using the zephyr docker which is at https://github.com/zephyrproject-rtos/ci-dockerfiles.git this CI works fine for me.
Regards, Hake
From: Nashif, Anas [mailto:anas.nashif@...]
Sent: Thursday, March 28, 2019 3:19 AM To: Hake Huang <hake.huang@...>; testing-wg@... Subject: RE: some idea on providing test services to community
Hake,
I looked at the code you referenced below. I am wondering why you do most of this instead of relying on the filter capabilities already available in sanitycheck. Right now you are able to run:
sanitycheck --device-testing --device-serial /dev/ttyACM0 -p frdm_k64f
and get all tests (and samples) run on the device and produce a junit file with the test results without having to maintain your own whitelist/blacklist or any device specific tests. This also will work on tests that are not part of Zephyr. It is available for all boards and works nicely. The results can be then uploaded to testrail for specific PRs.
This can of course all be done from Jenkins, but most developers or testers will be able to do that on their own and directly.
To make sure wall are using the same environment, we can use a docker image available already publicly with the SDK pre-installed.
Anas
From:
testing-wg@... [mailto:testing-wg@...]
On Behalf Of Hake Huang
Hi All,
Pre last night’s discussion, here is the briefing of my idea.
1. We can announce that we accept pull request testing.(binary test can be later support) a) A pull request ID shall be provide. b) A configure file shall include in the pull request, which defines: 1. platform to test, default all. 2. Test suites to test, default all.
2. once a pull request are created, a trigger script shall be created to pull the pull request to each test volunteer’s testing branch. This can be done via shippable.
3. The selected volunteers can start testing the pull request by his own condition.
4. Volunteers shall have a mechanism to upload their test report to testrails, with a given report name. A comments will be added automatically to the pull request, once the report is uploaded.
I demo my internal system to do #3 automatically, which is based on Jenkins blueocean pipeline files. https://jenkins.io/doc/book/blueocean/ and NXP board pipeline files are defined here https://github.com/hakehuang/zephyr_app_scm
I am working on the #4 for a unified junit report based on console log only.
Regards, Hake |
|
|
|
Zephyr: Testing WG weekly call - Mon, 04/01/2019 8:00am-9:00am
#cal-reminder
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Reminder: When: Where: Organizer: Description: An RSVP is requested. Click here to RSVP |
|
|
|
Zephyr: Testing WG weekly call - Mon, 04/08/2019 8:00am-9:00am
#cal-reminder
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Reminder: When: Where: Organizer: Description: An RSVP is requested. Click here to RSVP |
|
|
|
Re: some idea on providing test services to community
Hake Huang
Hi Anas,
In yesterday’s meeting, you mentioned that the developer will provide justify on the whitelist selection. Do those all listed in the individual pull request? Is there a filter to retrieve them?
Regards, Hake |
|
|
|
Re: some idea on providing test services to community
Nashif, Anas
Hi, No filter to find those, the whitelisting in the tests is happening for various reasons, sometimes just to limit level of execution in CI to a limited set of boards and architectures. It definitely needs to be optimized. Any whitelisting that happens now is required to have a justification the commit message and a comment in the yaml file.
Anas
From: Hake Huang [mailto:hake.huang@...]
Sent: Monday, April 8, 2019 11:12 PM To: Nashif, Anas <anas.nashif@...>; testing-wg@... Cc: Hake Huang <hake.huang@...> Subject: RE: [testing-wg] some idea on providing test services to community
Hi Anas,
In yesterday’s meeting, you mentioned that the developer will provide justify on the whitelist selection. Do those all listed in the individual pull request? Is there a filter to retrieve them?
Regards, Hake |
|
|
|
Cancelled Event: Zephyr: Testing WG weekly call - Monday, 15 April 2019
#cal-cancelled
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Cancelled: Zephyr: Testing WG weekly call This event has been cancelled. When: Where: Organizer: Description: |
|
|
|
Cancelled Event: Zephyr: Testing WG weekly call - Monday, 22 April 2019
#cal-cancelled
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Cancelled: Zephyr: Testing WG weekly call This event has been cancelled. When: Where: Organizer: Description: |
|
|
|
Zephyr: Testing WG weekly call - Mon, 04/29/2019 8:00am-9:00am
#cal-reminder
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Reminder: When: Where: Organizer: Description: An RSVP is requested. Click here to RSVP |
|
|
|
Cancelled Event: Zephyr: Testing WG weekly call - Monday, 13 May 2019
#cal-cancelled
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Cancelled: Zephyr: Testing WG weekly call This event has been cancelled. When: Where: Organizer: Description: |
|
|
|
Upcoming Event: Zephyr: Testing WG weekly call - Mon, 05/06/2019 8:00am-9:00am, Please RSVP
#cal-reminder
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Reminder: Zephyr: Testing WG weekly call When: Monday, 6 May 2019, 8:00am to 9:00am, (GMT-07:00) America/Los Angeles Where:https://zoom.us/j/679527144 An RSVP is requested. Click here to RSVP Organizer: testing-wg@... Description: Zephyr Testing WG Meeting |
|
|
|
Cancelled Event: Zephyr: Testing WG weekly call - Monday, 20 May 2019
#cal-cancelled
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Cancelled: Zephyr: Testing WG weekly call This event has been cancelled. When: Where: Organizer: Description: |
|
|
|
Upcoming Event: Zephyr: Testing WG weekly call - Mon, 06/03/2019 8:00am-9:00am, Please RSVP
#cal-reminder
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Reminder: Zephyr: Testing WG weekly call When: Monday, 3 June 2019, 8:00am to 9:00am, (GMT-07:00) America/Los Angeles Where:https://zoom.us/j/679527144 An RSVP is requested. Click here to RSVP Organizer: testing-wg@... Description: Zephyr Testing WG Meeting |
|
|
|
Cancelled Event: Zephyr: Testing WG weekly call - Monday, 10 June 2019
#cal-cancelled
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Cancelled: Zephyr: Testing WG weekly call This event has been cancelled. When: Where: Organizer: Description: |
|
|
|
Upcoming Event: Zephyr: Testing WG weekly call - Mon, 06/17/2019 8:00am-9:00am, Please RSVP
#cal-reminder
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Reminder: Zephyr: Testing WG weekly call When: Monday, 17 June 2019, 8:00am to 9:00am, (GMT-07:00) America/Los Angeles Where:https://zoom.us/j/679527144 An RSVP is requested. Click here to RSVP Organizer: testing-wg@... Description: Zephyr Testing WG Meeting |
|
|
|
Cancelled Event: Zephyr: Testing WG weekly call - Monday, 24 June 2019
#cal-cancelled
testing-wg@lists.zephyrproject.org Calendar <testing-wg@...>
Cancelled: Zephyr: Testing WG weekly call This event has been cancelled. When: Where: Organizer: Description: |
|
|
|
Testing Zephyr PTP in Renode, TestRail integration
pzierhoffer@...
Hi As I mentioned a few meetings back, we are doing gPTP tests in Renode - an open source simulation framework for multinode systems we made (www.renode.io). We were asked by Intel to work on general PTP support a few months ago, but after we were done there were some changes to Zephyr that broke it. Then we were asked to prepare a testing suite using Renode. Renode typically uses the Python-based Robot Framework (https://robotframework.org/) to describe tests. The main idea there is to write tests in text, using so called "keywords". A keyword can compare two values, run something on a host system, or run other keywords or Python code. The way it is structured it makes it relatively easy for a non-programmer to read/write tests. Additionally, Robot gives very comprehensive reports. Our PTP tests are based on two Microchip SAM E70 devices, connected via Ethernet, both running similar Zephyr code. It is possible to test it manually, following our tutorial: https://renode.readthedocs.io/en/latest/tutorials/zephyr-ptp-testing.html If you have Renode built, testing our pregenerated Zephyr binaries requires you to simply run "./test.sh tests/platforms/SAME70.robot" in the Renode directory. You can also provide your own binaries using command line switches. Right now we have six test cases: - Single Node Should Send A PDelay Request Packet (verifying if the protocol starts at all) - Slave Should Call The Phase Dis Callback (happens when a slave selects a master node) - Master Should Send Announce Packets With The Expected Parameters (here we analyze the content of packets) - Master Should Send Sync And Fup Packets With The Expected Parameters (verifying proper timestamping) - Master Should Send Syncs In Valid Intervals - Slave Should Sync Its Clock To Master (starting from very desynchronised clocks, we check if the clock on the Ethernet controller is roughly synced after some time) Adding more test cases is relatively simple and kind of depends on your imagination (everyone's welcome to contribute :) ). We are still working on adding gPTP support to our Linux-based model. Then we'll be able to verify Zephyr against Linux clients as well. The sources for these tests are here: https://github.com/renode/renode/blob/master/tests/platforms/SAME70.robot As I said before, you can run these tests manually if you'd like, but our main approach is to run them online. We have a cloud-based builder and test runner, for which I attach some screenshots. The main reason to have our own runner system is to be able to exploit Renode's capabilities and give users more information than they'd normally get. For example, right now we provide save files you can load into your local Renode installation to observe the last state of the tested environment - and continue running and debugging it. The possibilities are endless and we will add new options over time. I mentioned Robot's reports - I attach a sample report from a PTP test run on today's Zephyr. They are both in HTML and XML form. The next step for us, as requested by Intel, is to add our results to TestRail, and I would like to ask for guidance on how to proceed. I will be happy to answer all the questions you might have regarding the framework, Robot tests or this suite in particular. Best regards Piotr Zierhoffer |
|
|
