Testing Zephyr PTP in Renode, TestRail integration
As I mentioned a few meetings back, we are doing gPTP tests in Renode - an open source simulation framework for multinode systems we made (www.renode.io).
We were asked by Intel to work on general PTP support a few months ago, but after we were done there were some changes to Zephyr that broke it. Then we were asked to prepare a testing suite using Renode.
Renode typically uses the Python-based Robot Framework (https://robotframework.org/) to describe tests.
The main idea there is to write tests in text, using so called "keywords". A keyword can compare two values, run something on a host system, or run other keywords or Python code. The way it is structured it makes it relatively easy for a non-programmer to read/write tests. Additionally, Robot gives very comprehensive reports.
Our PTP tests are based on two Microchip SAM E70 devices, connected via Ethernet, both running similar Zephyr code.
It is possible to test it manually, following our tutorial: https://renode.readthedocs.io/en/latest/tutorials/zephyr-ptp-testing.html
If you have Renode built, testing our pregenerated Zephyr binaries requires you to simply run "./test.sh tests/platforms/SAME70.robot" in the Renode directory. You can also provide your own binaries using command line switches.
Right now we have six test cases:
- Single Node Should Send A PDelay Request Packet (verifying if the protocol starts at all)
- Slave Should Call The Phase Dis Callback (happens when a slave selects a master node)
- Master Should Send Announce Packets With The Expected Parameters (here we analyze the content of packets)
- Master Should Send Sync And Fup Packets With The Expected Parameters (verifying proper timestamping)
- Master Should Send Syncs In Valid Intervals
- Slave Should Sync Its Clock To Master (starting from very desynchronised clocks, we check if the clock on the Ethernet controller is roughly synced after some time)
Adding more test cases is relatively simple and kind of depends on your imagination (everyone's welcome to contribute :) ).
We are still working on adding gPTP support to our Linux-based model. Then we'll be able to verify Zephyr against Linux clients as well.
The sources for these tests are here: https://github.com/renode/renode/blob/master/tests/platforms/SAME70.robot
As I said before, you can run these tests manually if you'd like, but our main approach is to run them online.
We have a cloud-based builder and test runner, for which I attach some screenshots. The main reason to have our own runner system is to be able to exploit Renode's capabilities and give users more information than they'd normally get.
For example, right now we provide save files you can load into your local Renode installation to observe the last state of the tested environment - and continue running and debugging it. The possibilities are endless and we will add new options over time.
I mentioned Robot's reports - I attach a sample report from a PTP test run on today's Zephyr. They are both in HTML and XML form.
The next step for us, as requested by Intel, is to add our results to TestRail, and I would like to ask for guidance on how to proceed.
I will be happy to answer all the questions you might have regarding the framework, Robot tests or this suite in particular.