Hi Björn,

I asked this question on Slack some time back and Anas answered it's not possible (yet?) for various reasons, one of them related to github accounts and security.

Fortunately something even more useful is already possible: running the checks yourself. Just run this compliance_script.py script:

https://github.com/zephyrproject-rtos/ci-tools/tree/master/scripts

https://github.com/zephyrproject-rtos/ci-tools/pulls

I know run this script *before* uploading anything to github as it obviously saves round trips and time.

You also need the "sanitycheck" script in the other, main repo.

I think sanitycheck is mentioned in the online documentation and the check_compliance about to be.

On a related CI topic, here's why the code coverage report is very often bogus:

https://github.com/codecov/codecov-bash/issues/83 thumbs up to "vote" for it

Cheers,

Marc

Status is actually available as a comment in the PR. We are planning at some point to use the Checks API from GH which would make things more clear.

The scripts now provide the output so you do not have to look at the actual CI log which might be verbose and misleading. If any information is missing in the comments posted by CI, please let us know.

Anas

Hi Björn,

I asked this question on Slack some time back and Anas answered it's not possible (yet?) for various reasons, one of them related to github accounts and security.

Fortunately something even more useful is already possible: running the checks yourself. Just run this compliance_script.py script:

https://github.com/zephyrproject-rtos/ci-tools/tree/master/scripts

https://github.com/zephyrproject-rtos/ci-tools/pulls

I know run this script *before* uploading anything to github as it obviously saves round trips and time.

You also need the "sanitycheck" script in the other, main repo.

I think sanitycheck is mentioned in the online documentation and the check_compliance about to be.

On a related CI topic, here's why the code coverage report is very often bogus:

https://github.com/codecov/codecov-bash/issues/83 thumbs up to "vote" for it

Cheers,

Marc

Hi,

A couple things I find confusing:

- There's only the ("beautified") output but not the commands that produced them as is typical in CI, so no obvious indication of what to do to test the next git push and make sure it will actually fix the issues before pushing again.

- The comments with the output are typically some distance away from the red/green "traffic lights" with unrelated stuff in between

- The "Details" buttons on the other hand *are* in the traffic lights and seem really designed to point at the output, I think most people expect them to but they point at the documentation instead.

> you do not have to look at the actual CI log which might be verbose and misleading

Well (most of) the actual CI log is or should be what one gets anyway when running the sanitycheck / compliance.py / etc. scripts locally so if that output is confusing then users will just submit patches to make it better (as I just did) and keep everything consistent.

The less difference and distance between users and CI, the better.

> The actual output is always available in a comment posted by zephyrbot in the same PR. The comment is updated for every run

It's interesting that this comment preserves some review history unlike the aggressive way the "traffic lights" in particular and github in general rewrite and bury review history. So this is Good but unfortunately inconsistent with (Bad) github so a bit confusing again. For instance it's difficult to relate the history of this comment to the corresponding git pushes and commits.

> We are planning at some point to use the Checks API from GH which would make things more clear.

Do you know some other (unrelated) project(s) to look at that already use this API?

Marc

Status is actually available as a comment in the PR. We are planning at some point to use the Checks API from GH which would make things more clear.

The scripts now provide the output so you do not have to look at the actual CI log which might be verbose and misleading. If any information is missing in the comments posted by CI, please let us know.

Anas

Hi Björn,

I asked this question on Slack some time back and Anas answered it's not possible (yet?) for various reasons, one of them related to github accounts and security.

Fortunately something even more useful is already possible: running the checks yourself. Just run this compliance_script.py script:

https://github.com/zephyrproject-rtos/ci-tools/tree/master/scripts

https://github.com/zephyrproject-rtos/ci-tools/pulls

I know run this script *before* uploading anything to github as it obviously saves round trips and time.

You also need the "sanitycheck" script in the other, main repo.

I think sanitycheck is mentioned in the online documentation and the check_compliance about to be.

On a related CI topic, here's why the code coverage report is very often bogus:

https://github.com/codecov/codecov-bash/issues/83 thumbs up to "vote" for it

Cheers,

Marc

Hi,

As a funny coincidence I experienced three (!) CI issues relevant to this discussion today:

1. Some one-line "typo" fix failed to link some C code for a totally unrelated reason. Shippable seems to be the only CI element that provides complete logs. So I could instantly find the error in the logs and the next second someone on Slack pointed me at the candidate (and unrelated) fix. Searching github issues for "zephyr_write" also finds the candidate fix instantly, yay! [*]

2. As the opposite experience, I also got for another work in progress / hack / draft PR:

     Documentation build with 'make htmldocs' failed.

... and that's it; no logs or error message whatsoever, just this line. I have some guesses of what fails in CI while everything works for me (it is a documentation _hack_) but zero information on what exactly or how.

3. In the same draft PR as 2. I "forgot" a license header in a temporary file.  While it does not provide the command and log, the zephyrbot told me in very nice terms what was wrong. What's still confusing however is why the zephyr-ci-tools/scripts/check_compliance.py [-m Licence] script which I did run before pushing keeps telling me everything is fine and didn't save me the round trip time (and embarrassment). A typo would also cost (another) round-trip. This script really looks like the check zephyrbot runs but... it's apparently not or not quite?

Any layer of indirection in CI creates distance between developers and CI and makes developers care less about failures that don't affect them directly. Developers are not "consumers" and the vast majority seems to prefer having too much information than too little and solve problems by themselves but if the only option they have is not to care about some failures then they don't.

There may be reasons for not all CI details to be available and/or easy to reproduce but they don't apply to stuff like building docs or license checks or other simple sanitycheck.

1. https://app.shippable.com/github/zephyrproject-rtos/zephyr/runs/35577

2. & 3. https://github.com/zephyrproject-rtos/zephyr/pull/13805

[*] "yay!"... except it's not clear why tests run by Shippable are not deterministic: https://github.com/zephyrproject-rtos/zephyr/pull/13803 randconfig maybe?  I digress.

Marc

Hi,

A couple things I find confusing:

- There's only the ("beautified") output but not the commands that produced them as is typical in CI, so no obvious indication of what to do to test the next git push and make sure it will actually fix the issues before pushing again.

- The comments with the output are typically some distance away from the red/green "traffic lights" with unrelated stuff in between

- The "Details" buttons on the other hand *are* in the traffic lights and seem really designed to point at the output, I think most people expect them to but they point at the documentation instead.

> you do not have to look at the actual CI log which might be verbose and misleading

Well (most of) the actual CI log is or should be what one gets anyway when running the sanitycheck / compliance.py / etc. scripts locally so if that output is confusing then users will just submit patches to make it better (as I just did) and keep everything consistent.

The less difference and distance between users and CI, the better.

> The actual output is always available in a comment posted by zephyrbot in the same PR. The comment is updated for every run

It's interesting that this comment preserves some review history unlike the aggressive way the "traffic lights" in particular and github in general rewrite and bury review history. So this is Good but unfortunately inconsistent with (Bad) github so a bit confusing again. For instance it's difficult to relate the history of this comment to the corresponding git pushes and commits.

> We are planning at some point to use the Checks API from GH which would make things more clear.

Do you know some other (unrelated) project(s) to look at that already use this API?

Marc

Status is actually available as a comment in the PR. We are planning at some point to use the Checks API from GH which would make things more clear.

The scripts now provide the output so you do not have to look at the actual CI log which might be verbose and misleading. If any information is missing in the comments posted by CI, please let us know.

Anas

Hi Björn,

I asked this question on Slack some time back and Anas answered it's not possible (yet?) for various reasons, one of them related to github accounts and security.

Fortunately something even more useful is already possible: running the checks yourself. Just run this compliance_script.py script:

https://github.com/zephyrproject-rtos/ci-tools/tree/master/scripts

https://github.com/zephyrproject-rtos/ci-tools/pulls

I know run this script *before* uploading anything to github as it obviously saves round trips and time.

You also need the "sanitycheck" script in the other, main repo.

I think sanitycheck is mentioned in the online documentation and the check_compliance about to be.

On a related CI topic, here's why the code coverage report is very often bogus:

https://github.com/codecov/codecov-bash/issues/83 thumbs up to "vote" for it

Cheers,

Marc