Re: Seeing and running what CI does (was: CODEOWNERS check in CI)


Marc Herbert
 

Hi,

 

A couple things I find confusing:

- There's only the ("beautified") output but not the commands that produced them as is typical in CI, so no obvious indication of what to do to test the next git push and make sure it will actually fix the issues before pushing again.

- The comments with the output are typically some distance away from the red/green "traffic lights" with unrelated stuff in between

- The "Details" buttons on the other hand *are* in the traffic lights and seem really designed to point at the output, I think most people expect them to but they point at the documentation instead.

 

> you do not have to look at the actual CI log which might be verbose and misleading

 

Well (most of) the actual CI log is or should be what one gets anyway when running the sanitycheck / compliance.py / etc. scripts locally so if that output is confusing then users will just submit patches to make it better (as I just did) and keep everything consistent.

 

The less difference and distance between users and CI, the better.

 

> The actual output is always available in a comment posted by zephyrbot in the same PR. The comment is updated for every run

 

It's interesting that this comment preserves some review history unlike the aggressive way the "traffic lights" in particular and github in general rewrite and bury review history. So this is Good but unfortunately inconsistent with (Bad) github so a bit confusing again. For instance it's difficult to relate the history of this comment to the corresponding git pushes and commits.

 

> We are planning at some point to use the Checks API from GH which would make things more clear.

 

Do you know some other (unrelated) project(s) to look at that already use this API?

 

Marc

 

 

From: "Nashif, Anas" <anas.nashif@...>
Date: Monday, 25 February 2019 at 17:18
To: Marc Herbert <marc.herbert@...>, Björn Stenberg <bjorn@...>, "devel@..." <devel@...>
Subject: Re: [Zephyr-devel] Seeing and running what CI does (was: CODEOWNERS check in CI)

 

Status is actually available as a comment in the PR. We are planning at some point to use the Checks API from GH which would make things more clear.

The scripts now provide the output so you do not have to look at the actual CI log which might be verbose and misleading. If any information is missing in the comments posted by CI, please let us know.

 

Anas

 

From: <devel@...> on behalf of Marc Herbert <marc.herbert@...>
Date: Monday, 25 February 2019 at 17:28
To: Björn Stenberg <bjorn@...>, "devel@..." <devel@...>
Subject: [Zephyr-devel] Seeing and running what CI does (was: CODEOWNERS check in CI)

 

Hi Björn,

 

I asked this question on Slack some time back and Anas answered it's not possible (yet?) for various reasons, one of them related to github accounts and security.

 

Fortunately something even more useful is already possible: running the checks yourself. Just run this compliance_script.py script:

https://github.com/zephyrproject-rtos/ci-tools/tree/master/scripts

https://github.com/zephyrproject-rtos/ci-tools/pulls

 

I know run this script *before* uploading anything to github as it obviously saves round trips and time.

You also need the "sanitycheck" script in the other, main repo.

I think sanitycheck is mentioned in the online documentation and the check_compliance about to be.

 

On a related CI topic, here's why the code coverage report is very often bogus:

https://github.com/codecov/codecov-bash/issues/83 thumbs up to "vote" for it

 

Cheers,

 

Marc

 

 

From: <devel@...> on behalf of Stenberg <bjorn@...>
Date: Monday, 25 February 2019 at 00:09
To: "devel@..." <devel@...>
Subject: Re: [Zephyr-devel] CODEOWNERS check in CI

 

Can we please make the "Details" links for these checks show the actual output of the check command, rather than the current generic information about why the check exists?

 

--

 

Join devel@lists.zephyrproject.org to automatically receive all group messages.