Topics

Seeing and running what CI does (was: CODEOWNERS check in CI)


Marc Herbert
 

Hi Björn,

 

I asked this question on Slack some time back and Anas answered it's not possible (yet?) for various reasons, one of them related to github accounts and security.

 

Fortunately something even more useful is already possible: running the checks yourself. Just run this compliance_script.py script:

https://github.com/zephyrproject-rtos/ci-tools/tree/master/scripts

https://github.com/zephyrproject-rtos/ci-tools/pulls

 

I know run this script *before* uploading anything to github as it obviously saves round trips and time.

You also need the "sanitycheck" script in the other, main repo.

I think sanitycheck is mentioned in the online documentation and the check_compliance about to be.

 

On a related CI topic, here's why the code coverage report is very often bogus:

https://github.com/codecov/codecov-bash/issues/83 thumbs up to "vote" for it

 

Cheers,

 

Marc

 

 

From: <devel@...> on behalf of Stenberg <bjorn@...>
Date: Monday, 25 February 2019 at 00:09
To: "devel@..." <devel@...>
Subject: Re: [Zephyr-devel] CODEOWNERS check in CI

 

Can we please make the "Details" links for these checks show the actual output of the check command, rather than the current generic information about why the check exists?

 

--

 


Nashif, Anas
 

Status is actually available as a comment in the PR. We are planning at some point to use the Checks API from GH which would make things more clear.

The scripts now provide the output so you do not have to look at the actual CI log which might be verbose and misleading. If any information is missing in the comments posted by CI, please let us know.

 

Anas

 

From: <devel@...> on behalf of Marc Herbert <marc.herbert@...>
Date: Monday, 25 February 2019 at 17:28
To: Björn Stenberg <bjorn@...>, "devel@..." <devel@...>
Subject: [Zephyr-devel] Seeing and running what CI does (was: CODEOWNERS check in CI)

 

Hi Björn,

 

I asked this question on Slack some time back and Anas answered it's not possible (yet?) for various reasons, one of them related to github accounts and security.

 

Fortunately something even more useful is already possible: running the checks yourself. Just run this compliance_script.py script:

https://github.com/zephyrproject-rtos/ci-tools/tree/master/scripts

https://github.com/zephyrproject-rtos/ci-tools/pulls

 

I know run this script *before* uploading anything to github as it obviously saves round trips and time.

You also need the "sanitycheck" script in the other, main repo.

I think sanitycheck is mentioned in the online documentation and the check_compliance about to be.

 

On a related CI topic, here's why the code coverage report is very often bogus:

https://github.com/codecov/codecov-bash/issues/83 thumbs up to "vote" for it

 

Cheers,

 

Marc

 

 

From: <devel@...> on behalf of Stenberg <bjorn@...>
Date: Monday, 25 February 2019 at 00:09
To: "devel@..." <devel@...>
Subject: Re: [Zephyr-devel] CODEOWNERS check in CI

 

Can we please make the "Details" links for these checks show the actual output of the check command, rather than the current generic information about why the check exists?

 

--

 


Marc Herbert
 

Hi,

 

A couple things I find confusing:

- There's only the ("beautified") output but not the commands that produced them as is typical in CI, so no obvious indication of what to do to test the next git push and make sure it will actually fix the issues before pushing again.

- The comments with the output are typically some distance away from the red/green "traffic lights" with unrelated stuff in between

- The "Details" buttons on the other hand *are* in the traffic lights and seem really designed to point at the output, I think most people expect them to but they point at the documentation instead.

 

> you do not have to look at the actual CI log which might be verbose and misleading

 

Well (most of) the actual CI log is or should be what one gets anyway when running the sanitycheck / compliance.py / etc. scripts locally so if that output is confusing then users will just submit patches to make it better (as I just did) and keep everything consistent.

 

The less difference and distance between users and CI, the better.

 

> The actual output is always available in a comment posted by zephyrbot in the same PR. The comment is updated for every run

 

It's interesting that this comment preserves some review history unlike the aggressive way the "traffic lights" in particular and github in general rewrite and bury review history. So this is Good but unfortunately inconsistent with (Bad) github so a bit confusing again. For instance it's difficult to relate the history of this comment to the corresponding git pushes and commits.

 

> We are planning at some point to use the Checks API from GH which would make things more clear.

 

Do you know some other (unrelated) project(s) to look at that already use this API?

 

Marc

 

 

From: "Nashif, Anas" <anas.nashif@...>
Date: Monday, 25 February 2019 at 17:18
To: Marc Herbert <marc.herbert@...>, Björn Stenberg <bjorn@...>, "devel@..." <devel@...>
Subject: Re: [Zephyr-devel] Seeing and running what CI does (was: CODEOWNERS check in CI)

 

Status is actually available as a comment in the PR. We are planning at some point to use the Checks API from GH which would make things more clear.

The scripts now provide the output so you do not have to look at the actual CI log which might be verbose and misleading. If any information is missing in the comments posted by CI, please let us know.

 

Anas

 

From: <devel@...> on behalf of Marc Herbert <marc.herbert@...>
Date: Monday, 25 February 2019 at 17:28
To: Björn Stenberg <bjorn@...>, "devel@..." <devel@...>
Subject: [Zephyr-devel] Seeing and running what CI does (was: CODEOWNERS check in CI)

 

Hi Björn,

 

I asked this question on Slack some time back and Anas answered it's not possible (yet?) for various reasons, one of them related to github accounts and security.

 

Fortunately something even more useful is already possible: running the checks yourself. Just run this compliance_script.py script:

https://github.com/zephyrproject-rtos/ci-tools/tree/master/scripts

https://github.com/zephyrproject-rtos/ci-tools/pulls

 

I know run this script *before* uploading anything to github as it obviously saves round trips and time.

You also need the "sanitycheck" script in the other, main repo.

I think sanitycheck is mentioned in the online documentation and the check_compliance about to be.

 

On a related CI topic, here's why the code coverage report is very often bogus:

https://github.com/codecov/codecov-bash/issues/83 thumbs up to "vote" for it

 

Cheers,

 

Marc

 

 

From: <devel@...> on behalf of Stenberg <bjorn@...>
Date: Monday, 25 February 2019 at 00:09
To: "devel@..." <devel@...>
Subject: Re: [Zephyr-devel] CODEOWNERS check in CI

 

Can we please make the "Details" links for these checks show the actual output of the check command, rather than the current generic information about why the check exists?

 

--

 


Marc Herbert
 

Hi,

 

As a funny coincidence I experienced three (!) CI issues relevant to this discussion today:

 

1. Some one-line "typo" fix failed to link some C code for a totally unrelated reason. Shippable seems to be the only CI element that provides complete logs. So I could instantly find the error in the logs and the next second someone on Slack pointed me at the candidate (and unrelated) fix. Searching github issues for "zephyr_write" also finds the candidate fix instantly, yay! [*]

 

2. As the opposite experience, I also got for another work in progress / hack / draft PR:

     Documentation build with 'make htmldocs' failed.

... and that's it; no logs or error message whatsoever, just this line. I have some guesses of what fails in CI while everything works for me (it is a documentation _hack_) but zero information on what exactly or how.

 

3. In the same draft PR as 2. I "forgot" a license header in a temporary file.  While it does not provide the command and log, the zephyrbot told me in very nice terms what was wrong. What's still confusing however is why the zephyr-ci-tools/scripts/check_compliance.py [-m Licence] script which I did run before pushing keeps telling me everything is fine and didn't save me the round trip time (and embarrassment). A typo would also cost (another) round-trip. This script really looks like the check zephyrbot runs but... it's apparently not or not quite?

 

 

Any layer of indirection in CI creates distance between developers and CI and makes developers care less about failures that don't affect them directly. Developers are not "consumers" and the vast majority seems to prefer having too much information than too little and solve problems by themselves but if the only option they have is not to care about some failures then they don't.

 

There may be reasons for not all CI details to be available and/or easy to reproduce but they don't apply to stuff like building docs or license checks or other simple sanitycheck.

 

 

1. https://app.shippable.com/github/zephyrproject-rtos/zephyr/runs/35577

2. & 3. https://github.com/zephyrproject-rtos/zephyr/pull/13805

 

[*] "yay!"... except it's not clear why tests run by Shippable are not deterministic: https://github.com/zephyrproject-rtos/zephyr/pull/13803 randconfig maybe?  I digress.

 

 

Marc

 

 

From: Marc Herbert <marc.herbert@...>
Date: Monday, 25 February 2019 at 22:28
To: "Nashif, Anas" <anas.nashif@...>, Björn Stenberg <bjorn@...>, "devel@..." <devel@...>
Subject: Re: [Zephyr-devel] Seeing and running what CI does (was: CODEOWNERS check in CI)

 

Hi,

 

A couple things I find confusing:

- There's only the ("beautified") output but not the commands that produced them as is typical in CI, so no obvious indication of what to do to test the next git push and make sure it will actually fix the issues before pushing again.

- The comments with the output are typically some distance away from the red/green "traffic lights" with unrelated stuff in between

- The "Details" buttons on the other hand *are* in the traffic lights and seem really designed to point at the output, I think most people expect them to but they point at the documentation instead.

 

> you do not have to look at the actual CI log which might be verbose and misleading

 

Well (most of) the actual CI log is or should be what one gets anyway when running the sanitycheck / compliance.py / etc. scripts locally so if that output is confusing then users will just submit patches to make it better (as I just did) and keep everything consistent.

 

The less difference and distance between users and CI, the better.

 

> The actual output is always available in a comment posted by zephyrbot in the same PR. The comment is updated for every run

 

It's interesting that this comment preserves some review history unlike the aggressive way the "traffic lights" in particular and github in general rewrite and bury review history. So this is Good but unfortunately inconsistent with (Bad) github so a bit confusing again. For instance it's difficult to relate the history of this comment to the corresponding git pushes and commits.

 

> We are planning at some point to use the Checks API from GH which would make things more clear.

 

Do you know some other (unrelated) project(s) to look at that already use this API?

 

Marc

 

 

From: "Nashif, Anas" <anas.nashif@...>
Date: Monday, 25 February 2019 at 17:18
To: Marc Herbert <marc.herbert@...>, Björn Stenberg <bjorn@...>, "devel@..." <devel@...>
Subject: Re: [Zephyr-devel] Seeing and running what CI does (was: CODEOWNERS check in CI)

 

Status is actually available as a comment in the PR. We are planning at some point to use the Checks API from GH which would make things more clear.

The scripts now provide the output so you do not have to look at the actual CI log which might be verbose and misleading. If any information is missing in the comments posted by CI, please let us know.

 

Anas

 

From: <devel@...> on behalf of Marc Herbert <marc.herbert@...>
Date: Monday, 25 February 2019 at 17:28
To: Björn Stenberg <bjorn@...>, "devel@..." <devel@...>
Subject: [Zephyr-devel] Seeing and running what CI does (was: CODEOWNERS check in CI)

 

Hi Björn,

 

I asked this question on Slack some time back and Anas answered it's not possible (yet?) for various reasons, one of them related to github accounts and security.

 

Fortunately something even more useful is already possible: running the checks yourself. Just run this compliance_script.py script:

https://github.com/zephyrproject-rtos/ci-tools/tree/master/scripts

https://github.com/zephyrproject-rtos/ci-tools/pulls

 

I know run this script *before* uploading anything to github as it obviously saves round trips and time.

You also need the "sanitycheck" script in the other, main repo.

I think sanitycheck is mentioned in the online documentation and the check_compliance about to be.

 

On a related CI topic, here's why the code coverage report is very often bogus:

https://github.com/codecov/codecov-bash/issues/83 thumbs up to "vote" for it

 

Cheers,

 

Marc

 

 

From: <devel@...> on behalf of Stenberg <bjorn@...>
Date: Monday, 25 February 2019 at 00:09
To: "devel@..." <devel@...>
Subject: Re: [Zephyr-devel] CODEOWNERS check in CI

 

Can we please make the "Details" links for these checks show the actual output of the check command, rather than the current generic information about why the check exists?

 

--

 


Marc Herbert
 

Quick update.



1. [*] "yay!"... except it's not clear why tests run by Shippable are not deterministic: https://github.com/zephyrproject-rtos/zephyr/pull/13803 randconfig maybe? I digress.
Non-determinism found and explained: https://github.com/zephyrproject-rtos/zephyr/pull/13973

Thanks to the reviewers for the very quick reviews.



2. As the opposite experience, I also got for another work in progress / hack / draft PR:
Documentation build with 'make htmldocs' failed.
... and that's it; no logs or error message whatsoever, just this line. I have some guesses of what fails in CI while everything works for me (it is a documentation _hack_) but zero information on what exactly or how.
Still unclear.


3. In the same draft PR as 2. I "forgot" a license header in a temporary file. While it does not provide the command and log, the zephyrbot told me in very nice terms what was wrong. What's still confusing however is why the zephyr-ci-tools/scripts/check_compliance.py [-m Licence] script which I did run before pushing keeps telling me everything is fine and didn't save me the round trip time (and embarrassment). A typo would also cost (another) round-trip. This script really looks like the check zephyrbot runs but... it's apparently not or not quite?
Mystery solved: https://github.com/zephyrproject-rtos/ci-tools/pull/25


Another mystery being fixed by Ulf: CODEOWNERS failing locally but not in CI
https://github.com/zephyrproject-rtos/ci-tools/pull/23
https://github.com/zephyrproject-rtos/zephyr/pull/13978




Any layer of indirection in CI creates distance between developers and CI and makes developers care less about failures that don't affect them directly. Developers are not "consumers" and the vast majority seems to prefer having too much information than too little and solve problems by themselves but if the only option they have is not to care about some failures then they don't.

There may be reasons for not all CI details to be available and/or easy to reproduce but they don't apply to stuff like building docs or license checks or other simple sanitycheck.


1. https://app.shippable.com/github/zephyrproject-rtos/zephyr/runs/35577
2. & 3. https://github.com/zephyrproject-rtos/zephyr/pull/13805



Nashif, Anas
 

2. As the opposite experience, I also got for another work in progress / hack / draft PR:
>> Documentation build with 'make htmldocs' failed.
>> ... and that's it; no logs or error message whatsoever, just this line. I have some guesses of what fails in CI while everything works for me (it is a documentation _hack_) but zero information on what exactly or how.
>>

> Still unclear.

A python exception happens in this case, you can always reproduce it locally by running 'make htmldocs'. This used to generate no reports at all previously, so progress. I am not sure how useful would it be putting the python exception in the comments.

Anas

On 01/03/2019, 20:03, "Herbert, Marc" <marc.herbert@...> wrote:

Quick update.



> 1. [*] "yay!"... except it's not clear why tests run by Shippable are not deterministic: https://github.com/zephyrproject-rtos/zephyr/pull/13803 randconfig maybe? I digress.

Non-determinism found and explained: https://github.com/zephyrproject-rtos/zephyr/pull/13973

Thanks to the reviewers for the very quick reviews.



> 2. As the opposite experience, I also got for another work in progress / hack / draft PR:
> Documentation build with 'make htmldocs' failed.
> ... and that's it; no logs or error message whatsoever, just this line. I have some guesses of what fails in CI while everything works for me (it is a documentation _hack_) but zero information on what exactly or how.
>

Still unclear.


> 3. In the same draft PR as 2. I "forgot" a license header in a temporary file. While it does not provide the command and log, the zephyrbot told me in very nice terms what was wrong. What's still confusing however is why the zephyr-ci-tools/scripts/check_compliance.py [-m Licence] script which I did run before pushing keeps telling me everything is fine and didn't save me the round trip time (and embarrassment). A typo would also cost (another) round-trip. This script really looks like the check zephyrbot runs but... it's apparently not or not quite?
>

Mystery solved: https://github.com/zephyrproject-rtos/ci-tools/pull/25


Another mystery being fixed by Ulf: CODEOWNERS failing locally but not in CI
https://github.com/zephyrproject-rtos/ci-tools/pull/23
https://github.com/zephyrproject-rtos/zephyr/pull/13978



>
> Any layer of indirection in CI creates distance between developers and CI and makes developers care less about failures that don't affect them directly. Developers are not "consumers" and the vast majority seems to prefer having too much information than too little and solve problems by themselves but if the only option they have is not to care about some failures then they don't.
>
> There may be reasons for not all CI details to be available and/or easy to reproduce but they don't apply to stuff like building docs or license checks or other simple sanitycheck.
>
>
> 1. https://app.shippable.com/github/zephyrproject-rtos/zephyr/runs/35577
> 2. & 3. https://github.com/zephyrproject-rtos/zephyr/pull/13805
>
>
>


Marc Herbert
 

On 3 Mar 2019, at 09:09, Nashif, Anas <anas.nashif@...> wrote:

2. As the opposite experience, I also got for another work in progress / hack / draft PR:
Documentation build with 'make htmldocs' failed.
... and that's it; no logs or error message whatsoever, just this line. I have some guesses of what fails in CI while everything works for me (it is a documentation _hack_) but zero information on what exactly or how.
2. & 3. https://github.com/zephyrproject-rtos/zephyr/pull/13805

Still unclear.

A python exception happens in this case, you can always reproduce it locally by running 'make htmldocs'.

As I wrote above "make htmldocs" works for me, this PR is actually
an htmldocs demo.


I am not sure how useful would it be putting the python exception...


It would point at the difference(s) between my environment and
CI = the vast majority of CI failures when testing locally and
thoroughly first.

Again I do have a couple of educated guesses about these differences
(help NOT desired here yet) but they're just that: guesses. No proof.

... in the comments.

Not in the comments but behind some kind of "Details" button with all
the rest of the logs and commands that produced them; exactly like
Shippable (and only Shippable now) does.

Having some smart "error analyzer" summarizing in the github comments
how it understood the unexpected failure would be cool. However it's not
mutually exclusive with sharing complete logs and commands as most
successful CIs do.