[Zephyr-users] Introducing west, Zephyr's meta-tool


Luiz Augusto von Dentz
 

Hi Carles,

How about pull request, how does west deals with them? Or it doesn't
and which case we have to deal with individual git repositories and
issue git push ourselves? This looks like repo which I never liked.

On Mon, Jan 28, 2019 at 10:43 AM Cufi, Carles <carles.cufi@nordicsemi.no> wrote:

Hi all,

For a few months now we have been working on a meta-tool named west, designed to act both as a command-line tool to build, flash an debug Zephyr-based applications and also to manage the additional Git repositories that Zephyr will require in order to build in the near future.

West is written in Python 3, lives in its own repository [1] and its bootstrapper is available on PyPi [2].
As soon as we merge the topic-west [3] branch into master (which should happen in the next few days) you will need to install west in order to flash and debug, since those commands will rely on havig west present as part of your Zephyr workflow.

The updated west documentation will be online in the Zephyr documentation website [4] as soon as the topic-west branch is merged into master. In the meantime you can also look at the unrendered documentation sources in the topic-west branch [5] and the existing GitHub issues [6][7].

In order to install the west boostrapper (which you can do today) run:

# On Linux
pip3 install --user west

# On macOS and Windows
pip3 install west

Once the west bootstrapper is installed and the topic-west branch is merged into master, you will have 2 choices to get a west main installation.

1. Clone a new copy of the zephyr repository using west:

west init zephyrproject
cd zephyrproject
west update

2. Install west around an existing local zephyr clone:

Move the cloned zephyr/ repository to an empty enclosing folder (for example zephyrproject/zephyr/), and from this enclosing folder zephyrproject/ run:

west init -l zephyr/
west update
The -l <path to zephyr> parameter instructs west to use an existing local copy instead of cloning a remote repository.

Feedback and patches are welcome, but note that the feature will likely be merged as-is since we consider we have left ample time to raise any major issues regarding the workflow and decisions taken during the development of west. This is however only the beginning of west's (hopefully) long existence as the Zephyr command-line meta-tool, and it will certainly evolve and change considerably in the future with the help of the community.

Please submit feedback in the form of GitHub issues on the west repository itself [1] unless it affects features or functionality in the zephyr repository directly.

I will reply to this email once the topic-west branch is merged into master.

Thank you to everybody who has contributed so far!

Regards,

Carles

[1] https://github.com/zephyrproject-rtos/west
[2] https://pypi.org/project/west/
[3] https://github.com/zephyrproject-rtos/zephyr/tree/topic-west
[4] https://docs.zephyrproject.org/latest/tools/west/index.html
[5] https://github.com/zephyrproject-rtos/zephyr/tree/topic-west/doc/tools/west
[6] https://github.com/zephyrproject-rtos/zephyr/issues/6205
[7] https://github.com/zephyrproject-rtos/zephyr/issues/6770


--
Luiz Augusto von Dentz


Carles Cufi
 

Hi Luiz,

-----Original Message-----
From: devel@lists.zephyrproject.org <devel@lists.zephyrproject.org> On
Behalf Of Luiz Augusto von Dentz
Sent: 28 January 2019 21:00
Subject: Re: [Zephyr-devel] [Zephyr-users] Introducing west, Zephyr's
meta-tool

Hi Carles,

How about pull request, how does west deals with them? Or it doesn't and
which case we have to deal with individual git repositories and issue
git push ourselves? This looks like repo which I never liked.
You can see that dealing with Pull Requests is listed as an unresolved issue in the description here:
https://github.com/zephyrproject-rtos/zephyr/issues/6770

There are several possibilities to deal with this, but first and since that you mention Google repo let me make an important distinction. With repo you typically (though not always) track the tip of master on multiple repositories. With west, although that is also possible, we will require specific SHAs for all repos (projects) that are not the main zephyr repo (i.e. the manifest repo).

Options we have (informally, in calls and chats) considered:
1. Send 2 PRs to the 2 repos affected (say for example zephyr + mbedtls), where the branch name used matches. Then CI running in zephyr could look at all repositories listed in the manifest and check if an outstanding PR for it exists with the same GitHub username and branch name.
2. Send the PR to the external project (mbedtls in our example) and another PR to zephyr with the manifest modified so that it points to the fork and branch name of the project's PR. When those are merged, CI could then replace the contents of the manifest with the proper SHA.
3. Introduce some sort of per-PR ChangeId that CI knows how to parse.
4. GitHub adds support for cross-repository PRs. There have been multiple hints that this is coming, but obviously we cannot know when, if ever, this feature will be available.

Note that we have not at any point considered moving away from GitHub. The plan is to adapt to what the service offers and try to find a solution that works well with it. This is also one of the reasons we are first introducing west without actually splitting the ext/ folder into multiple external repos: we want to expose users to west before we start doing more radical changes.

Ideas and feedback welcome!

Thanks,

Carles


On Mon, Jan 28, 2019 at 10:43 AM Cufi, Carles
<carles.cufi@nordicsemi.no> wrote:

Hi all,

For a few months now we have been working on a meta-tool named west,
designed to act both as a command-line tool to build, flash an debug
Zephyr-based applications and also to manage the additional Git
repositories that Zephyr will require in order to build in the near
future.

West is written in Python 3, lives in its own repository [1] and its
bootstrapper is available on PyPi [2].
As soon as we merge the topic-west [3] branch into master (which
should happen in the next few days) you will need to install west in
order to flash and debug, since those commands will rely on havig west
present as part of your Zephyr workflow.

The updated west documentation will be online in the Zephyr
documentation website [4] as soon as the topic-west branch is merged
into master. In the meantime you can also look at the unrendered
documentation sources in the topic-west branch [5] and the existing
GitHub issues [6][7].

In order to install the west boostrapper (which you can do today) run:

# On Linux
pip3 install --user west

# On macOS and Windows
pip3 install west

Once the west bootstrapper is installed and the topic-west branch is
merged into master, you will have 2 choices to get a west main
installation.

1. Clone a new copy of the zephyr repository using west:

west init zephyrproject
cd zephyrproject
west update

2. Install west around an existing local zephyr clone:

Move the cloned zephyr/ repository to an empty enclosing folder (for
example zephyrproject/zephyr/), and from this enclosing folder
zephyrproject/ run:

west init -l zephyr/
west update
The -l <path to zephyr> parameter instructs west to use an existing
local copy instead of cloning a remote repository.

Feedback and patches are welcome, but note that the feature will
likely be merged as-is since we consider we have left ample time to
raise any major issues regarding the workflow and decisions taken during
the development of west. This is however only the beginning of west's
(hopefully) long existence as the Zephyr command-line meta-tool, and it
will certainly evolve and change considerably in the future with the
help of the community.

Please submit feedback in the form of GitHub issues on the west
repository itself [1] unless it affects features or functionality in the
zephyr repository directly.

I will reply to this email once the topic-west branch is merged into
master.

Thank you to everybody who has contributed so far!

Regards,

Carles

[1] https://github.com/zephyrproject-rtos/west
[2] https://pypi.org/project/west/
[3] https://github.com/zephyrproject-rtos/zephyr/tree/topic-west
[4] https://docs.zephyrproject.org/latest/tools/west/index.html
[5]
https://github.com/zephyrproject-rtos/zephyr/tree/topic-west/doc/tools
/west [6] https://github.com/zephyrproject-rtos/zephyr/issues/6205
[7] https://github.com/zephyrproject-rtos/zephyr/issues/6770



--
Luiz Augusto von Dentz


Luiz Augusto von Dentz
 

Hi Carles,

On Mon, Jan 28, 2019 at 10:54 PM Cufi, Carles <Carles.Cufi@nordicsemi.no> wrote:

Hi Luiz,

-----Original Message-----
From: devel@lists.zephyrproject.org <devel@lists.zephyrproject.org> On
Behalf Of Luiz Augusto von Dentz
Sent: 28 January 2019 21:00
Subject: Re: [Zephyr-devel] [Zephyr-users] Introducing west, Zephyr's
meta-tool

Hi Carles,

How about pull request, how does west deals with them? Or it doesn't and
which case we have to deal with individual git repositories and issue
git push ourselves? This looks like repo which I never liked.
You can see that dealing with Pull Requests is listed as an unresolved issue in the description here:
https://github.com/zephyrproject-rtos/zephyr/issues/6770

There are several possibilities to deal with this, but first and since that you mention Google repo let me make an important distinction. With repo you typically (though not always) track the tip of master on multiple repositories. With west, although that is also possible, we will require specific SHAs for all repos (projects) that are not the main zephyr repo (i.e. the manifest repo).

Options we have (informally, in calls and chats) considered:
1. Send 2 PRs to the 2 repos affected (say for example zephyr + mbedtls), where the branch name used matches. Then CI running in zephyr could look at all repositories listed in the manifest and check if an outstanding PR for it exists with the same GitHub username and branch name.
2. Send the PR to the external project (mbedtls in our example) and another PR to zephyr with the manifest modified so that it points to the fork and branch name of the project's PR. When those are merged, CI could then replace the contents of the manifest with the proper SHA.
3. Introduce some sort of per-PR ChangeId that CI knows how to parse.
4. GitHub adds support for cross-repository PRs. There have been multiple hints that this is coming, but obviously we cannot know when, if ever, this feature will be available.
From these alternatives I only see 4 really working in practice, but
it is limited to repositories being in github in which case the PR
requests can perhaps be generated from a single clone and github could
then forward to each individual project but still tracking on the
original. For other options even if west learn how to properly
separate each PR the user still needs its own repository for each of
those if they are hosted in github which would have to be tracked
individually, or in case it is not hosted in github it may have to
deal with mailing list, gerrit, etc.

Note that we have not at any point considered moving away from GitHub. The plan is to adapt to what the service offers and try to find a solution that works well with it. This is also one of the reasons we are first introducing west without actually splitting the ext/ folder into multiple external repos: we want to expose users to west before we start doing more radical changes.
Well good, though Im not sure what problem does west solves then? For
pulling it sounds like submodules is the preferable solution (to linux
folks at least):

https://public-inbox.org/git/338901fc-6dc8-2684-c116-393e603f85e9@suse.de/t/

Ideas and feedback welcome!

Thanks,

Carles


On Mon, Jan 28, 2019 at 10:43 AM Cufi, Carles
<carles.cufi@nordicsemi.no> wrote:

Hi all,

For a few months now we have been working on a meta-tool named west,
designed to act both as a command-line tool to build, flash an debug
Zephyr-based applications and also to manage the additional Git
repositories that Zephyr will require in order to build in the near
future.

West is written in Python 3, lives in its own repository [1] and its
bootstrapper is available on PyPi [2].
As soon as we merge the topic-west [3] branch into master (which
should happen in the next few days) you will need to install west in
order to flash and debug, since those commands will rely on havig west
present as part of your Zephyr workflow.

The updated west documentation will be online in the Zephyr
documentation website [4] as soon as the topic-west branch is merged
into master. In the meantime you can also look at the unrendered
documentation sources in the topic-west branch [5] and the existing
GitHub issues [6][7].

In order to install the west boostrapper (which you can do today) run:

# On Linux
pip3 install --user west

# On macOS and Windows
pip3 install west

Once the west bootstrapper is installed and the topic-west branch is
merged into master, you will have 2 choices to get a west main
installation.

1. Clone a new copy of the zephyr repository using west:

west init zephyrproject
cd zephyrproject
west update

2. Install west around an existing local zephyr clone:

Move the cloned zephyr/ repository to an empty enclosing folder (for
example zephyrproject/zephyr/), and from this enclosing folder
zephyrproject/ run:

west init -l zephyr/
west update
The -l <path to zephyr> parameter instructs west to use an existing
local copy instead of cloning a remote repository.

Feedback and patches are welcome, but note that the feature will
likely be merged as-is since we consider we have left ample time to
raise any major issues regarding the workflow and decisions taken during
the development of west. This is however only the beginning of west's
(hopefully) long existence as the Zephyr command-line meta-tool, and it
will certainly evolve and change considerably in the future with the
help of the community.

Please submit feedback in the form of GitHub issues on the west
repository itself [1] unless it affects features or functionality in the
zephyr repository directly.

I will reply to this email once the topic-west branch is merged into
master.

Thank you to everybody who has contributed so far!

Regards,

Carles

[1] https://github.com/zephyrproject-rtos/west
[2] https://pypi.org/project/west/
[3] https://github.com/zephyrproject-rtos/zephyr/tree/topic-west
[4] https://docs.zephyrproject.org/latest/tools/west/index.html
[5]
https://github.com/zephyrproject-rtos/zephyr/tree/topic-west/doc/tools
/west [6] https://github.com/zephyrproject-rtos/zephyr/issues/6205
[7] https://github.com/zephyrproject-rtos/zephyr/issues/6770



--
Luiz Augusto von Dentz


--
Luiz Augusto von Dentz


lairdjm
 

Hi Carles,
Note that we have not at any point considered moving away from GitHub. The plan is to adapt to what the service offers and try to find a solution that works well with it. This is also one of the reasons we are first introducing west without actually splitting the ext/ folder into multiple external repos: we want to expose users to west before we start doing more radical changes.
Well good, though Im not sure what problem does west solves then? For
pulling it sounds like submodules is the preferable solution (to linux
folks at least):
https://public-inbox.org/git/338901fc-6dc8-2684-c116-393e603f85e9@suse.de/t/
I agree with Luiz here, what does west provide that submodules doesn't provide? Why do I and others now need yet another piece of software to build Zephyr code (python)?
Thanks,
Jamie


Carles Cufi
 

Hi Jamie, Luiz,

-----Original Message-----
From: Jamie Mccrae <Jamie.Mccrae@lairdtech.com>
Sent: 29 January 2019 10:35
To: Luiz Augusto von Dentz <luiz.dentz@gmail.com>; Cufi, Carles
<Carles.Cufi@nordicsemi.no>
Cc: zephyr-devel <zephyr-devel@lists.zephyrproject.org>; zephyr-
users@lists.zephyrproject.org
Subject: RE: [Zephyr-devel] [Zephyr-users] Introducing west, Zephyr's
meta-tool

Hi Carles,
Note that we have not at any point considered moving away from
GitHub. The plan is to adapt to what the service offers and try to find
a solution that works well with it. This is also one of the reasons we
are first introducing west without actually splitting the ext/ folder
into multiple external repos: we want to expose users to west before we
start doing more radical changes.

Well good, though Im not sure what problem does west solves then? For
pulling it sounds like submodules is the preferable solution (to linux
folks at least):
https://public-inbox.org/git/338901fc-6dc8-2684-c116-393e603f85e9@suse
.de/t/
I agree with Luiz here, what does west provide that submodules doesn't
provide? Why do I and others now need yet another piece of software to
build Zephyr code (python)?
There are a number of reasons why we don't use Git submodules, all of which are described in the GitHub issue describing multi-repo:
https://github.com/zephyrproject-rtos/zephyr/issues/6770

I encourage you to read through the (admittedly quite long) list of comments there and continue the conversation.

Regards,

Carles


Luiz Augusto von Dentz
 

Hi Carles,
On Tue, Jan 29, 2019 at 11:40 AM Cufi, Carles <Carles.Cufi@nordicsemi.no> wrote:

Hi Jamie, Luiz,

-----Original Message-----
From: Jamie Mccrae <Jamie.Mccrae@lairdtech.com>
Sent: 29 January 2019 10:35
To: Luiz Augusto von Dentz <luiz.dentz@gmail.com>; Cufi, Carles
<Carles.Cufi@nordicsemi.no>
Cc: zephyr-devel <zephyr-devel@lists.zephyrproject.org>; zephyr-
users@lists.zephyrproject.org
Subject: RE: [Zephyr-devel] [Zephyr-users] Introducing west, Zephyr's
meta-tool

Hi Carles,
Note that we have not at any point considered moving away from
GitHub. The plan is to adapt to what the service offers and try to find
a solution that works well with it. This is also one of the reasons we
are first introducing west without actually splitting the ext/ folder
into multiple external repos: we want to expose users to west before we
start doing more radical changes.

Well good, though Im not sure what problem does west solves then? For
pulling it sounds like submodules is the preferable solution (to linux
folks at least):
https://public-inbox.org/git/338901fc-6dc8-2684-c116-393e603f85e9@suse
.de/t/
I agree with Luiz here, what does west provide that submodules doesn't
provide? Why do I and others now need yet another piece of software to
build Zephyr code (python)?
There are a number of reasons why we don't use Git submodules, all of which are described in the GitHub issue describing multi-repo:
https://github.com/zephyrproject-rtos/zephyr/issues/6770
Well I didn't have to go as far since someone already commented similar thing:

https://github.com/zephyrproject-rtos/zephyr/issues/6770#issuecomment-445496706

So attached vs detached approach, there seems to be quite many on the
camp that we need to have the repositories attached while some
considered the detached approach a must have. As individual
contributor I think using the attached approach is a must, specially
if one wants to actually run sanity checks which sometimes is
necessary e.g. when you are actually writing a test case which depends
on a feature from a different repository.

I encourage you to read through the (admittedly quite long) list of comments there and continue the conversation.

Regards,

Carles

--
Luiz Augusto von Dentz


Carles Cufi
 

Hi Luiz,

-----Original Message-----
From: Luiz Augusto von Dentz <luiz.dentz@gmail.com>
Sent: 29 January 2019 12:25
To: Cufi, Carles <Carles.Cufi@nordicsemi.no>
Cc: Jamie Mccrae <Jamie.Mccrae@lairdtech.com>; zephyr-devel <zephyr-
devel@lists.zephyrproject.org>; zephyr-users@lists.zephyrproject.org
Subject: Re: [Zephyr-devel] [Zephyr-users] Introducing west, Zephyr's
meta-tool

Hi Carles,
On Tue, Jan 29, 2019 at 11:40 AM Cufi, Carles
<Carles.Cufi@nordicsemi.no> wrote:

Hi Jamie, Luiz,

-----Original Message-----
From: Jamie Mccrae <Jamie.Mccrae@lairdtech.com>
Sent: 29 January 2019 10:35
To: Luiz Augusto von Dentz <luiz.dentz@gmail.com>; Cufi, Carles
<Carles.Cufi@nordicsemi.no>
Cc: zephyr-devel <zephyr-devel@lists.zephyrproject.org>; zephyr-
users@lists.zephyrproject.org
Subject: RE: [Zephyr-devel] [Zephyr-users] Introducing west,
Zephyr's meta-tool

Hi Carles,
Note that we have not at any point considered moving away from
GitHub. The plan is to adapt to what the service offers and try to
find a solution that works well with it. This is also one of the
reasons we are first introducing west without actually splitting the
ext/ folder into multiple external repos: we want to expose users to
west before we start doing more radical changes.

Well good, though Im not sure what problem does west solves then?
For pulling it sounds like submodules is the preferable solution
(to linux folks at least):
https://public-inbox.org/git/338901fc-6dc8-2684-c116-393e603f85e9@
suse
.de/t/
I agree with Luiz here, what does west provide that submodules
doesn't provide? Why do I and others now need yet another piece of
software to build Zephyr code (python)?
There are a number of reasons why we don't use Git submodules, all of
which are described in the GitHub issue describing multi-repo:
https://github.com/zephyrproject-rtos/zephyr/issues/6770
Well I didn't have to go as far since someone already commented similar
thing:

https://github.com/zephyrproject-rtos/zephyr/issues/6770#issuecomment-
445496706

So attached vs detached approach, there seems to be quite many on the
camp that we need to have the repositories attached while some
considered the detached approach a must have. As individual contributor
I think using the attached approach is a must, specially if one wants to
actually run sanity checks which sometimes is necessary e.g. when you
are actually writing a test case which depends on a feature from a
different repository.
If by "attached" you mean that projects in the manifest are locked at a particular SHA, then that is exactly what we will do upstream (https://github.com/zephyrproject-rtos/zephyr/blob/master/west.yml#L37). That is not to say that some downstreams may choose to free float on the tip of a branch, but this is *not* going to be the case for upstream Zephyr. I will edit the body of the issue to make that clear.

Regards,

Carles