Zephyr: TSC Meeting Minutes (7/15/20 - Virtual TSC F2F)

Brett Preston

Zephyr Project TSC - 15 July 2020


Join Microsoft Teams Meeting

+1 321-558-6518 United States, Orlando (Toll)

Conference ID: 435 743 999#

Local numbers | Reset PIN | Learn more about Teams | Meeting options

[In blue meeting notes, in purple post meeting addendums]

Voting TSC Members in Attendance [12 = quorum] - Quorum reached.

Voting TSC Member


Voting TSC Member


Voting TSC Member


Amber Hibberd


Ioannis Glaropoulos


Nate Graff


Anas Nashif


Johan Hedberg


Piotr Mienkowski


Asger Munk Nielsen 


Johann Fischer


Rasmus Abildgren


Carles Cufi


Kumar Gala (Kevin Townsend - alternate)


Ruud Derwig


Danny Ørndrup


Mark Grosen


Ryan Erickson


David Brown


Maureen Helm


Scott Shawcroft


David Eriksson (Joel Westerberg - alternate)


Michael Gielda (Peter Gielda - alternate)


Stefan Mijovic


David Leach


Michael Scott


Out of Office / Vacations

  • TSC Member Name - Dates

    • Stefan Mijovic: until July 29

    • Johan Hedberg: August 3 - August 23 


  • GitHub Permissions - Next Steps:

    • Map GitHub terms (code owner, member, collaborator, owner) to Zephyr Project charter terms (maintainer, committer, contributor)

    • Define process to grant and revoke merge rights and code ownership

  • Safety Committee [Amber]


  • Process Improvement Forum [Ioannis]

    • Why do we need?

      • Discuss how we do things in Zephyr?

        • Not a technical forum

        • Not specific to Release activities

        • How we “organize”

          • Ourselves

          • Our Processes

      • Submit proposals to introduce/modify existing procedures

    • Goals

      • Encourage people to contribute to Zephyr by making our process

        • Well-defined

        • More transparent

        • Visible to all

    • Activities 2020 H1

      • Defined Individual Roles and Responsibilities in the Project

        • Contributors’ levels

        • Release Team membership

        • Drafted a Zephyr Charter update to align with TSC Roles/Policies

      • Enhanced contribution workflow

        • Introduced Assignee role in pull-requests (to be enforced soon)

        • 2-approval policy for PRs

        • Defined process for escalation

        • Defined policies for amending contributions by other developers, and proper attribution

      • Introducing a Maintainers’ database (to appear soon)

        • Use this for facilitating PR reviewing and acceptance

    • Organizing the work with Zephyr Modules

      • Scope, structure, conventions

      • Policies for acceptance, deprecation, removal

      • Contribution workflow

    • In-Progress & Backlog

      • Finalize the draft proposal for Zephyr modules organization

        • Bring to TSC

        • Backlog

          • Improve communication & transparency during crunch times

          • Best practices to help maintainers

            • Assess the engagement and/or skills of contributors

            • Coaching less experienced developers

        • Always busy, but always looking for additional topics to discuss

    • Participate in Process WG: https://lists.zephyrproject.org/g/tsc/calendar

    • Zephyr Modules

      • [link to presentation]

      • Modules - Naming

      • Modules - Content & scope

        • HALs

        • Libraries

        • Tools

        • Applications (e.g. mcuboot)

      • Module porting layers

        • Zephyr module porting layers (Zephyr glue code)

          • Should be part of Zephyr code, brought in the tree

      • Contributing

        • Individual Roles & Responsibilities

          • Administration

            • Managing access to the module repositories

            • Role to be filled-in by administrators’ team

          • Code ownership

            • Each module should have an individual acting as an “owner”

          • Merging

      • Contribution workflow - Generic

        • Contribution guidelines

          • Reviews and pull requests

            • Following generic contribution guidelines

            • PRs always coupled with a main Zephyr repo PR with a manifest update for CI testing

          • Issue and bug reporting, labeling

            • Single place to report Issues/Bugs related to modules.

            • Enhance labeling for modules, with additional labels corresponding to each module, when applicable (e.g. not for HALs - there we have platform-vendor labels already)

            • Mirror module bugs (that have an upstream bug report) allowed. Should not block Zephyr releases.

          • Pull requests

            • Commit message format guidelines - as in Zephyr main tree

      • Contribution workflow - roles and responsibilities

        • Scenario: a Zephyr patch requires updates in Module code

          • Responsibility for keeping Module code up-to-date

          • No formal distinction (with respect to applied policy) between

            • Modules Zephyr depends on

            • Modules that use Zephyr

      • Contribution workflow - synchronization

        • Policy

          • Trivial changes and necessary changes

          • Non-trivial (e.g. functional / design) changes in modules shall, instead, be sent to the upstream project first

      • Modules - Licensing

        • License headers

        • Master license files

        • It should be always be made clear which license applies to each file

      • Modules - Licensing Policies

        • Policies

          • License checks (via CI) shall be enabled on every module PR, not only on PR Zephyr’s main repository

        • Licensing Problems

          • Reaching out to vendors/module contributors to discuss and eventually fix issues around licensing

        • Follow-up

      • Modules - Documentation

        • Documentation in module repositories

          • A template of module information - should be required for the inclusion of the module

            • Scope and purpose of the module

            • Integration with Zephyr

            • Maintainer-ship

            • Synchronization info (commit, SHA information, etc.)

          • Should be present (e.g. as .rst) in each module repository

          • Documentation in the main tree specific to each module

            • A landing page and a tool that pulls-in all module metadata pages and lists all module information in one page (as we have for Boards)

      • [Scott] Examples to reference 

      • [Question] Who is the Documentation Maintainer in Zephyr?

        • Carles volunteered

      • Timeline?

        • By the next LTS (to include licensing modules)

      • Modules - Testing

        • Testing modules’ code

          • HALs: implicitly tested via Zephyr test running on targets

          • Libraries, tools, etc: we will provide integration testing

            • Basic samples, tests, developed by Zephyr, residing in Zephyr tree

              • Covering basic use configurations of the module integrated to Zephyr

              • Requirements for module integration testing on a case-by-case basis

          • Certain modules have upstream tests

          • Upstream module tests explicitly written for Zephyr, could potentially be added in our testing 

      • Policies

        • Adding a new module

          • Module candidate: a project with its own life-cycle outside Zephyr (not written exclusively for Zephyr)

          • We should have a checklist for accepting a new module

        • Depreciate, and Remove modules

          • Generic policies for depreciation

          • List depreciate status in module metadata

          • Reasons may vary

      • Modules - Security

        • Security vulnerabilities

          • Aware of upstream project security issues and monitoring fixes

          • Update to latest versions before releasing Zephyr

        • Static analysis

          • Won’t perform scanning on modules’ code

      • Next steps

        • Continue discussion for 1-2 more meetings of the Process forum

        • Offline review of Draft

        • TSC to vote for the proposal?

      • [David B] How do we manage security vulnerabilities?

        • [Ioannis] Linking to existing security vulnerabilities in the upstream project (if it is present)

        • [Ioannis] Be aware and monitor fixes -- responsibility of module owner

    • Maintainers List

      • Good to have centrally located and pending PR is #24152.

      • YML + script which assigns code reviewers and PR.

      • Need to check and TSC approve terminology in YML

        • No objections to ARC architecture maintainer

        • No objections to Bluetooth collaborators

        • No objections to Bluetooth controller collaborators

        • [Anas] Are the maintainers and collaborators still on-board with maintaining?

          • [Ioannis] Will follow up with folks after we’re OK with folks

        • Devicetree maintainers? Remove ulfalizer? Yes, we can always add back.

        • [David Leach] Only maintainer needs TSC approval. Collaborators can be added by maintainers.

        • Do full review of the file once a year.

        • Documentation: Carles as maintainer? Doxygen might be too much but happy to maintain .rst. No TSC objections.

          • [Kumar] Subsystem docs should be owned by the subsystem maintainer and not the documentation maintainer.

          • Cannot scale having a doc maintainer review all PRs that change any doc.

        • Drivers: ADC: anangl as maintainer? No objections.

        • Drivers: Clock control: Christof (handle?)? Need to reach out. No TSC objections

        • Drivers: Console: pfalcon? [Kumar] Would list as unmaintained.

        • Drivers: Crypto: Any ideas? 

        • Drivers: DAC: Need to add to YML. Martin Jaegar. Alexander Wachter says Martin would accept.

        • Drivers: DMA: Any ideas?

        • Drivers: EEPROM? Henrikbrixandresen? No objections but need to confirm with them.

        • Drivers: Entropy? Flavio? No objections. Intel?

        • Drivers: ESPI? Anas? Intel? MicroChip?

        • Drivers: Ethernet: tbursztyka. Jukkar and pfalcon ok as collaborators.

        • Drivers: GPIO: mnkp plus any others

        • Should maintainers be singular? Yes, maybe. Collaborators can be added by the maintainer to help share review load without needing TSC approval.

        • Drivers: HW Info: alexanderwachter? Yup, ok doing it and no TSC objections.

        • Drivers: I2C: Marti Bolivar? No, was interested but ended up working on other things.

        • Drivers: I2S: Andrzej? Carles has discussed it with them. TSC has no objections. Armando Visconti, avisconti, has been maintaining I2S microphone drivers.

        • Drivers: Interrupt Controllers: Any ideas? One of the Andrews? Stephanos, non cortex-m ARM work.

        • Drivers: LED: @Mani-Sadhasivam as maintainer? Need to add them to pending PRs. No objections.

        • Drivers: LED Strip: Marti is accurate.

        • Drivers: Lora: Marti? @Mani-Sadhasivam as maintainer? Andreas Sandberg has been working on it. Check with Mani about time availability and TSC ok with them.

        • Drivers: Neural networking: Someone from Intel.

        • Drivers: PCI: Add someone from Broadcom? Up to the maintainer.

        • Drivers: PTP clock: jukkar? Confirm with them.

        • Drivers: PWM: Andrzej? Confirm with them and TSC has no objections.

        • Drivers: Video: loicpoulain: Confirm with them and TSC has no objections.

        • Drivers: Watchdog: Any suggestions? Propose a new maintainer who has been working in that space.

        • Filesystems: Lots of collaborators but no maintainer. Generally too broad of a category. [Kumar] Dominic? [Anas] Split FS so the existing entry is *just* the subsystem, not the implementations. That way we can have maintainers for each implementation.

        • JSON Web Token: [Carles] Have had recent contributions from Marcus Fuks (?) [Anas] Asking could be a deterrent to further contributions. [David Leach] Add the person as a collaborator. Should JWT be a subsystem or a lib? [David Brown] It’s there for linker reasons. Maybe David should be the maintainer?

        • Kconfig: ulfalizer no longer maintaining. [Anas] Wants to improve the Kconfig script. Switch Orphaned and needs further discussion.

        • MCU Manager: [Carles] Probably just forking this because upstream isn’t active. Dominic Arma? Check with them and no objections from TSC. Kevin Townsend also interested, Runtime isn’t. Don’t want iOS and Android parts in Zephyr.

        • NIOS-2 arch: Should be nashif? Yes. Not Ruud.

        • Power management: Check with wentongwu. No objections from TSC.

        • RISCV arch: Not really maintained, nategraff-sifive isn’t active. No objections to making it orphaned.

        • Shell: Jakub-uC can be readded because they are back to nordic.

        • Storage: [Carles] Is actually only SDCard? No flash map and… Andrzej P? No objections, check with them.

        • Tracing: Is maintained.

        • USB: finikorg? [Anas] Are they actually active still? Need to confirm with them. [Anas] Suggest orphan and check with them. That way we can start fresh. No TSC objections.

        • X86 arch: Switch to flavio. [Anas] No need to check with andreboie. Intel has already switched ownership internally.

      • [Kumar] Let’s make sure we aren’t missing subsystems.

      • [Anas] Push to merging soon and follow up.

      • Ok, to merge soon and then iterate because this is a new system.

      • One or two weeks to hear back from folks. If they don’t get back to us then we should leave as orphaned and modify later once we hear back.

      • Shoot for next Wednesday’s TSC meeting for final approval.

  • Developer Workflow, Slack [Kumar]

    • PRs from new contributors frequently fail remote git lint checks.

    • Thoughts on pre-commit hooks? Trade-off is needing more tools locally.

    • [Scott] https://pre-commit.com/ - Tools still need to be installed.

    • [Anas] check-patch doesn’t need to be per commit, should be per push. Per-commit is really annoying.

    • [Kumar] `git commit -n` will skip it.

    • [David Leach] drive-by PRs will either be fixed or go stale. Is it worth incorporating if folks won’t iterate on the PR?

    • [someone] pre-commit can give feedback privately, earlier than when PR is pushed.

    • [Anas] pre-commit hook is there already and we have issues. Some PRs don’t compile too.

    • [David Leach] If it’s important, we can help fix things they unexpectedly broke. Don’t run sanity locally, CI is helpful to do it.

    • [Anas] How do we balance local checks vs remote CI checks? Remote checks should fail earlier if very broken. Also stop checks early if git lint fails.

    • [Maureen] pre-commit git lint, not sanity check.

    • [Kumar] git lint, check patch could be local.

    • [Anas] Some folks don’t know how to find CI error output. Check patch fails because folks never set it up.

    • [Kumar] https://github.com/home-assistant/core

    • [David B] We’ve had people use the Git-UI to commit

    • Sometimes new people need local support

    • [Anas] Learning curve.

    • [Carles] Agree with discussion. most impactful - disabling sanity check if any of the pre-checks pass

    • [Carles] /home-assistant triggered automatically

    • [Anas] If you have to do it, provide the hooks as we have it

    • [Anas] Move between so many Git trees; forget to add the hooks

    • [Scott] pre-commit.com makes it easy; python script

    • Check patches - when you pass a series of commits to checkpatch, it can get confused

      • [Anas] think it’s a different issue not related to this. Agree- it would have to be fixed

    • [Kumar] If pre-commit worked out, we can look to implement.

    • [Carles] Weary of users on Windows (checkpatch may not work for them)

    • [Anas] Should be able to disable on Windows

    • [Kumar] This needs to work cleanly on Windows too

    • [Kumar] If we don’t already have a channel on messaging forum -- add a getting started development channel

    • [Kumar] Discussion re-start re Slack

    • [Kumar] Slack - lack of history becoming more of a detriment; cost model doesn’t work. Matrix, Discord, others? How do we move forward?

    • [Anas] Need to make the decision and move

    • [Anas] Multiple choice vote?

    • [Johan] Align voice conferencing tool with chat tool? (e.g. Teams)

    • [Kumar] Appreciate the desire, but the reasons we picked Teams was for a certain reason

    • [Carles] Any reason we can’t just use Teams for everything?

    • [David B] Need an enterprise account to use Teams

    • [Anas] Focus on chat right now

    • [David B] Pros and Cons of the chat tools we have looked at?

    • [Scott] Was unable to get on Matrix

    • [Kumar] Ease of use is easier with Discord (versus Matrix)

    • [Scott] Discord is no longer for gaming only; development going broader

    • [Kumar] Fedora uses it

    • [Carles] Concern would be if they eventually move to a paid model, similar to Slack (proprietary)

    • [Maureen] Do they support unlimited history?

    • [Scott] Has been using circuit python discord since 2017 -- can access history

    • [Carles] Discord doesn’t have threads

    • [Kumar] Voting? Community Survey? TSC vote? 

      • Option 1: Stay with Slack

      • Option 2: Move to Discord

    • Zephyr Discord = https://discord.gg/XJjaCgy

    • [Carles] How do you find the server?

      • [Scott] Use the link - it will work forever

    • [Maureen] If we go down this route, we need a clear motivation for doing it (e.g. what are the reasons for each choice)

      • [Kumar] Importance of history in discussions

      • [Kumar] Slack benefit: Subscribe base built up

    • [Carles] GitHub integration on Discord?

      • [Scott] Yes - set up hooks

    • Discord = Electron based

  • UX Study [Maureen]

  • Documentation for early-phase developers:

    • What is the development model for RTOS (differences from bare metal or standard OS application)?


    • What is device tree and how to use it?:


    • How to integrate sensors, drivers, and set configurations:


    • Zephyr vs. Linux concepts (where the same, where different); leveraging Linux experience with Zephyr:


    • Porting to custom boards; step-by-step tutorial:


    • Memory management, preventing stack overflows, debugging tips:


    • Document how Bluetooth and TCP/IP stack calls in Zephyr are different than Linux or Windows:


    • Introduction to new tools like West. Be clear about optional versus required use:


    • More information about West and how it uses Kconfig, DTS, Cmake:


    • Create architecture diagrams to help with some of the topics above:


    • Update error messages to be very specific to the error experienced to enable root cause identification:


    • Adding tools to help with multi-thread application development and de-bug:


  • Comprehensive sample app and tutorial.

  • Wrap Up [All]

    • [Maureen] Prioritize what things we want to focus on

      • Process discussion - good way forward on that

      • Maureen - actions to incorporate feedback from Strategic Objectives

      • Path forward for Discord? 

        • [Kumar] Email to dev list (include history, benefits of each, poll that provides 2 options) OR specify history is important; TSC has made the decision to transition to a different solution; TSC votes to finalize.

        • Decision: Announce over mail list as FYI (1 week of comments) + Schedule as topic in next TSC meeting to review any additional feedback and TSC Vote on to finalize

    • [Ioannis] TSC schedule w.r.t. Holidays?

      • TSC members requested to add their known vacations to TSC working doc and/or notify Maureen


Brett Preston
The Linux Foundation
+1 (971) 303-9030

Google Talk: bpreston@...
Skype: bprestoncf

Join tsc@lists.zephyrproject.org to automatically receive all group messages.