Date   

Re: RFC: Clarifying UART API

Daniel Thompson <daniel.thompson@...>
 

On 08/05/17 11:45, Paul Sokolovsky wrote:
Hello,
We have a problem in the UART (and UART driver) API very akin to the
proverbial half-empty vs half-full glass problem. So, there're 2
interrupt testing predicates:
* uart_irq_tx_ready()
* uart_irq_tx_empty()
What would each of them mean? Reading docstrings won't get you very
far, e.g. description of uart_irq_tx_ready() is "Check if Tx IRQ has
been raised", which is absolutely correct and absolutely useless, as
uart_irq_tx_empty() also means that Tx IRQ was raised, albeit a
different kind of IRQ.
Then thinking what they would mean, it becomes clear that they may mean
two *opposite* things:
Interpretation 1:
"ready" means "ready to accept more chars to transmit", "empty" means
"transmission complete".
Interpretation 2:
"empty" means "there's empty room in TX FIFO (so more chars can be
written to it)", "ready" means "transmission complete".
Perhaps we should note that interpretation 1 is the "right" one in the sense that this is what the current implementations of the API currently approximate to.

I also view interpretation #2 is only possible when the reasoner neglects the FIFO since when there is a FIFO we have to redefine the normal meaning of empty[1] to mean not-full.

Having said that, an API definition that relies on the reader intuiting the existence of FIFOs that may not event exist on their favorite test platform solely in order to comprehend an API is *seriously* user hostile!

[1] empty (adj.), Devoid of content; containing nothing or nobody;
vacant [this is the definition from wiktionary ;-) ].


The possibility of such U-turn'ish interpretation is alone a problem
for the good API design, and coupled with vagueness of the docs, it
becomes a real hazard, and we already saw signs of these completely
different interpretation being used in drivers/serial (specifically,
STM32 driver looked like trying to use interpretation 2).
I'm proposing to make the following changes to deconfuse it:
1. Given that there's a (sole) uart_irq_rx_ready() with the meaning
"there's a char ready to be received", it's natural to leave
uart_irq_tx_ready() as it is, and explicitly describe as "TX block is
ready to accept a char for a transmission".
Agree. Perhaps phrase as "accept at least one char for a transmission".

Note a couple of existing drivers check their interrupt mask and only admit to being ready if their interrupt is enabled. These drivers are anomalous and should be fixed to meet the above anyway.


2. Naming of uart_irq_tx_empty() is however strictly confusing, so
the proposal is to rename it to uart_irq_tx_complete(), and
explicitly describe that usecase behind it is knowing when the UART (or
the whole system) can be powered off, and not for the buffer control
during normal RX/TX UART operations.
Agree entirely on the proposal although with an alternative reasoning.

For me the real problem with uart_irq_tx_empty() is that I struggle to find any use case for it!

It is broken for power management because having an empty FIFO does not imply that it is safe to turn off the cell because it may still be shifting out the final character (in most designs the TX shift register is not part of the FIFO).

It is also pointless for implementing half duplex protocols for pretty much the same reason.

So what is this uart_irq_tx_empty() *for*? If we don't have a good answer then we should discard or depreciate it.

Similarly I agree that uart_irq_tx_complete() is a useful addition to the API.

Sorry to make you read to far just to find out that this RFC gets a +1 from me...


Daniel.


RFC: Clarifying UART API

Paul Sokolovsky
 

Hello,

We have a problem in the UART (and UART driver) API very akin to the
proverbial half-empty vs half-full glass problem. So, there're 2
interrupt testing predicates:

* uart_irq_tx_ready()
* uart_irq_tx_empty()

What would each of them mean? Reading docstrings won't get you very
far, e.g. description of uart_irq_tx_ready() is "Check if Tx IRQ has
been raised", which is absolutely correct and absolutely useless, as
uart_irq_tx_empty() also means that Tx IRQ was raised, albeit a
different kind of IRQ.

Then thinking what they would mean, it becomes clear that they may mean
two *opposite* things:

Interpretation 1:
"ready" means "ready to accept more chars to transmit", "empty" means
"transmission complete".

Interpretation 2:
"empty" means "there's empty room in TX FIFO (so more chars can be
written to it)", "ready" means "transmission complete".


The possibility of such U-turn'ish interpretation is alone a problem
for the good API design, and coupled with vagueness of the docs, it
becomes a real hazard, and we already saw signs of these completely
different interpretation being used in drivers/serial (specifically,
STM32 driver looked like trying to use interpretation 2).


I'm proposing to make the following changes to deconfuse it:

1. Given that there's a (sole) uart_irq_rx_ready() with the meaning
"there's a char ready to be received", it's natural to leave
uart_irq_tx_ready() as it is, and explicitly describe as "TX block is
ready to accept a char for a transmission".

2. Naming of uart_irq_tx_empty() is however strictly confusing, so
the proposal is to rename it to uart_irq_tx_complete(), and
explicitly describe that usecase behind it is knowing when the UART (or
the whole system) can be powered off, and not for the buffer control
during normal RX/TX UART operations.



--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog


Re: BSD Sockets like API: prototyping report #3

Paul Sokolovsky
 

Hello,

On Thu, 4 May 2017 17:47:17 +0300
Paul Sokolovsky <Paul.Sokolovsky@...> wrote:

[]

b) My "ultimate" test was supposed to be downloading large files (tens
of MBs) from http://archive.ubuntu.com/, checking their hash (I'm
using that site as it readily offers hash values for different
algorithms), then repeating to collect enough traffic (ultimately
mentioned gigabyte). Unfortunately, I didn't have much luck with this
"ambitious" goal. Overall, I got 100% TCP hang on byte counts ranging
from 100Kbs to ~10MB.
I figured out this issue - it's because Zephyr IP stack doesn't
anticipate the possibility that its ACK may be lost. Then peer will
resend a packet Zephyr already saw, and it doesn't like that. This is
ticketed as https://jira.zephyrproject.org/browse/ZEP-2121 .

With workaround for that applied, I didn't have problems with prolonged
downloads from the real Internet:

========
...
Received already seen pkt - ACKing
Received already seen pkt - ACKing
Received already seen pkt - ACKing
Making sure FIN goes with ACK; adjusting seq by 2; now fire up Wireshark and check that all's ok
Size : 7015896
Hash : 337c37d7ec00348414224baa6bdb2d43f2a34ef5676bafcdcad916f148b5b317
Total downloaded so far: 160M
========


Summing up, basic testing with a convenient to use API (BSD Sockets)
and easy to use environment to exercise that API (MicroPython) quickly
uncovered 3 critical issues in Zephyr IP stack, and few more less
serious, which shows the potential ;-).


[]

--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog


Help required on reading UART device and print on console

Dhamodharan Krishnan <dhamukrish@...>
 

Hi,

I am a day old since I started in Zephyr and from Arduino background. Could you please help with an example how to read from UART ports which are from 

Here is the code I have tried but failed.

#include <zephyr.h>
#include <misc/printk.h>
#include <uart.h>

int main(void) {

struct device *uartGPS;
uartGPS = device_get_binding(UART_1);

while (1) {
unsigned char* msg;
uart_poll_in(uartGPS, msg);

printk("%s", msg);
}
}

Regards
digidhamu


RFC: Fault injection framework for Zephyr

Paul Sokolovsky
 

Hello,

It's fair to say that we're seeing the need to better test various
error handling paths in Zephyr, for example, in networking code.

This is commonly handled with fault injection. I grepped thru Jira, and
didn't find anything related, so decided to open a story on that:
https://jira.zephyrproject.org/browse/ZEP-2118 .

I don't think it's anyhow a priority for 1.8, so I'm posting this
mostly FYI and to avoid possible duplication. If you have any ideas,
it's probably best to add the comments to the ticket above.


Thanks,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog


BSD Sockets like API: prototyping report #3

Paul Sokolovsky
 

Hello,

This report is quite overdue, and I hoped to have sent it weeks ago, as
the initial implementation of all the basic functions were done long
ago. But I wanted to back it with automated easily reproducible
testing, as well as test against real-world sites or well-known
networking servers with real data sizes (like an odd gigabyte) - and
both aims turned out to be elusive, requiring excursions in unrelated
areas (like UART handling in Zephyr, still ongoing) and many more
investigation and fixes.

Well, now things more or less got together:

1. There's implementation of the following function (with Python
interface, which would still require some refactoring to turn them into
C ones): socket(), bind(), connect(), listen(), accept(), recv(),
send(), close(), getaddrinfo(). Only blocking sockets are supported so
far.

2. These are implemented in "usocket" module of MicroPython, in
"zephyr-socket2" branch:
https://github.com/pfalcon/micropython/blob/zephyr-socket2/zephyr/modusocket.c
To work as expected, this requires master with some additional patches
applied: https://github.com/pfalcon/zephyr/tree/master-micropython .

3. It was tested in following ways:

a) There few (4) automated tests, for basic UDP functionality.
Unfortunately, TCP tests don't work in this manner,
https://jira.zephyrproject.org/browse/ZEP-2001.

b) My "ultimate" test was supposed to be downloading large files (tens
of MBs) from http://archive.ubuntu.com/, checking their hash (I'm using
that site as it readily offers hash values for different algorithms),
then repeating to collect enough traffic (ultimately mentioned
gigabyte). Unfortunately, I didn't have much luck with this "ambitious"
goal. Overall, I got 100% TCP hang on byte counts ranging from
100Kbs to ~10MB. A raw TCP connection hanging is an expected situation
in general (in real-world apps, that's countered by having connection
timeout), but hitting that in 100% of cases would mean that something
not yet fully right with Zephyr stack handling of non-successful
conditions (like retransmits, backoffs, etc.).

c) So, I had to scale down to use my local Apache server. There
everything went much better, and I was able to download ~300MB at the
first try (sorry, keep forgetting to leave it overnight for the
proverbial gigibyte). So, overall, "success" flow is ok both on Zephyr
side and on BSD prototype side, there're no buffer leaks, etc. (Again,
"non-success" cases where there's need for some recovery and retries
is a different story). This concludes (initial) client-side testing.

d) Trying server-side handling, I immediately hit an issue with
(case-dependent) FIN handling:
https://jira.zephyrproject.org/browse/ZEP-2104. Working that around
(proper fix yet needs investigation), I'm happy to report that a simple
HTTP server sample sustains 1000 requests from ApacheBench (ab -n 1000
http://192.0.2.1:8080/).



Further plans:

1. There are some optimizations to do yet on the current code, before
separating socket functions from MicroPython boilerplate.

2. It's clear that in-tree patches won't be ready before 1.8 release
freeze, so instead I plan to concentrate on trying to flush any
smaller changes which may be useful for mainline already, and on
investigating identified issues.




--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog


Re: RFC: TCP receive/send window handling in Zephyr IP stack

Paul Sokolovsky
 

Hello Jukka,

On Tue, 02 May 2017 17:32:30 +0300
Jukka Rissanen <jukka.rissanen@...> wrote:

[]

https://jira.zephyrproject.org/browse/ZEP-1999 .

So, as the issue is confirmed, I would like to proceed with
resolving it, and this RFC is about ways to do it.
Thanks for pushing this, we definitely have an issue with the rcv wnd
handling.
Thanks for supporting it. Given that I won't have time to prepare BSD
Sockets API patches for 1.8, I'm switching instead on trying to resolve
as many issues as was spotted during its prototyping work for 1.8
timeframe.

lwIP offers a very simple and easy to understand model of that: IP
stack only decreases the available receive window, never increases
it,
it's a task of application code. When an application really
processes received data (vs just buffering it), it calls a special
function (called tcp_recved()) to advance the window.
[]

way to do that would be to change recv callback signature from:

typedef void (*net_context_recv_cb_t)(struct net_context *context,
                                      struct net_pkt *pkt,
                                      int status,
                                      void *user_data);

to one returning u32_t, the size by which to advance the receive
window. For each existing recv callback, that would be
net_pkt_appdatalen(pkt), so we can try to optimize that one step
further by defining some constant and allowing to return it:

return NET_ADVANCE_WHOLE_PKT;
I like this better than an application manually calling a special
function to do the trick like what was found in the gerrit patch.
I see. And I on my side was pondering these days about the comment you
gave in Gerrit review: "Why cannot we call this directly inside the
stack when we receive data?"

Is it likely that the application would not move the window for full
packet? I mean what would happen if we always move the window the
received data amount automatically by the core stack after we return
from recv cb?
So, that would be almost the same as it is now, except that currently
receive window moved (i.e. ack value updated) before calling recv cb.
It's definitely more correct to move it after, but the difference in
behavior probably will be negligible in practice, and it won't help
applications which *buffer* incoming packets, like what BSD Sockets API
does.

The net_buf's should be released anyway by the
application asap, otherwise we have possible memory leak.
Not memory leak, but buffering ;-). So, returning to your idea of doing
it completely automatically, like a grown-up IP stack. The whole idea
is that receive window should be moved not when packet *received* (and
that's the event which recv_cb signals), but when data in packet is
*processed* (which may be much later due to buffering).

How can we know that application finished processing? Again, simple
stacks like lwIP require explicit notification of that. But how to do
it otherwise? Well, we know that data processing was 100% finished when
a network buffer holding it is freed. Hopefully, that gives enough of
insight on the idea, and you can comment - how do you like that
net_buf_frag_del() and friends will now update TCP structures related
to a fragment - at the minimum, at the maximum they may trigger an
outgoing packet to be sent (to notify peer of the window update).

Unfortunately, that's inevitable - TCP for efficient implementation
requires close enough coupling with IP and buffer management. Again,
apparently to simplify that lwIP put window update step in the user app
instead. I'll submit a patch for MTU handling for outgoing packets
soon, which already exposes this problem of the need to intermingle
different "layers" of API (so, we would need to stop thinking about
them as different layers, just as one).

Is it likely that the application would not move the window for full
packet?
Fairly speaking, we should move window as soon as we're able to
receive new data chunk. As we allocate data chunks in fragment buffers,
freeing one would be already a good ground to update recv window. But:
recv window is measured in terms of data bytes, but we receive data in
terms of packets, which have protocol headers overhead. So, if we got 1
fragment buffer freed (128 bytes), if we announce that to peer, we'll
get 128 + 40 (IPv4) bytes back. If we had only one fragment buffer,
this packet will be dropped, and everything will be as bad as it is now
(exponential backoff triggering, etc.)

So, it's not that easy, but I'm excited about the idea to try to do it
fully automatic, and as I said, I was pondering about it all these days,
and hope to proceed to experimenting a bit later. Overall, recv/send
window handling will require various rules, majority of which would be
heuristic. lwIP for one, even if receives window updates from an app,
does quite a lot of magic with them before they go into real packets.

[]

--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog


Re: Move to Github completed and some issue

Leandro Pereira
 

Andrew,

On 05/03/2017 08:30 AM, Boie, Andrew P wrote:

I would like to leverage this so that I can have Zephyr notifications sent to my Intel email address and not my personal Gmail. I can’t figure out how to do this however. I have both email accounts verified in the Github system. Any pointers?
This is only possible if you're a member of the organization. Right now, as far as I can tell, people are mostly contributors to the "zephyr" repository there. This is in the pipeline to be fixed.

Leandro


Re: Zephyr JSON API can't find missing fields in sub-objects

Leandro Pereira
 

Marti,

On 05/02/2017 10:07 PM, Marti Bolivar wrote:

However, when I parse this, I can tell whether "config" or "_links" is
present in the top-level JSON object using the return value from
json_obj_parse(), but I can't tell if its subobjects (the
"deploymentBase", "cancelAction" etc. in "_links") are present or not.
That's indeed a known limitation.


One way to do this would be to introduce a u32_t in each structure
(hawkbit_ctl_res_sleep, hawkbit_ctl_res_polling, hawkbit_ctl_res_href,
hawkbit_ctl_res_links) recording whether or not each of the "real"
fields is present, and have the JSON API fill this value recursively
as it parses. This would let me decide what fields are present at all
levels in the parsed hierarchy.

Does this seem reasonable? Does anyone else have other suggestions?
In this case, it might be sufficient to zero out the struct hawkbit_ctl_res struct, and then just NULL-check _links.cancelAction.href or _links.deploymentBase.href. It's not the ideal generic solution but will work for strings at least.

For the general case, it might be worth doing something like you mentioned, but making the added integer member optional. One way to do this is to consider cases like this as union cases:

struct links {
union {
struct href deploymentBase;
struct href cancelAction;
};
struct href configData;
u32 fields;
};

The `fields` member would have 1<<0 set if deploymentBase was decoded, 1<<1 if cancelAction was decoded, and 1<<2 if configData was decoded. The href struct wouldn't need a `fields` member of its own since it's not a union case.

Now, using a union wouldn't work if deploymentBase and cancelAction could be set at the same time. In this case, one just removes the union declaration, and things should work. The only confusing bit is that this would still be a union case, but without the declaration of a union properly.

Cheers,
Leandro


Re: Move to Github completed and some issue

Boie, Andrew P
 

Github appears to have a feature where you can have notification emails to sent to different email addresses depending on the ‘organization’ that owns a particular project. Under Personal Settings -> Notifications -> Custom Routing: “You can send notifications to different verified email addresses depending on the organization that owns the repository.”

 

I would like to leverage this so that I can have Zephyr notifications sent to my Intel email address and not my personal Gmail. I can’t figure out how to do this however. I have both email accounts verified in the Github system. Any pointers?

 

 

 

From: zephyr-devel-bounces@... [mailto:zephyr-devel-bounces@...] On Behalf Of Nashif, Anas
Sent: Wednesday, May 3, 2017 2:19 AM
To: Zephyr Devel <devel@...>
Subject: [Zephyr-devel] Move to Github completed and some issue

 

Hi,

We are now on github and PRs are flowing in, CI is operational as expected and we are getting used to the new flow.

 

Thank you for all who helped make this happen.

 

Now, we have been on gerrit for a while and we have been used to doing things in a certain way and a change in mindset is needed. Github has its own way of doing things and we need to adapt, one thing we immediately noticed is the fact that no everyone can add reviewers… This can only be done by users with write access to a specific repo. We are trying a few things to deal with that. In gerrit we used to add many reviewers using a script and we want to continue doing this using github APIs. In Github we have the concept of Assignee and Reviewers, we want to make use of that.

 

The other issue we are noticing is with subsystem branches, i.e. Bluetooth, net and ARM. Force push will not be possible if we decide to protect those branches and apply some rules on them and working around this through merges is not clean as well (leaves duplicate, phantom commits in the branch). We have tried to enable force push on the branches and removed protection, this results in PRs getting lots of unrelated commits when branches are rebased by maintainers.

This raises the question if in the github world we still need the subsystem branches on the main tree and if we should request all PRs to be submitted to master and have subsystem owners manage incoming changes to their subsystem on master, given that we have to go via  a fork anyways. To make it easier on maintainers of the various subsystem we can use the ‘project’ concept in Github and apply it to filter and triage pull requests and later issues if we decide to move to github issues in the future.

 

Moving all development to master has a few other advantages IMO. First, it is easy on new comers and occasional contributors, you do not need to figure out where to submit. There is only one entry point for all patches and changes. The other thing is the fact that when working on branches, those turn into isolated Silos of development with own policies and rules and end up in master as one large change. Finally, when developing against master and submitting changes to master we will keep master up to date all the time and there is no lag  in introducing features to master.

 

We had a lengthy discussion on IRC today about this but not everyone is on IRC, so posting this to get more views on the topic.

 

Regards,

Anas

 

 


Re: Zephyr JSON API can't find missing fields in sub-objects

Marti Bolivar <marti.bolivar@...>
 

Hit send too soon.

On 3 May 2017 at 00:55, Marti Bolivar <marti.bolivar@...> wrote:

Hi,

I am trying to convert an application to use the Zephyr JSON library in lib/json from jsmn [1], and have run into an issue that I think will req

...uire changes to the Zephyr API.

Here is the reference on the JSON I am trying to parse, for the
hawkBit device management system:

http://www.eclipse.org/hawkbit/documentation/interfaces/ddi-api.html

In particular, I need to parse responses that sometimes look like this:

{
"config": {
"polling": {
"sleep": "00:00:30"
}
},
"_links": {
"deploymentBase": {
"href": "https://some_url"
},
"configData": {
"href": "https://some_url"
}
}
}

And sometimes look like this:

{
"config": {
"polling": {
"sleep": "00:00:30"
}
},
"_links": {
"cancelAction": {
"href": "https://some_url"
},
"configData": {
"href": "https://some_url"
}
}
}

I've done that with something like what I've attached in foo.c.

However, when I parse this, I can tell whether "config" or "_links" is
present in the top-level JSON object using the return value from
json_obj_parse(), but I can't tell if its subobjects (the
"deploymentBase", "cancelAction" etc. in "_links") are present or not.

This leaves me unable to tell which kind of response I got -- one
containing a deploymentBase, or one containing a cancelAction.

I think there's no way around this without changing the Zephyr JSON API.

One way to do this would be to introduce a u32_t in each structure
(hawkbit_ctl_res_sleep, hawkbit_ctl_res_polling, hawkbit_ctl_res_href,
hawkbit_ctl_res_links) recording whether or not each of the "real"
fields is present, and have the JSON API fill this value recursively
as it parses. This would let me decide what fields are present at all
levels in the parsed hierarchy.

Does this seem reasonable? Does anyone else have other suggestions?

Thanks,
Marti



[1] https://github.com/zserge/jsmn


Zephyr JSON API can't find missing fields in sub-objects

Marti Bolivar <marti.bolivar@...>
 

Hi,

I am trying to convert an application to use the Zephyr JSON library in lib/json from jsmn [1], and have run into an issue that I think will req

[1] https://github.com/zserge/jsmn


Move to Github completed and some issue

Nashif, Anas
 

Hi,

We are now on github and PRs are flowing in, CI is operational as expected and we are getting used to the new flow.

 

Thank you for all who helped make this happen.

 

Now, we have been on gerrit for a while and we have been used to doing things in a certain way and a change in mindset is needed. Github has its own way of doing things and we need to adapt, one thing we immediately noticed is the fact that no everyone can add reviewers… This can only be done by users with write access to a specific repo. We are trying a few things to deal with that. In gerrit we used to add many reviewers using a script and we want to continue doing this using github APIs. In Github we have the concept of Assignee and Reviewers, we want to make use of that.

 

The other issue we are noticing is with subsystem branches, i.e. Bluetooth, net and ARM. Force push will not be possible if we decide to protect those branches and apply some rules on them and working around this through merges is not clean as well (leaves duplicate, phantom commits in the branch). We have tried to enable force push on the branches and removed protection, this results in PRs getting lots of unrelated commits when branches are rebased by maintainers.

This raises the question if in the github world we still need the subsystem branches on the main tree and if we should request all PRs to be submitted to master and have subsystem owners manage incoming changes to their subsystem on master, given that we have to go via  a fork anyways. To make it easier on maintainers of the various subsystem we can use the ‘project’ concept in Github and apply it to filter and triage pull requests and later issues if we decide to move to github issues in the future.

 

Moving all development to master has a few other advantages IMO. First, it is easy on new comers and occasional contributors, you do not need to figure out where to submit. There is only one entry point for all patches and changes. The other thing is the fact that when working on branches, those turn into isolated Silos of development with own policies and rules and end up in master as one large change. Finally, when developing against master and submitting changes to master we will keep master up to date all the time and there is no lag  in introducing features to master.

 

We had a lengthy discussion on IRC today about this but not everyone is on IRC, so posting this to get more views on the topic.

 

Regards,

Anas

 

 


Re: Arduino DUE with ESP6288

Jukka Rissanen
 

Hi Kevin,

there is no driver for ESP6288 in Zephyr. Patches are welcome to fix
this situation.

Cheers,
Jukka

On Fri, 2017-04-28 at 09:57 +0000, Kevin Stöckl wrote:
Hello,
I have connected the Arduino DUE with the ESP6288 to get an internet
connection. Now I want to send my data to thingspeak.com or
thingsboard.io. How is this possible?
_______________________________________________
Zephyr-devel mailing list
Zephyr-devel@...
https://lists.zephyrproject.org/mailman/listinfo/zephyr-devel


Re: RFC: TCP receive/send window handling in Zephyr IP stack

Jukka Rissanen
 

Hi Paul,

On Fri, 2017-04-28 at 21:28 +0300, Paul Sokolovsky wrote:
Hello,

Some time ago I "noticed" that Zephyr IP stack doesn't do any TCP
receive window handling, instead using static value for each packet
it
sends. Today I confirmed the expectation that it would be problematic
for BSD Sockets implementation I'm working on. The whole story is
available as https://jira.zephyrproject.org/browse/ZEP-1999 .

So, as the issue is confirmed, I would like to proceed with resolving
it, and this RFC is about ways to do it.
Thanks for pushing this, we definitely have an issue with the rcv wnd
handling.


lwIP offers a very simple and easy to understand model of that: IP
stack only decreases the available receive window, never increases
it,
it's a task of application code. When an application really processes
received data (vs just buffering it), it calls a special function
(called tcp_recved()) to advance the window.

I don't think there can be more general and flexible solution than
that, so I propose that we add a similar function to explicitly
advance
the window, and add the code to track decrease of the current window
with each packet received.

However, it means that each and every existing receive callback will
need to be patched to call that function, and if that's not done, the
communication will cease to work silently. So, we may want (to
consider) a way to "automate" it, or at least be a compile-time
breaking
change, instead of silent runtime failure.

One way to do that would be to change recv callback signature from:

typedef void (*net_context_recv_cb_t)(struct net_context *context,
                                      struct net_pkt *pkt,
                                      int status,
                                      void *user_data);

to one returning u32_t, the size by which to advance the receive
window. For each existing recv callback, that would be
net_pkt_appdatalen(pkt), so we can try to optimize that one step
further by defining some constant and allowing to return it:

return NET_ADVANCE_WHOLE_PKT;
I like this better than an application manually calling a special
function to do the trick like what was found in the gerrit patch.

Is it likely that the application would not move the window for full
packet? I mean what would happen if we always move the window the
received data amount automatically by the core stack after we return
from recv cb? The net_buf's should be released anyway by the
application asap, otherwise we have possible memory leak.



Note that I would be in favor of there's-one-way-to-do-it solution of
explicit function call to advance window. However, as a compromise,
having a return value from recv callback also makes sense, and I
mention "special constant" choice just for a perspective, I'd be -0.5
on it.
Thanks,
Paul


Cheers,
Jukka


RFC: HTTP client API for review

Jukka Rissanen
 

Hi all,

I just submitted a simple HTTP client API for review at https://github.
com/zephyrproject-rtos/zephyr/pull/50

The API is very simple providing following functions:

* http_client_send_req(): This is the base function that both the GET
and POST helper functions will call. The function supports both
push/pull (async/sync) model of working. If the callback is given, then
the request is done in asynchronous way.

* http_client_send_get_req(): Helper to create a GET request
* http_client_send_post_req(): Helper to create a POST request
* http_client_init(): Initialize HTTP context needed by above
functions.
* http_client_release(): Release the HTTP context.

This API will move most of the code from http-client app into this new
library and make the new http-client sample application very simple.


The pull-request also has support for new API for the network sample
applications. So instead of each net sample application doing various
setup things itself, an API is provided which initializes the network
stack and makes the sample network application startup much simpler.
The plan is to convert other network sample apps to use this new API
after the commit has been merged.


For testing purposes, the net-tools project now contains a simple http
server that can be used in Qemu testing. See this pull-request for
details https://github.com/zephyrproject-rtos/net-tools/pull/1


Below are the function documentation from include file:

/**
 * @brief Send a HTTP request to peer.
 *
 * @param http_ctx HTTP context.
 * @param req HTTP request to perform.
 * @param cb Callback to call when the response has been received from
peer.
 * @param result_buf Caller supplied buffer where the HTTP request will
be
 * stored
 * @param result_buf_len Length of the caller suppied buffer.
 * @param user_data A valid pointer on some user data or NULL
 * @param timeout Amount of time to wait for a reply. If the timeout is
0,
 * then we return immediately and the callback (if set) will be called
later.
 *
 * @return Return 0 if ok, and <0 if error.
 */
int http_client_send_req(struct http_client_ctx *http_ctx,
 struct http_client_request *req,
 http_request_cb_t cb,
 u8_t *result_buf,
 size_t result_buf_len,
 void *user_data,
 s32_t timeout);

/**
 * @brief Send a HTTP GET request to peer.
 *
 * @param http_ctx HTTP context.
 * @param url URL to use.
 * @param host Host field in HTTP header. If set to NULL, then server
 * name is used.
 * @param extra_header_fields Any extra header fields that caller wants
 * to add. This can be set to NULL.
 * @param cb Callback to call when the response has been received from
peer.
 * @param result_buf Caller supplied buffer where the HTTP request will
be
 * stored
 * @param result_buf_len Length of the caller suppied buffer.
 * @param user_data A valid pointer on some user data or NULL
 * @param timeout Amount of time to wait for a reply. If the timeout is
0,
 * then we return immediately and the callback (if set) will be called
later.
 *
 * @return Return 0 if ok, and <0 if error.
 */
static inline int http_client_send_get_req(struct http_client_ctx
*http_ctx,
   const char *url,
   const char *host,
   const char
*extra_header_fields,
   http_request_cb_t cb,
   u8_t *result_buf,
   size_t result_buf_len,
   void *user_data,
   s32_t timeout)
{
struct http_client_request req = {
.method = "GET ",
.url = url,
.host = host,
.protocol = " " HTTP_PROTOCOL
HTTP_CRLF,
.header_fields = extra_header_fields,
};

http_ctx->req.method = HTTP_GET;

return http_client_send_req(http_ctx, &req, cb, result_buf,
    result_buf_len, user_data,
timeout);
}

/**
 * @brief Send a HTTP POST request to peer.
 *
 * @param http_ctx HTTP context.
 * @param url URL to use.
 * @param host Host field in HTTP header. If set to NULL, then server
 * name is used.
 * @param extra_header_fields Any extra header fields that caller wants
 * to add. This can be set to NULL.
 * @param content_type Content type of the data.
 * @param payload Payload data.
 * @param cb Callback to call when the response has been received from
peer.
 * @param result_buf Caller supplied buffer where the HTTP request will
be
 * stored
 * @param result_buf_len Length of the caller suppied buffer.
 * @param user_data A valid pointer on some user data or NULL
 * @param timeout Amount of time to wait for a reply. If the timeout is
0,
 * then we return immediately and the callback (if set) will be called
later.
 *
 * @return Return 0 if ok, and <0 if error.
 */
static inline int http_client_send_post_req(struct http_client_ctx
*http_ctx,
    const char *url,
    const char *host,
    const char
*extra_header_fields,
    const char *content_type,
    const char *payload,
    http_request_cb_t cb,
    u8_t *result_buf,
    size_t result_buf_len,
    void *user_data,
    s32_t timeout)
{
struct http_client_request req = {
.method = "POST ",
.url = url,
.host = host,
.protocol = " " HTTP_PROTOCOL
HTTP_CRLF,
.header_fields = extra_header_fields,
.content_type_value = content_type,
.payload = payload,
};

http_ctx->req.method = HTTP_POST;

return http_client_send_req(http_ctx, &req, cb, result_buf,
    result_buf_len, user_data,
timeout);
}

/**
 * @brief Initialize user supplied HTTP context.
 *
 * @detail Caller can set the various fields in http_ctx after this
call
 * if needed.
 *
 * @param http_ctx HTTP context.
 * @param server HTTP server address or host name. If host name is
given,
 * then DNS resolver support (CONFIG_DNS_RESOLVER) must be enabled. If
caller
 * sets the server parameter as NULL, then it no attempt is done to
figure out
 * the remote address and caller must set the address in
http_ctx.tcp.remote
 * itself.
 * @param server_port HTTP server TCP port.
 *
 * @return Return 0 if ok, <0 if error.
 */
int http_client_init(struct http_client_ctx *http_ctx,
     const char *server, u16_t server_port);

/**
 * @brief Release all the resources allocated for HTTP context.
 *
 * @param http_ctx HTTP context.
 */
void http_client_release(struct http_client_ctx *http_ctx);


Please comment directly in github pull request.


Cheers,
Jukka


Re: Zephyr is moving to Github on May 1st

Marti Bolivar <marti.bolivar@...>
 

Wonderful; glad to hear it!

Marti

On 1 May 2017 at 10:56, Anas Nashif <nashif@...> wrote:
Marti,

Yes, those will be enabled on github as well, emails should start flowing in during this week.

Anas

On Mon, May 1, 2017 at 10:08 AM, Marti Bolivar <marti.bolivar@...> wrote:
Hi Anas,

Thanks for the information. I read through the emails on this and the Contribution Guide on the Zephyr GitHub wiki, which was clear.

One thing I don't see is whether the daily Gerrit digest emails will be replaced with a GitHub-based equivalent. Are there plans for that, or should we be using the repository "watch" feature instead?

IMO, the daily digests are an improvement (less email, easier to drill down on things that are interesting) over repository watching.

Thanks,
Marti



On 27 April 2017 at 19:37, Nashif, Anas <anas.nashif@...> wrote:

Hi,

First, sorry for the short notice.

 

Some of you might have heard about this already, it was mentioned in discussions and events. Most recently it was proposed officially to the board and was approved.

 

The Zephyr project launched last year with Gerrit as a code review system and it was meant to be used for this purpose until we have critical mass and a larger community. Over the last year we had a lot of feedback regarding the choice of Gerrit for the Zephyr project, most of which was negative feedback, however, I have talked to many developers and contributors who were actually happy with Gerrit, however in the 1st Zephyr mini-summit held last year in Austin we decided unanimously to move to github.com.

 

Our main goal is to have most of services we use for the project  in one place and have the best integration possible to enhance the development experience.

 

=== What does this mean for you as a Zephyr project contributor? ===

 

Starting next week, all code submissions have to be done using Pull Request on the Zephyr project github page:

 

https://github.com/zephyrproject-rtos/zephyr

 

The latest tree and the latest releases (https://github.com/zephyrproject-rtos/zephyr/releases )

will be available from Github directly.

 

Detailed documentation on how to submit changes will appear in the top-level README.md file which should also appear in the repository web page on github. For those not familiar with github, please take a look at the Pull Request method for submitting code changes:

 

https://guides.github.com/introduction/flow/

 

 

=== What else is moving? ===

 

Beside the hosting of the GIT repo and the review process using pull requests, the following will be moved to github:

 

-          Releases: https://github.com/zephyrproject-rtos/zephyr/releases

-          Wiki: We will move content from wiki.zephyrproject.org to three possible place:

o    the built-in wiki on github,

o   in some cases to the source tree in RST format

o   some content will move to the website

 

=== What about Jira? ===

 

At this moment we will continue using Jira as is, we are discussing and exploring usage of github issues, but this is still being evaluated.

 

 

=== What happens to my pending changes in gerrit? ===

 

Gerrit will be switched to operate in offline mode and all the existing changes will be available for processing and transition through May. However, we encourage you to resubmit your changes to Github and abandon changes on github. We already gone through the existing changes in github and did some initial cleanup.

 

 

Additional information and the new process for submitting changes will be documented on the github page and will be available already tomorrow. CI will also be available through github with additional details.

 

Your comments are welcome.

 

Anas

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


_______________________________________________
Zephyr-devel mailing list
Zephyr-devel@...ct.org
https://lists.zephyrproject.org/mailman/listinfo/zephyr-devel



_______________________________________________
Zephyr-devel mailing list
Zephyr-devel@...ct.org
https://lists.zephyrproject.org/mailman/listinfo/zephyr-devel




Re: Zephyr is moving to Github on May 1st

Anas Nashif
 

Marti,

Yes, those will be enabled on github as well, emails should start flowing in during this week.

Anas

On Mon, May 1, 2017 at 10:08 AM, Marti Bolivar <marti.bolivar@...> wrote:
Hi Anas,

Thanks for the information. I read through the emails on this and the Contribution Guide on the Zephyr GitHub wiki, which was clear.

One thing I don't see is whether the daily Gerrit digest emails will be replaced with a GitHub-based equivalent. Are there plans for that, or should we be using the repository "watch" feature instead?

IMO, the daily digests are an improvement (less email, easier to drill down on things that are interesting) over repository watching.

Thanks,
Marti



On 27 April 2017 at 19:37, Nashif, Anas <anas.nashif@...> wrote:

Hi,

First, sorry for the short notice.

 

Some of you might have heard about this already, it was mentioned in discussions and events. Most recently it was proposed officially to the board and was approved.

 

The Zephyr project launched last year with Gerrit as a code review system and it was meant to be used for this purpose until we have critical mass and a larger community. Over the last year we had a lot of feedback regarding the choice of Gerrit for the Zephyr project, most of which was negative feedback, however, I have talked to many developers and contributors who were actually happy with Gerrit, however in the 1st Zephyr mini-summit held last year in Austin we decided unanimously to move to github.com.

 

Our main goal is to have most of services we use for the project  in one place and have the best integration possible to enhance the development experience.

 

=== What does this mean for you as a Zephyr project contributor? ===

 

Starting next week, all code submissions have to be done using Pull Request on the Zephyr project github page:

 

https://github.com/zephyrproject-rtos/zephyr

 

The latest tree and the latest releases (https://github.com/zephyrproject-rtos/zephyr/releases )

will be available from Github directly.

 

Detailed documentation on how to submit changes will appear in the top-level README.md file which should also appear in the repository web page on github. For those not familiar with github, please take a look at the Pull Request method for submitting code changes:

 

https://guides.github.com/introduction/flow/

 

 

=== What else is moving? ===

 

Beside the hosting of the GIT repo and the review process using pull requests, the following will be moved to github:

 

-          Releases: https://github.com/zephyrproject-rtos/zephyr/releases

-          Wiki: We will move content from wiki.zephyrproject.org to three possible place:

o    the built-in wiki on github,

o   in some cases to the source tree in RST format

o   some content will move to the website

 

=== What about Jira? ===

 

At this moment we will continue using Jira as is, we are discussing and exploring usage of github issues, but this is still being evaluated.

 

 

=== What happens to my pending changes in gerrit? ===

 

Gerrit will be switched to operate in offline mode and all the existing changes will be available for processing and transition through May. However, we encourage you to resubmit your changes to Github and abandon changes on github. We already gone through the existing changes in github and did some initial cleanup.

 

 

Additional information and the new process for submitting changes will be documented on the github page and will be available already tomorrow. CI will also be available through github with additional details.

 

Your comments are welcome.

 

Anas

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


_______________________________________________
Zephyr-devel mailing list
Zephyr-devel@...ct.org
https://lists.zephyrproject.org/mailman/listinfo/zephyr-devel



_______________________________________________
Zephyr-devel mailing list
Zephyr-devel@lists.zephyrproject.org
https://lists.zephyrproject.org/mailman/listinfo/zephyr-devel



Re: Zephyr is moving to Github on May 1st

Marti Bolivar <marti.bolivar@...>
 

Hi Anas,

Thanks for the information. I read through the emails on this and the Contribution Guide on the Zephyr GitHub wiki, which was clear.

One thing I don't see is whether the daily Gerrit digest emails will be replaced with a GitHub-based equivalent. Are there plans for that, or should we be using the repository "watch" feature instead?

IMO, the daily digests are an improvement (less email, easier to drill down on things that are interesting) over repository watching.

Thanks,
Marti



On 27 April 2017 at 19:37, Nashif, Anas <anas.nashif@...> wrote:

Hi,

First, sorry for the short notice.

 

Some of you might have heard about this already, it was mentioned in discussions and events. Most recently it was proposed officially to the board and was approved.

 

The Zephyr project launched last year with Gerrit as a code review system and it was meant to be used for this purpose until we have critical mass and a larger community. Over the last year we had a lot of feedback regarding the choice of Gerrit for the Zephyr project, most of which was negative feedback, however, I have talked to many developers and contributors who were actually happy with Gerrit, however in the 1st Zephyr mini-summit held last year in Austin we decided unanimously to move to github.com.

 

Our main goal is to have most of services we use for the project  in one place and have the best integration possible to enhance the development experience.

 

=== What does this mean for you as a Zephyr project contributor? ===

 

Starting next week, all code submissions have to be done using Pull Request on the Zephyr project github page:

 

https://github.com/zephyrproject-rtos/zephyr

 

The latest tree and the latest releases (https://github.com/zephyrproject-rtos/zephyr/releases )

will be available from Github directly.

 

Detailed documentation on how to submit changes will appear in the top-level README.md file which should also appear in the repository web page on github. For those not familiar with github, please take a look at the Pull Request method for submitting code changes:

 

https://guides.github.com/introduction/flow/

 

 

=== What else is moving? ===

 

Beside the hosting of the GIT repo and the review process using pull requests, the following will be moved to github:

 

-          Releases: https://github.com/zephyrproject-rtos/zephyr/releases

-          Wiki: We will move content from wiki.zephyrproject.org to three possible place:

o    the built-in wiki on github,

o   in some cases to the source tree in RST format

o   some content will move to the website

 

=== What about Jira? ===

 

At this moment we will continue using Jira as is, we are discussing and exploring usage of github issues, but this is still being evaluated.

 

 

=== What happens to my pending changes in gerrit? ===

 

Gerrit will be switched to operate in offline mode and all the existing changes will be available for processing and transition through May. However, we encourage you to resubmit your changes to Github and abandon changes on github. We already gone through the existing changes in github and did some initial cleanup.

 

 

Additional information and the new process for submitting changes will be documented on the github page and will be available already tomorrow. CI will also be available through github with additional details.

 

Your comments are welcome.

 

Anas

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


_______________________________________________
Zephyr-devel mailing list
Zephyr-devel@lists.zephyrproject.org
https://lists.zephyrproject.org/mailman/listinfo/zephyr-devel



Re: Migration to Github - Status update

Paul Sokolovsky
 

Hello Anas,

On Sat, 29 Apr 2017 03:20:58 +0000
"Nashif, Anas" <anas.nashif@...> wrote:

Hi,
We are in the process of enabling services on github and configuring
the various projects. To allow a smooth transition and to have Github
ready for development on May 1st globally, Gerrit was put in
read-only mode, meaning that no further changes will be accepted on
Gerrit, starting Friday evening.

If you are planning to submit anything before May, this will need to
happen in github. The following initial how-to has been drafted which
will help you with creating pull requests:

https://github.com/zephyrproject-rtos/zephyr/wiki/Contribution-Guide
I hope you guys are aware of CONTRIBUTING.md:
https://github.com/blog/1184-contributing-guidelines , so hope the
content will be moved there to give people a chance to ignore^W read
it during the submission process ;-).


Btw, I remembered about some unpleasant issue with Github pull
requests: it seems that merging via Github UI may overwrite the
original author email with whatever is set to Github's primary user
email. More specifically, I noticed that my JerryScript patches, which
were submitted from @linaro.org email, appear in the repository as
authored from my personal email.

I have suspicion that the issue either applies to "squash" submit mode,
or appeared after it it was introduced (which was relatively recently).
Well, it seems that I didn't actually proceed to investigate it
further :-(. But I definitely would like to have the right email with
Zephyr. And Github only offers 2 merge modes: 1) squash all PullReq
commits to one, perform fast-forward merge; 2) merge with explicit
non-fast-forward merge (git --no-ff), i.e. even if a patch could be
merged as fast-forward, create a merge commit.

I hope we are going to enjoy one of the benefits of Github - ability to
submit PullReq with multiple commits (and merge it like that). Then, as
can be seen, Github doesn't offer a way to merge this cleanly at all.
So, how everyone works that around is by having a special helper
script, which fetches PullReq, rebases it, and --ff-only merges it.
Hope you guys are ready to use such script.


--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

5741 - 5760 of 8632