Re: RFC: BSD Socket (like) API


Luiz Augusto von Dentz
 

Hi Paul,

On Wed, Mar 29, 2017 at 1:31 AM, Paul Sokolovsky
<paul.sokolovsky@linaro.org> wrote:
Hello Luiz Augusto,

On Tue, 28 Mar 2017 23:34:25 +0300
Luiz Augusto von Dentz <luiz.dentz@gmail.com> wrote:

[]

net_context). (We might need a queue per context to properly
emulate the read() API, though, so the reference might not be
exactly to a net_context pool, but the same idea applies.)
Indeed we would have to put a queue per net_context which quickly adds
up to the memory footprint,
As I point in the reply to Leandro, these additions would be #ifdef'ed
on CONFIG_NET_BSD_SOCKETS or something, so application developers will
have a choice whether save memory or have more familiar API. Putting
the queue, etc. into yet different structure (and waste some memory to
link this new structure to net_context) is yet another choice.

luckily we can use k_poll and avoid having
extra threads for each context.
Note that the work I'm going to do is based on the previous experience
with implement BSD-Sockets like pull-style API on top of push-style
native ("raw") lwIP API. There, we were able to achieve that without
any threads (indeed, even with a cooperative RTOS underneath). So, no
planning of having extra threads on the initial stage of work
(push-to-pull "impedance conversion"). And presence of k_poll makes me
positive that we'll be able to implement even poll() or epoll() without
them (but that's 2nd stage of work).

But I guess the fundamental problem is
this is probably intended to interface with real BSD socket/posix
applications/components, which I guess will not be enough for most, so
at the end will it be worth adding this code? What are the things we
want to port on top, can someone present at the mini-summit make a
list?
As my initial RFC message pointed, the initial target is MicroPython
with its "usocket" (micro-socket) module. That's a single app, but it
wraps a large chunk of BSD Sockets API for Python language. And of
course, this RFC is to see what other projects people are interested to
port.
I suppose you need much more than sockets to make a python program to
work on top of zephyr. Now I do agree having python would be very
useful to prototype, or js, but for products it might be a different
story.

Casting a pointer to an integer as suggested might work, but will
most likely be an issue with the common pattern of checking for
error by just comparing the sign of the return value, instead of
the (technically correct) comparison with -1.
We can check if there are any paddings on net_context, if there is it
might not be a bad idea to add an id to them, or really make them
compatible with k_poll.

In general Im much more concern about the stack fragmentation, though.
Adding compatibility layers and offloading works against the stack
itself,
Well, I don't see how proper, configurable (on/off) layering works
against the stack. I'd say, vice-versa, it allows to leverage it for
more usecases. Offloading, let me skip that part ;-).
I was referring to APIs like net_buf or application layer protocols
like coap, while I agree we can make all configurable that doesn't
dismiss the fact these interfaces might be reimplemented on top of the
socket layer. In fact the case of net_buf there is probably no
solution since the BSD Socket interface will most likely be using
plain pointers or iovec, so we lose zero copy and possible add yet
another buffer pool on top.

as they either eat the available memory for enabling features
in the stack
Let users select whether they want to save memory or development
effort by using a conventional API ;-).

or reimplement part of the stack in other layers,
Well, here I, as present at the mini-summit, can quote what Anas said
on this: he literally said - don't do that. If "higher level wrapper"
needs to reimplement or otherwise put upside-down what native layer
does, let's just rework native layer to be (more) POSIX/etc.
compatible. And well, that's pretty much how we already do it, with
several features for 1.7 done to make it closer to POSIX spirit, and
there're more such things in queue (here's the latest merged:
https://gerrit.zephyrproject.org/r/12455). So, no worries, or rather,
we all are aware of the problem, and ready to do homework on it.

not to mention this takes a lot of time that could be spend in other
areas like security and proper accelerators for heavy tasks like for
example crypto.
As a maintainer in other projects, I can easily relate to that thought
(that I, a maintainer, know better what contributors rather be working
on ;-) ). But here's how it works at Linaro: our aim is to reduce/avoid
fragmentation among ARM vendors (and not at the expense of more
fragmentation with other architectures). So if a member comes to us and
says "we'll use Zephyr if ...", we'd better listen, because there's no
lack of alternatives to Zephyr, and if interested parties select
something else instead, there won't be anything good from that to
Zephyr, or to the industry at all, which will continue in a chaos of
several dozens of adhoc (vs full-featured) RTOSes. And yeah, BSD
Sockets is a frequent request we heard for a while. Other things you
mention are in the plans (ours including) too.
That is a fair point, but ultimately the memory constraint comes from
the very same vendors. So are there new boards coming with more
memory? 128 K or more ram and a fair bit more flash? Because I can
tell you in most boards memory is already quite tight and that is
without debug enable, in fact echo server don't even work with qemu
when all debugs options are enabled:

/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/libexec/i586-zephyr-elf/gcc/i586-zephyr-elf/6.2.0/real-ld:
zephyr_prebuilt.elf section `noinit' will not fit in region `RAM'
/opt/zephyr-sdk/sysroots/x86_64-pokysdk-linux/usr/libexec/i586-zephyr-elf/gcc/i586-zephyr-elf/6.2.0/real-ld:
region `RAM' overflowed by 5520 bytes


--
Luiz Augusto von Dentz

Join devel@lists.zephyrproject.org to automatically receive all group messages.