Some thoughts on Zephyr Console API


Paul Sokolovsky
 

Hello,

I would like to share some thoughts/woes with using Zephyr console API,
based on the experience with (initial) porting of two console
application to Zephyr, both are scripting languages, JerryScript and
MicroPython. As those are scripting languages, interactive prompt (aka
REPL) play important role for them.

Both JrS and uPy are intended for embedded usage, and both share a
similar architecture, where there's "unix" port, which can serve as a
demo, and as a starting port for a port to specific RTOS/hardware. Both
of these projects for reading a console input require classical
K&R-style getchar()-like function, with a semantics of "read a single
char from console, block if not available". Alternatively, they can
interpret a whole line at once, that's classical gets() with semantics
of "block until whole line from console is available, then return it".
Using gets()-like call has a benefit that line-editing capabilities can
be provided, and transparent to an application.

Let's see what effort was required to port them to Zephyr:

1. JerryScript port was originally done by Sergio Rodriguez (Zephyr
team member or at least member of another team in Intel which uses
Zephyr closely). Instead of pull-style gets(), it uses push-style
whole line delivery via callback from Zephyr shell subsystem. Works
well, and there's line editing, completion support offered by Zephyr
shell, which is really nice. And while not as easy as a single gets(),
set up of it is pretty easy and high-level. And it works nice, until
you try to enter a line with at least 10 spaces, e.g.:

for (var i = 0; i < 10000; i++) { a.push("foo"); }

That ends up with "Too many parameters (max 10)" error. That's because
shell unconditionally tokenizes an input lines into space-separate
tokens, and the default (non-configurable) limit is 10. And of course,
JerryScript then needs to join them back.

2. With MicroPython, I decided to follow "getchar" approach, especially
that uPy has an own readline-like implementation with more capabilities
then Zephyr. So, I started to look how to make Zephyr pass me input
chars. I saw drivers/console/uart_pipe.c, and thought that's the best
approach, as it allows the leanest way to do that. Unfortunately,
playing with for few hours, I couldn't make it work. I then followed
another lead - using uart_irq_input_hook. With some extra effort I was
able to make it work, and created a standalone module which just
exposes zephyr_getchar() function:
https://github.com/pfalcon/zephyr_getchar


Conclusions:

Getting console input with Zephyr is *HARD*. I originally reported
an issue #1 above as https://jira.zephyrproject.org/browse/ZEP-532 ,
and suggested that I could fork console_handler_shell.c, remove
tokenizing and would get what I want. But arguably, that's a lot of
work to just gets() which is just there on a lot of other systems.
However, with "getchar" trial, I essentially did just that - coded
simple, but not exactly trivial code on top of what Zephyr.

So, I guess I could do similar for "gets" too, but one problem is that
it unlikely to help many other people - it really would be nice to have
that in the main Zephyr repo, and engineered properly (for example,
zephyr_getchar module above isn't compatible with Zephyr shell).

Would that make sense for Zephyr? Requirements would be simple:

1. Add ability to receive a single char from a console; also, a single
line (with editing and other niceties).

2. Push-style interface with a callback is probably OK, but having
pull-style interface would make porting a lot of software easier. And
with a push-style, a multiple consumers problem immediately manifests
itself.


Thanks,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog


Flavio Santes <flavio.santes@...>
 

Hello Paul,

Zephyr shell is useful for writing sample code, and it works fine if you just want to create prototypes. However, you are right, the shell is not quite useful for general interactive input processing.
Running JS, Lua or Python code embedded into a Zephyr application sounds great. Even so, I do not see why more interactive functionality is required. Perhaps I am missing something, can you provide a use-case?

Regards,
Flavio


Paul Sokolovsky
 

Hello Flavio,

On Sun, 24 Jul 2016 00:58:45 -0000
"Flavio Santes" <flavio.santes(a)intel.com> wrote:

Hello Paul,

Zephyr shell is useful for writing sample code, and it works fine if
you just want to create prototypes. However, you are right, the shell
is not quite useful for general interactive input processing. Running
JS, Lua or Python code embedded into a Zephyr application sounds
great. Even so, I do not see why more interactive functionality is
required. Perhaps I am missing something, can you provide a use-case?
Thanks for your reply, and sorry for dropping the ball from my side -
vacation time and other tasks. With all the arguments regarding Zephyr
shell, I agree that it's mistake to try to use for generic console
input, and going to close https://jira.zephyrproject.org/browse/ZEP-532
as invalid.

I'm not sure what you mean by providing a use-case - in the original
mail, I gave 2 examples: porting JerryScript's and MicroPython's REPLs.
Perhaps, you mean why it's important to support REPL (interactive
prompt) for these systems? Well, JS/Lua/Python are interpreted
interactive languages (some less, some more, but every has basic REPL
for sure).

As long as you agree that prospect of running JS/Lua/Python in a deeply
embedded systems sounds interesting, trying to cut off REPL
functionality (and leave e.g. only precompiled bytecode execution)
means cutting off their functionality in half. Perhaps, REPL isn't
useful to ship finished products to classical end users (though
nowadays "makers" emerged as noticeable consumer group), but it's
definitely useful for development/debugging and initial acquaintance
with a system. And even the latter alone is very important - in the
current overcrowded landscape, making it easy for a customer to get
one's initial blinking-led demo up and running fast is what can
influence selection of one product over others.

Hope that wasn't too long and philosophical ;-). I figure, this is a
niche topic for now, and there're many more priority tasks to do in
Zephyr. So, I'm happy that I at least brought up this topic for future
consideration, though now it seems that some adhoc (but reusable)
out-of-tree solution is the way to go. I shaped that up in a
github repo: https://github.com/pfalcon/zephyr_console_helpers .


Thanks for the discussion!



Regards,
Flavio


--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog


Flavio Santes <flavio.santes@...>
 

Hello Paul,

I was thinking about this a few days ago. It seems fair to ask for that functionality considering your view points. This feature could be very useful for rapid prototyping, on-the-fly updates and debugging, as you comment. So, as soon the user base increases, I think more developers will ask for something similar!

Regards,
Flavio

Hello Flavio,

On Sun, 24 Jul 2016 00:58:45 -0000
"Flavio Santes" <flavio.santes(a)intel.com&gt; wrote:


Thanks for your reply, and sorry for dropping the ball from my side -
vacation time and other tasks. With all the arguments regarding Zephyr
shell, I agree that it's mistake to try to use for generic console
input, and going to close https://jira.zephyrproject.org/browse/ZEP-532
as invalid.

I'm not sure what you mean by providing a use-case - in the original
mail, I gave 2 examples: porting JerryScript's and MicroPython's REPLs.
Perhaps, you mean why it's important to support REPL (interactive
prompt) for these systems? Well, JS/Lua/Python are interpreted
interactive languages (some less, some more, but every has basic REPL
for sure).

As long as you agree that prospect of running JS/Lua/Python in a deeply
embedded systems sounds interesting, trying to cut off REPL
functionality (and leave e.g. only precompiled bytecode execution)
means cutting off their functionality in half. Perhaps, REPL isn't
useful to ship finished products to classical end users (though
nowadays "makers" emerged as noticeable consumer group), but it's
definitely useful for development/debugging and initial acquaintance
with a system. And even the latter alone is very important - in the
current overcrowded landscape, making it easy for a customer to get
one's initial blinking-led demo up and running fast is what can
influence selection of one product over others.

Hope that wasn't too long and philosophical ;-). I figure, this is a
niche topic for now, and there're many more priority tasks to do in
Zephyr. So, I'm happy that I at least brought up this topic for future
consideration, though now it seems that some adhoc (but reusable)
out-of-tree solution is the way to go. I shaped that up in a
github repo: https://github.com/pfalcon/zephyr_console_helpers .


Thanks for the discussion!


Martinez Rodriguez, Sergio <sergio.martinez.rodriguez@...>
 

Hi Flavio,

I wrote a shell based on the original one that uses a similar philosophy to what Paul is describing.
This shell is for zephyr.js to be able to update the javascript code dynamically, execute commands and run JS dynamically.

On this shell you can register callbacks to get the data in raw for later processing
Or just get the raw line so you can do any operation yourself.
https://github.com/jslaunchbox/ihex_uploader/

You can see our use case on this short video working on Arduino 101
https://www.youtube.com/watch?v=L2pF48BOPBc

In the video the shell on the right is the regular shell from Zephyr running in parallel with the shell to run Javascript commands.
You can see how I evaluate Javascript and run commands on that terminal.

The application can register a process to capture data streams and process them in different ways.

My use case for this is:
Maybe you need a Intel HEX parser...
Maybe you need a JSON parser...
Maybe you want to talk to the Bluetooth interface...
...
So you can register dynamically the callbacks for the parser.

Sergio

-----Original Message-----
From: Flavio Santes [mailto:flavio.santes(a)intel.com]
Sent: Wednesday, August 10, 2016 5:13 PM
To: devel(a)lists.zephyrproject.org
Subject: [devel] Re: Re: Re: Some thoughts on Zephyr Console API

Hello Paul,

I was thinking about this a few days ago. It seems fair to ask for that functionality considering your view points. This feature could be very useful for rapid prototyping, on-the-fly updates and debugging, as you comment. So, as soon the user base increases, I think more developers will ask for something similar!

Regards,
Flavio

Hello Flavio,

On Sun, 24 Jul 2016 00:58:45 -0000
"Flavio Santes" <flavio.santes(a)intel.com&gt; wrote:


Thanks for your reply, and sorry for dropping the ball from my side -
vacation time and other tasks. With all the arguments regarding Zephyr
shell, I agree that it's mistake to try to use for generic console
input, and going to close
https://jira.zephyrproject.org/browse/ZEP-532
as invalid.

I'm not sure what you mean by providing a use-case - in the original
mail, I gave 2 examples: porting JerryScript's and MicroPython's REPLs.
Perhaps, you mean why it's important to support REPL (interactive
prompt) for these systems? Well, JS/Lua/Python are interpreted
interactive languages (some less, some more, but every has basic REPL
for sure).

As long as you agree that prospect of running JS/Lua/Python in a
deeply embedded systems sounds interesting, trying to cut off REPL
functionality (and leave e.g. only precompiled bytecode execution)
means cutting off their functionality in half. Perhaps, REPL isn't
useful to ship finished products to classical end users (though
nowadays "makers" emerged as noticeable consumer group), but it's
definitely useful for development/debugging and initial acquaintance
with a system. And even the latter alone is very important - in the
current overcrowded landscape, making it easy for a customer to get
one's initial blinking-led demo up and running fast is what can
influence selection of one product over others.

Hope that wasn't too long and philosophical ;-). I figure, this is a
niche topic for now, and there're many more priority tasks to do in
Zephyr. So, I'm happy that I at least brought up this topic for future
consideration, though now it seems that some adhoc (but reusable)
out-of-tree solution is the way to go. I shaped that up in a github
repo: https://github.com/pfalcon/zephyr_console_helpers .


Thanks for the discussion!


Flavio Santes <flavio.santes@...>
 

Hello Sergio,

Hi Flavio,

I wrote a shell based on the original one that uses a similar philosophy to what Paul is
describing.
This shell is for zephyr.js to be able to update the javascript code dynamically, execute
commands and run JS dynamically.

On this shell you can register callbacks to get the data in raw for later processing
Or just get the raw line so you can do any operation yourself.
https://github.com/jslaunchbox/ihex_uploader/

You can see our use case on this short video working on Arduino 101
https://www.youtube.com/watch?v=L2pF48BOPBc

In the video the shell on the right is the regular shell from Zephyr running in parallel
with the shell to run Javascript commands.
You can see how I evaluate Javascript and run commands on that terminal.
After a quick reading of the git hub repo, it seems you have a very good prototype with a lot of features. So, I wonder if this kind of applications could become a tool for more complex tasks. This use-case is new for me, so perhaps I am missing the big picture here.

The application can register a process to capture data streams and process them in
different ways.

My use case for this is:
Maybe you need a Intel HEX parser...
Maybe you need a JSON parser...
Maybe you want to talk to the Bluetooth interface...
...
So you can register dynamically the callbacks for the parser.

Sergio
Regards,
Flavio

-----Original Message-----
From: Flavio Santes [mailto:flavio.santes´╝áintel.com]
Sent: Wednesday, August 10, 2016 5:13 PM
To: devel(a)lists.zephyrproject.org
Subject: [devel] Re: Re: Re: Some thoughts on Zephyr Console API

Hello Paul,

I was thinking about this a few days ago. It seems fair to ask for that functionality
considering your view points. This feature could be very useful for rapid prototyping,
on-the-fly updates and debugging, as you comment. So, as soon the user base increases, I
think more developers will ask for something similar!

Regards,
Flavio