Re: RFC: extend sanitycheck testcase filtering expressiveness


Dmitriy Korovkin
 

On Fri, 25 Mar 2016, Boie, Andrew P wrote:

Proposed solution:

We need a more expressive language for filtering test cases. I propose a
simple boolean expression language with the following grammar:

expression ::= expression "and" expression
| expression "or" expression
| "not" expression
| "(" expression ")"
| symbol "==" constant
| symbol "!=" constant
| symbol "<" number
| symbol ">" number
| symbol ">=" number
| symbol "<=" number
| symbol "in" list
| symbol

list ::= "[" list_contents "]"

list_contents ::= constant
| list_contents "," constant

constant ::= number
| string

When symbols are encountered, they are looked up in an environment
dictionary. In sanitycheck the environment will initially consist of:

{
ARCH : <architecture>,
PLATFORM : <platform>,
<all CONFIG_* key/value pairs in the test's generated defconfig>
}

We can later augment this with additional interesting metadata if we
want. For example I was thinking of ways we could integrate some
footprint information for large tests.

For the case where

expression ::= symbol

it evaluates to true if the symbol is defined to a non-empty string.

For all comparison operators, if the config symbol is undefined, it will
be treated as a 0 (for > < >= <=) or an empty string "" (for == != in).
For numerical comparisons it doesn't matter if the environment stores
the value as an integer or string, it will be cast appropriately.

Operator precedence, starting from lowest to highest:

or (left associative)
and (left associative)
not (right associative)
all comparison operators (non-associative)

In testcase.ini the 'config_whitelist' directive will be removed and
replaced with 'filter' directives containing one of these expressions.
arch_whitelist, arch_exclude, platform_whitelist, platform_exclude
are all syntactic sugar for these expressions. For instance

arch_exclude = x86 arc

Is the same as:

filter = not ARCH in ["x86", "arc"]
I see nothing bad in the proposal.

Implementation details:

Writing a parser by hand is a pain and prone to bugs. The best way to do
a language like this is to use a LR parser generator like lex/yacc.
There exists an open source library called PLY which does lex/yacc for
Python, it has been around for a long time and works very well:

http://www.dabeaz.com/ply/index.html

The sample implementation I have drops a copy of PLY directly in the
Zephyr source tree since its license is compatible and this will result
in the least pain for users as they won't have to do anything. If this
is unacceptable we can either find a way to stick it in the SDK, or add
it to the list of workstation dependencies (either have the user install
with pip or their distribution's package manager. Fedora appears to have
PLY installed by default).
I would refrain from copying PLY into the source tree for the following
reasons:
- Support. Updating the library, which is not the part of the
project may be a headache.
- Licensing. There may be issues with distributing with Zephyr the code
that we have not developed.
- Importing. In case of side projects that, for instance, run all
sanity_chk projects on real hardware, using sanity_chk combined with
additional library may create problems.
- Distributions. Ubuntu has PLY 3.4 as a part of the distribution. As you
have mentioned, Fedora has it as well.
Regards,
/Dmitriy

Join devel@lists.zephyrproject.org to automatically receive all group messages.