Re: RFC: extend sanitycheck testcase filtering expressiveness

Wang, Jing J

QA test suite actually has similar requirement for test case filtering by expression. We took a lightweight approach, directly use "python expression" instead of creating a new one. The benefit of "python expression" is, the expression can be parsed by python eval() function directly. Thus, below filter expression is naturally supported with a small code snippets.
@tag_exp('soc not in ["quark_se", "quark_d2000"]')
def test_xxxx(self):


-----Original Message-----
From: Dmitriy Korovkin [mailto:dmitriy.korovkin(a)]
Sent: Tuesday, March 29, 2016 4:23 AM
To: Boie, Andrew P <andrew.p.boie(a)>
Cc: devel(a)
Subject: [devel] Re: RFC: extend sanitycheck testcase filtering expressiveness

On Fri, 25 Mar 2016, Boie, Andrew P wrote:

Proposed solution:

We need a more expressive language for filtering test cases. I propose
a simple boolean expression language with the following grammar:

expression ::= expression "and" expression
| expression "or" expression
| "not" expression
| "(" expression ")"
| symbol "==" constant
| symbol "!=" constant
| symbol "<" number
| symbol ">" number
| symbol ">=" number
| symbol "<=" number
| symbol "in" list
| symbol

list ::= "[" list_contents "]"

list_contents ::= constant
| list_contents "," constant

constant ::= number
| string

When symbols are encountered, they are looked up in an environment
dictionary. In sanitycheck the environment will initially consist of:

ARCH : <architecture>,
PLATFORM : <platform>,
<all CONFIG_* key/value pairs in the test's generated defconfig>

We can later augment this with additional interesting metadata if we
want. For example I was thinking of ways we could integrate some
footprint information for large tests.

For the case where

expression ::= symbol

it evaluates to true if the symbol is defined to a non-empty string.

For all comparison operators, if the config symbol is undefined, it
will be treated as a 0 (for > < >= <=) or an empty string "" (for == != in).
For numerical comparisons it doesn't matter if the environment stores
the value as an integer or string, it will be cast appropriately.

Operator precedence, starting from lowest to highest:

or (left associative)
and (left associative)
not (right associative)
all comparison operators (non-associative)

In testcase.ini the 'config_whitelist' directive will be removed and
replaced with 'filter' directives containing one of these expressions.
arch_whitelist, arch_exclude, platform_whitelist, platform_exclude are
all syntactic sugar for these expressions. For instance

arch_exclude = x86 arc

Is the same as:

filter = not ARCH in ["x86", "arc"]
I see nothing bad in the proposal.

Implementation details:

Writing a parser by hand is a pain and prone to bugs. The best way to
do a language like this is to use a LR parser generator like lex/yacc.
There exists an open source library called PLY which does lex/yacc for
Python, it has been around for a long time and works very well:

The sample implementation I have drops a copy of PLY directly in the
Zephyr source tree since its license is compatible and this will
result in the least pain for users as they won't have to do anything.
If this is unacceptable we can either find a way to stick it in the
SDK, or add it to the list of workstation dependencies (either have
the user install with pip or their distribution's package manager.
Fedora appears to have PLY installed by default).
I would refrain from copying PLY into the source tree for the following
- Support. Updating the library, which is not the part of the
project may be a headache.
- Licensing. There may be issues with distributing with Zephyr the code
that we have not developed.
- Importing. In case of side projects that, for instance, run all
sanity_chk projects on real hardware, using sanity_chk combined with
additional library may create problems.
- Distributions. Ubuntu has PLY 3.4 as a part of the distribution. As you
have mentioned, Fedora has it as well.

Join to automatically receive all group messages.