On Thu, 15 Jun 2017 10:46:29 +0300
Jukka Rissanen <jukka.rissanen@...> wrote:

[]

as part of sanitycheck testsuite! But, 8 tests of tests/net/ have
build_only=true, any wonder they're broken?

When we had gerrit and jenkins, some of the net tests run slightly
longer that what was desired, so they were marked as build only. Now
that situation is different with github and shippable, we can change
this. So I will prepare a patch that activates those tests that can be
activated.

That explains it, thanks.

[]

All other tests programs (24 pieces), that consists of quite many
individual tests, are run automatically by CI, so the situation is not
so bleak as you indicated here.

Great, thanks for clarifying this. Though I hope you'd agree that seeing
such tests as "context" or "tcp" fail does lead to concerns.

I will fix the relevant failing tests as they have bit rotted after
the tests were written.

Nice, thanks for finding time for this!

Converting two mqtt tests to not use real qemu
requires a bit more work.

[]

One would think that if we have 22 repetitive implementations of
test interfaces, whose main purpose is to be just loopback
interfaces,

No, the interface is not a loopback interface although it might look
like that. The purpose of the interface that is created in each of the
test is to simulate a real network so that we do not have to connect
to outside world but the test is self contained. So it kind of looks
like a loopback interface but in this case the source and destination
IP addresses are not the same (as would be the case with loopback
interface), as typically we want to test some real behavior of the
system so src/dest addresses should differ.

I see. I may imagine they offer more functionality than just a loopback
interface. I also may imagine that each of 22 test device
implementations are slightly different to cater for particular
testcases. Nothing of that help someone wanting to write a new test
unfortunately. How would one know that there's no standard device
implementation for dependency-free testing, and having figured that,
how one would choose which of 22 cases to use as a template?

The loopback support has limited use cases actually and we probably
need to make that optional (behind Kconfig option) in the code as
normally there should be no need to send anything back to itself in
the real world.

I absolutely agree that loopback device would be mostly useful for
development and testing, not for production. It also offers limited
testing indeed. But it has one big advantage: tests using it can be
easily run using existing sanitycheck framework. And if loopback
device existed in the main code, such tests would be also easy to write,
unlike now. Summing up, I'd like to give a try to implement one for the
mainline.

then
we'd have a loopback interface in the main codebase. But nope, as
confirmed by Jukka on IRC, we don't.

Summary:

1. Writing networking tests is hard, but it Zephyr, it takes
extraordinary, agonizing effort. The most annoying is that all
needed pieces are there, but instead of presenting a nice picture,
they form a mess which greets you with crashes if you try to change
anything.

I am not sure what kind of mess you mean here but patches are welcome
as always to rectify this.

Well, patches alone won't help here. Writing tests is always hard (for
various reasons, including "tests are code, so why not write 'real'
code instead?"). So, it would be nice to think how to facilitate that.
Specific proposal is to add a loopback netif, I assume it's ok, so will
go for a patch.

[]

there's
something wrong with my system, and yep, I'd be glad to know what
I'm still don't do right with Zephyr after working on it for a
year.)

Hmm, I missed the point of your last sentence.

Well, it's the same issue: various things in Zephyr are "harder than
necessary", so one can never know if something really broke or one
doesn't do all the needed things to run it successfully. It would be
nice to think of making the default config of Zephyr either run OOB, or
fail with clear error messages, not crash or lockup. That's again a big
meta-task, not something which can be "fixed with a patch", but would
be nice to see if different stakeholders of Zephyr agree that there's
an issue which needs attention.



Thanks for all the replies!

--
Best Regards,
Paul

Linaro.org | Open source software for ARM SoCs
Follow Linaro: http://www.facebook.com/pages/Linaro
http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog