On 22/03/2023 08:56, Beata Michalska wrote:
On Wed, Mar 15, 2023 at 02:03:40PM +0000, Teo Couprie Diaz wrote:
On 15/03/2023 12:18, Kevin Brodsky wrote:
On 15/03/2023 12:21, Beata Michalska wrote:
On Wed, Mar 15, 2023 at 10:50:11AM +0000, Teo Couprie Diaz wrote:
On 06/03/2023 11:59, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v3: - Added *_16 tests to the skip list - Added tests from kernel/security as we don't build them yet - Mentioned Morello 1.6 release v2: - Removed mentions of COMPAT - Reworded comment around Alpine - Added bpf_prog02 to the musl skips after further testing README.rst | 25 ++++ runtest/syscalls_aarch64_pcuabi.skip | 172 +++++++++++++++++++++++++++
Going through the tests in purecap it turns out we would need a skip list for it as well (`process_vm_{read,write}v` tests pass capabilities between processes through memory mappings, which is explicitly disallowed in purecap) Thus I wonder if the name I chose here is good enough ? Would it make sense to have `syscalls_aarch64_pcuabi.skip` and `syscalls_pcuabi.skip` ? Or maybe `syscalls_purecap_pcuabi.skip` ?
Given that in purecap we need to skip all those that are present in the compat skip list as it's just system issues (leapsec) and tests that aren't built, I'm not sure of the naming anymore. I'm not too keen on moving out the not built tests outside of the aarch64 skip list to an additional one.
What do you think ?
It's good this has been brought up (great timing as well as the patch was about to be merged).
Glad I didn't think about it longer then, hopefully the timing wasn't too inconvenient !
So it looks like we need 3 flavours of skip tests for different configuration, plus additional set for tests we are deliberately ignoring due to test environment limitations (leapsec) or due to build setup. I was thinking we could have 3 skip files as:
- syscalls_morello_skip - for purecap skip tests
- syscalls_morello_aarch64_skip - for plan aarch64 skip tests
I think using "morello" as a general suffix is fine, but then I would swap those two: syscalls_morello_skip for what we always skip regardless of the ABI, and then syscalls_morello_purecap_skip for what we only skip in purecap. There should be no test we skip in the standard AArch64 ABI but not in purecap.
I didn't think of using Morello, I like it. I'd agree with Kevin on the swap though: in theory if we skip something in AArch64 it should be skipped in purecap too, but not the reverse.
How that theory looks in practice ? That does sound reasonable.
For now: practice matches the theory ! The main reasons I can see the skip lists being updated is a test that turns out to be flaky in CI or if a new test is added which is broken with Musl, which should affect both AArch64 and Purecap.
Kevin
- syscalls_morello_musl_skip - for Musl related skip tests
(note the *-morello-* tagging, which might be nice to have as we have already that notion in runtest cmd files) As per those tests that are not being build or have dependencies on run env, how about adding install step to append those automatically to either syscalls_morello_skip or syscalls_morello_aarch64_skip, depending on which variant is being considered. (note the actual naming can be different)
Mmmh... It would make it cleaner in a clean repository, but at the same time it would make it a bit less explicit what tests are being skipped and why as it would end up in the build scripts.
Why we skip *_16 & *_64 is already in the build script.
You are right, in the sense that we skip them because we don't build them it's already there. (Aside : why don't we test with the *_64 variants ? At least in compat the few that I somehow ran seemed OK)
We could always add some documentation detailing what isn't built and why, which could help. There's also the fact that you would need the AArch64 skip list in purecap as mentioned above. But I guess while we're doing build time shenanigans we could append it to the purecap list as well.
Unless I got things wrong, we want to have single skip list for both: plain aarch64 and purecap so that would only mean adding the non-built tests to skip list despite the build flavor. So we would end up with syscalls_morello_skip being partially auto-populated by build step.
I might have misunderstood you or just got lost in my train of thought: indeed if we do that we would only need to populate syscalls_morello_skip, which would then be used during both compat and purecap tests.
I guess I have an idea of how this could be done, although I am having trouble with it currently. I will continue to investigate further.
Instead of having a list of *_16 and *_64 we could use the make targets to clear those entries from syscalls list ?
I'm not too sure on removing the entries from the syscalls list : it's an additional step if you want to do anything with them and you lose track that they are part of the "regular" list but are not run here. Maybe it's not that important, I don't really know.
You are right here - modifying the 'syscalls' list is not the best approach, I agree. Either way we are to end up with 3 skip files:
- syscalls_morello_skip : as suggested to cover all tests that are not being build + leapsec
- syscalls_morello_purecap_skip : for tests that are invalid for purecap
- syscalls_morello_musl_skip: for tests that need either Musl or LTP changes (to be applied for both: plain aarch64 and purecap ? )
I agree, that's how I see things working out currently. The musl skip list would currently be used for both, indeed.
Thanks for replying, Best regards Téo
BR B.
BR B.
Thanks both, T�o
runtest/syscalls_musl.skip | 18 +++ 3 files changed, 215 insertions(+) create mode 100644 runtest/syscalls_aarch64_pcuabi.skip create mode 100644 runtest/syscalls_musl.skipThanks,
T�o
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org