Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com --- v3: - Added *_16 tests to the skip list - Added tests from kernel/security as we don't build them yet - Mentioned Morello 1.6 release v2: - Removed mentions of COMPAT - Reworded comment around Alpine - Added bpf_prog02 to the musl skips after further testing README.rst | 25 ++++ runtest/syscalls_aarch64_pcuabi.skip | 172 +++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 18 +++ 3 files changed, 215 insertions(+) create mode 100644 runtest/syscalls_aarch64_pcuabi.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a7769d0..ef1282f2b93f 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,31 @@ Running tests:
runltp -f morello_transitional,morello_transitional_extended
+Skipping tests: +--------------- + +In certain cases, you might want to skip tests that are known to be failing or are yet to be supported. + +This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``. + +We currently provide the following skip files under ``runtest/``: + - ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system. + - ``syscalls_aarch64_pcuabi.skip``: Skips all tests not considered while testing plain Aarch64 + with a PCuABI kernel. + +``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution. + +For example, in plain Aarch64 on a Musl system with a PCuABI kernel to test all syscalls but skip all +failures unrelated to the PCuABI kernel, one could run the following command: + +.. code-block:: + + ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip ) + +Using those two skip lists should result in a PASS on Morello release 1.6 and after, as multiple patches +to Musl and the Linux kernel are needed on top of the 1.5 release.
Developers playground --------------------- diff --git a/runtest/syscalls_aarch64_pcuabi.skip b/runtest/syscalls_aarch64_pcuabi.skip new file mode 100644 index 000000000000..2097341de32b --- /dev/null +++ b/runtest/syscalls_aarch64_pcuabi.skip @@ -0,0 +1,172 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in plain Aarch64. +# LINUX - Depends on kernel release > morello-release-1.5.0 + +# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01 + +# We do not build tests under kernel/security yet, skip them for now. +prot_hsymlinks +dirtyc0w +dirtypipe + +# We do not build *_16 tests, skip them. +chown01_16 +chown02_16 +chown03_16 +chown04_16 +chown05_16 +fchown01_16 +fchown02_16 +fchown03_16 +fchown04_16 +fchown05_16 +getegid01_16 +getegid02_16 +geteuid01_16 +geteuid02_16 +getgid01_16 +getgid03_16 +getgroups01_16 +getgroups03_16 +getresgid01_16 +getresgid02_16 +getresgid03_16 +getresuid01_16 +getresuid02_16 +getresuid03_16 +getuid01_16 +getuid03_16 +lchown01_16 +lchown02_16 +lchown03_16 +setfsgid01_16 +setfsgid02_16 +setfsgid03_16 +setfsuid01_16 +setfsuid02_16 +setfsuid03_16 +setfsuid04_16 +setgid01_16 +setgid02_16 +setgid03_16 +setgroups01_16 +setgroups02_16 +setgroups03_16 +setgroups04_16 +setregid01_16 +setregid02_16 +setregid03_16 +setregid04_16 +setresgid01_16 +setresgid02_16 +setresgid03_16 +setresgid04_16 +setresuid01_16 +setresuid02_16 +setresuid03_16 +setresuid04_16 +setresuid05_16 +setreuid01_16 +setreuid02_16 +setreuid03_16 +setreuid04_16 +setreuid05_16 +setreuid06_16 +setreuid07_16 +setuid01_16 +setuid03_16 +setuid04_16 + +# We do not build *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +stat04_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64 diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000000..aa3dff6c9449 --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,18 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +# MUSL - Depends on Musl release > morello-release-1.5.0 +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01
Hi Teo,
On Mon, Mar 06, 2023 at 11:59:10AM +0000, Teo Couprie Diaz wrote:
It's minor, but it might be better (for readability) to skip the sentence below altogether: the purpose of skip files has already been described above and I guess we only want to present a way on how to use both skip files for a single run (?),
I might have missed smth but was there a conclusion on this approach potentially failing on some distros ?
+Using those two skip lists should result in a PASS on Morello release 1.6 and after, as multiple patches +to Musl and the Linux kernel are needed on top of the 1.5 release.
How about: In order for the skip files to successfully filter-out expected/known failures, morello-release-1.6.0 release, or newer is required.
(?)
Minor, but we did use the patter below earlier, so it might be good to stay consistent (?): #KERN - *: depends on Morello Linux kernel release > morello-release-1.5.0
I can apply the changes (if any) while merging the patches.
--- BR B.
Hi Beata,
On 09/03/2023 13:33, Beata Michalska wrote:
Mh, yeah I get what you mean. I feel that's fair, and it would follow directly after mentioning process substitution.
I did some more testing and could not reproduce Kevin's issue, which in itself was weird (it worked, but not in the script ?). Since he is on leave he hasn't had a chance to provide more details. I'm fine with waiting for a definitive answer, but I feel this could be easily fixed if needed later.
I like your phrasing better, I'm happy with replacing it.
Ah right, it would be better to stay consistent for sure.
I can apply the changes (if any) while merging the patches.
Happy with you applying everything you mentioned if you decide to merge !
Thanks for the review, Best regards Téo
On 06/03/2023 11:59, Teo Couprie Diaz wrote:
Going through the tests in purecap it turns out we would need a skip list for it as well (`process_vm_{read,write}v` tests pass capabilities between processes through memory mappings, which is explicitly disallowed in purecap) Thus I wonder if the name I chose here is good enough ? Would it make sense to have `syscalls_aarch64_pcuabi.skip` and `syscalls_pcuabi.skip` ? Or maybe `syscalls_purecap_pcuabi.skip` ?
Given that in purecap we need to skip all those that are present in the compat skip list as it's just system issues (leapsec) and tests that aren't built, I'm not sure of the naming anymore. I'm not too keen on moving out the not built tests outside of the aarch64 skip list to an additional one.
What do you think ?
Thanks,
Téo
On Wed, Mar 15, 2023 at 10:50:11AM +0000, Teo Couprie Diaz wrote:
It's good this has been brought up (great timing as well as the patch was about to be merged). So it looks like we need 3 flavours of skip tests for different configuration, plus additional set for tests we are deliberately ignoring due to test environment limitations (leapsec) or due to build setup. I was thinking we could have 3 skip files as: - syscalls_morello_skip - for purecap skip tests - syscalls_morello_aarch64_skip - for plan aarch64 skip tests - syscalls_morello_musl_skip - for Musl related skip tests
(note the *-morello-* tagging, which might be nice to have as we have already that notion in runtest cmd files) As per those tests that are not being build or have dependencies on run env, how about adding install step to append those automatically to either syscalls_morello_skip or syscalls_morello_aarch64_skip, depending on which variant is being considered. (note the actual naming can be different) Instead of having a list of *_16 and *_64 we could use the make targets to clear those entries from syscalls list ?
--- BR B.
On 15/03/2023 12:21, Beata Michalska wrote:
I think using "morello" as a general suffix is fine, but then I would swap those two: syscalls_morello_skip for what we always skip regardless of the ABI, and then syscalls_morello_purecap_skip for what we only skip in purecap. There should be no test we skip in the standard AArch64 ABI but not in purecap.
Kevin
On 15/03/2023 12:18, Kevin Brodsky wrote:
Glad I didn't think about it longer then, hopefully the timing wasn't too inconvenient !
I didn't think of using Morello, I like it. I'd agree with Kevin on the swap though: in theory if we skip something in AArch64 it should be skipped in purecap too, but not the reverse.
Mmmh... It would make it cleaner in a clean repository, but at the same time it would make it a bit less explicit what tests are being skipped and why as it would end up in the build scripts. We could always add some documentation detailing what isn't built and why, which could help. There's also the fact that you would need the AArch64 skip list in purecap as mentioned above. But I guess while we're doing build time shenanigans we could append it to the purecap list as well.
Instead of having a list of *_16 and *_64 we could use the make targets to clear those entries from syscalls list ?
I'm not too sure on removing the entries from the syscalls list : it's an additional step if you want to do anything with them and you lose track that they are part of the "regular" list but are not run here. Maybe it's not that important, I don't really know.
Thanks both, Téo
On Wed, Mar 15, 2023 at 02:03:40PM +0000, Teo Couprie Diaz wrote:
How that theory looks in practice ? That does sound reasonable.
Why we skip *_16 & *_64 is already in the build script.
Unless I got things wrong, we want to have single skip list for both: plain aarch64 and purecap so that would only mean adding the non-built tests to skip list despite the build flavor. So we would end up with syscalls_morello_skip being partially auto-populated by build step.
You are right here - modifying the 'syscalls' list is not the best approach, I agree. Either way we are to end up with 3 skip files: - syscalls_morello_skip : as suggested to cover all tests that are not being build + leapsec - syscalls_morello_purecap_skip : for tests that are invalid for purecap - syscalls_morello_musl_skip: for tests that need either Musl or LTP changes (to be applied for both: plain aarch64 and purecap ? )
--- BR B.
On 22/03/2023 08:56, Beata Michalska wrote:
For now: practice matches the theory ! The main reasons I can see the skip lists being updated is a test that turns out to be flaky in CI or if a new test is added which is broken with Musl, which should affect both AArch64 and Purecap.
You are right, in the sense that we skip them because we don't build them it's already there. (Aside : why don't we test with the *_64 variants ? At least in compat the few that I somehow ran seemed OK)
I might have misunderstood you or just got lost in my train of thought: indeed if we do that we would only need to populate syscalls_morello_skip, which would then be used during both compat and purecap tests.
I guess I have an idea of how this could be done, although I am having trouble with it currently. I will continue to investigate further.
I agree, that's how I see things working out currently. The musl skip list would currently be used for both, indeed.
Thanks for replying, Best regards Téo
On Wed, Mar 22, 2023 at 11:49:31AM +0000, Teo Couprie Diaz wrote:
But that would meant the theory still matches reality. I was just wondering if there is any reason why test would work on purecap but not in plain aarch64 (we already know some cases for the reverse). But it seems we are good for now so it sounds sane to stick to that assumption.
The "*_64" is for syscalls that got extended to facilitate 64-bit values on 32-bit archs (like large file sizes), so there is no point in building/running those.
I'd suggest no to spend too much time on it. It's perfectly fine if we keep those listed explicitly in the skip files as they are.
Agreed then.
--- BR B.
On 22/03/2023 15:04, Beata Michalska wrote:
I couldn't think of any, what I mentioned would still match reality indeed. I added a mention of that assumption in the README in v4.
Oh, that makes a lot of sense then, thanks for the explanation !
I spent a bit of time trying to make something work. I have an imperfect solution that I will share separately from the v4, if anyone want to have a look at it someday.
I will share the v4 with those changes and those you mentioned in the previous thread.
Thanks for the discussion, Téo
linux-morello-ltp@op-lists.linaro.org
-
Beata Michalska -
Kevin Brodsky -
Teo Couprie Diaz