Hi Beata,
On 09/03/2023 13:33, Beata Michalska wrote:
Hi Teo,
On Mon, Mar 06, 2023 at 11:59:10AM +0000, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v3:
- Added *_16 tests to the skip list
- Added tests from kernel/security as we don't build them yet
- Mentioned Morello 1.6 release
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 25 ++++ runtest/syscalls_aarch64_pcuabi.skip | 172 +++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 18 +++ 3 files changed, 215 insertions(+) create mode 100644 runtest/syscalls_aarch64_pcuabi.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a7769d0..ef1282f2b93f 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,31 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system.
- ``syscalls_aarch64_pcuabi.skip``: Skips all tests not considered while testing plain Aarch64
- with a PCuABI kernel.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
It's minor, but it might be better (for readability) to skip the sentence below altogether: the purpose of skip files has already been described above and I guess we only want to present a way on how to use both skip files for a single run (?),
Mh, yeah I get what you mean. I feel that's fair, and it would follow directly after mentioning process substitution.
+For example, in plain Aarch64 on a Musl system with a PCuABI kernel to test all syscalls but skip all +failures unrelated to the PCuABI kernel, one could run the following command:
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip )
I might have missed smth but was there a conclusion on this approach potentially failing on some distros ?
I did some more testing and could not reproduce Kevin's issue, which in itself was weird (it worked, but not in the script ?). Since he is on leave he hasn't had a chance to provide more details. I'm fine with waiting for a definitive answer, but I feel this could be easily fixed if needed later.
+Using those two skip lists should result in a PASS on Morello release 1.6 and after, as multiple patches +to Musl and the Linux kernel are needed on top of the 1.5 release.
How about: In order for the skip files to successfully filter-out expected/known failures, morello-release-1.6.0 release, or newer is required.
I like your phrasing better, I'm happy with replacing it.
(?)
Developers playground
diff --git a/runtest/syscalls_aarch64_pcuabi.skip b/runtest/syscalls_aarch64_pcuabi.skip new file mode 100644 index 000000000000..2097341de32b --- /dev/null +++ b/runtest/syscalls_aarch64_pcuabi.skip @@ -0,0 +1,172 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in plain Aarch64. +# LINUX - Depends on kernel release > morello-release-1.5.0
Minor, but we did use the patter below earlier, so it might be good to stay consistent (?): #KERN - *: depends on Morello Linux kernel release > morello-release-1.5.0
Ah right, it would be better to stay consistent for sure.
I can apply the changes (if any) while merging the patches.
Happy with you applying everything you mentioned if you decide to merge !
Thanks for the review, Best regards Téo
BR B.
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build tests under kernel/security yet, skip them for now. +prot_hsymlinks +dirtyc0w +dirtypipe
+# We do not build *_16 tests, skip them. +chown01_16 +chown02_16 +chown03_16 +chown04_16 +chown05_16 +fchown01_16 +fchown02_16 +fchown03_16 +fchown04_16 +fchown05_16 +getegid01_16 +getegid02_16 +geteuid01_16 +geteuid02_16 +getgid01_16 +getgid03_16 +getgroups01_16 +getgroups03_16 +getresgid01_16 +getresgid02_16 +getresgid03_16 +getresuid01_16 +getresuid02_16 +getresuid03_16 +getuid01_16 +getuid03_16 +lchown01_16 +lchown02_16 +lchown03_16 +setfsgid01_16 +setfsgid02_16 +setfsgid03_16 +setfsuid01_16 +setfsuid02_16 +setfsuid03_16 +setfsuid04_16 +setgid01_16 +setgid02_16 +setgid03_16 +setgroups01_16 +setgroups02_16 +setgroups03_16 +setgroups04_16 +setregid01_16 +setregid02_16 +setregid03_16 +setregid04_16 +setresgid01_16 +setresgid02_16 +setresgid03_16 +setresgid04_16 +setresuid01_16 +setresuid02_16 +setresuid03_16 +setresuid04_16 +setresuid05_16 +setreuid01_16 +setreuid02_16 +setreuid03_16 +setreuid04_16 +setreuid05_16 +setreuid06_16 +setreuid07_16 +setuid01_16 +setuid03_16 +setuid04_16
+# We do not build *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +stat04_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64 diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000000..aa3dff6c9449 --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,18 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +# MUSL - Depends on Musl release > morello-release-1.5.0 +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01
+sigwaitinfo01
2.25.1
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org