Add a set of skip files for running the syscalls set of tests in compat or purecap: - syscalls_morello_musl_skip skips all Musl related failures, - syscalls_morello_skip is for tests skipped and not built for Morello, - syscalls_morello_purecap_skip is for tests skipped in purecap.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com --- v6: - Clarified wording on build setup and version compatibility - Added two missing tests to the relevant skip lists v5: - Clarified the wording around syscalls_morello_skip thanks to Kevin's comments. - Added command line example for purecap skip files. v4: - Some README rephrasing to address Beata's comments - Changed kernel version dependency comment in the musl skip file - Renamed all skip files to accomodate for a purecap one - Added purecap skip file - Updated README and commit message for purecap skip file v3: - Added *_16 tests to the skip list - Added tests from kernel/security as we don't build them yet - Mentioned Morello 1.6 release v2: - Removed mentions of COMPAT - Reworded comment around Alpine - Added bpf_prog02 to the musl skips after further testing
README.rst | 30 +++++ runtest/syscalls_morello_musl_skip | 18 +++ runtest/syscalls_morello_purecap_skip | 12 ++ runtest/syscalls_morello_skip | 173 ++++++++++++++++++++++++++ 4 files changed, 233 insertions(+) create mode 100644 runtest/syscalls_morello_musl_skip create mode 100644 runtest/syscalls_morello_purecap_skip create mode 100644 runtest/syscalls_morello_skip
diff --git a/README.rst b/README.rst index 8b6e3a7769d0..e8293e24401d 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,36 @@ Running tests:
runltp -f morello_transitional,morello_transitional_extended
+Skipping tests: +--------------- + +In certain cases, you might want to skip tests that are known to be failing or are yet to be supported. + +This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``. + +We currently provide the following skip files under ``runtest/``: + - ``syscalls_morello_musl_skip``: Skips all tests failing on a regular AArch64 Musl based system. + - ``syscalls_morello_skip``: Skips all tests not considered due to the Morello build setup, + or the run environment. + - ``syscalls_morello_purecap_skip``: Skips all tests not considered while testing with purecap binaries. + +``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution. + +.. code-block:: + + ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl}_skip ) + +To use all relevant skip files relevant for plain AArch64. + +.. code-block:: + + ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl,_purecap}_skip ) + +To use all relevant skip files relevant for pure-capability. + +These skip files are comprehensive only for morello release 1.6 and above.
Developers playground --------------------- diff --git a/runtest/syscalls_morello_musl_skip b/runtest/syscalls_morello_musl_skip new file mode 100644 index 000000000000..93fcb7e89886 --- /dev/null +++ b/runtest/syscalls_morello_musl_skip @@ -0,0 +1,18 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +# MUSL - Depends on Musl release >= morello-release-1.6.0 +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01 diff --git a/runtest/syscalls_morello_purecap_skip b/runtest/syscalls_morello_purecap_skip new file mode 100644 index 000000000000..4f60d9aff005 --- /dev/null +++ b/runtest/syscalls_morello_purecap_skip @@ -0,0 +1,12 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in purecap. + +# io_uring is not currently supported in purecap, +# this test is expected to fail. +io_uring01 + +# These tests share capabilities between proccesses through shared memory. +# This is explicitly prevented by the PCuABI, so there's no point running them. +process_vm_readv02 +process_vm_readv03 +process_vm_writev02 diff --git a/runtest/syscalls_morello_skip b/runtest/syscalls_morello_skip new file mode 100644 index 000000000000..bfa60570cb2f --- /dev/null +++ b/runtest/syscalls_morello_skip @@ -0,0 +1,173 @@ +# This skip file contains all the tests skipped from the syscalls command file +# because of either the build steup for Morello or the system the tests run on. +# KERN - *: depends on Morello Linux kernel release >= morello-release-1.6.0 + +# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01 + +# We do not build tests under kernel/security yet, skip them for now. +prot_hsymlinks +dirtyc0w +dirtyc0w_shmem +dirtypipe + +# We do not build *_16 tests, skip them. +chown01_16 +chown02_16 +chown03_16 +chown04_16 +chown05_16 +fchown01_16 +fchown02_16 +fchown03_16 +fchown04_16 +fchown05_16 +getegid01_16 +getegid02_16 +geteuid01_16 +geteuid02_16 +getgid01_16 +getgid03_16 +getgroups01_16 +getgroups03_16 +getresgid01_16 +getresgid02_16 +getresgid03_16 +getresuid01_16 +getresuid02_16 +getresuid03_16 +getuid01_16 +getuid03_16 +lchown01_16 +lchown02_16 +lchown03_16 +setfsgid01_16 +setfsgid02_16 +setfsgid03_16 +setfsuid01_16 +setfsuid02_16 +setfsuid03_16 +setfsuid04_16 +setgid01_16 +setgid02_16 +setgid03_16 +setgroups01_16 +setgroups02_16 +setgroups03_16 +setgroups04_16 +setregid01_16 +setregid02_16 +setregid03_16 +setregid04_16 +setresgid01_16 +setresgid02_16 +setresgid03_16 +setresgid04_16 +setresuid01_16 +setresuid02_16 +setresuid03_16 +setresuid04_16 +setresuid05_16 +setreuid01_16 +setreuid02_16 +setreuid03_16 +setreuid04_16 +setreuid05_16 +setreuid06_16 +setreuid07_16 +setuid01_16 +setuid03_16 +setuid04_16 + +# We do not build *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +stat04_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64
On 30/03/2023 18:16, Teo Couprie Diaz wrote:
Add a set of skip files for running the syscalls set of tests in compat or purecap:
- syscalls_morello_musl_skip skips all Musl related failures,
- syscalls_morello_skip is for tests skipped and not built for Morello,
- syscalls_morello_purecap_skip is for tests skipped in purecap.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v6:
- Clarified wording on build setup and version compatibility
- Added two missing tests to the relevant skip lists
v5:
- Clarified the wording around syscalls_morello_skip thanks to Kevin's comments.
- Added command line example for purecap skip files.
v4:
- Some README rephrasing to address Beata's comments
- Changed kernel version dependency comment in the musl skip file
- Renamed all skip files to accomodate for a purecap one
- Added purecap skip file
- Updated README and commit message for purecap skip file
v3:
- Added *_16 tests to the skip list
- Added tests from kernel/security as we don't build them yet
- Mentioned Morello 1.6 release
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 30 +++++ runtest/syscalls_morello_musl_skip | 18 +++ runtest/syscalls_morello_purecap_skip | 12 ++ runtest/syscalls_morello_skip | 173 ++++++++++++++++++++++++++ 4 files changed, 233 insertions(+) create mode 100644 runtest/syscalls_morello_musl_skip create mode 100644 runtest/syscalls_morello_purecap_skip create mode 100644 runtest/syscalls_morello_skip
diff --git a/README.rst b/README.rst index 8b6e3a7769d0..e8293e24401d 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,36 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_morello_musl_skip``: Skips all tests failing on a regular AArch64 Musl based system.
- ``syscalls_morello_skip``: Skips all tests not considered due to the Morello build setup,
- or the run environment.
- ``syscalls_morello_purecap_skip``: Skips all tests not considered while testing with purecap binaries.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl}_skip )
+To use all relevant skip files relevant for plain AArch64.
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl,_purecap}_skip )
+To use all relevant skip files relevant for pure-capability.
+These skip files are comprehensive only for morello release 1.6 and above. Developers playground
diff --git a/runtest/syscalls_morello_musl_skip b/runtest/syscalls_morello_musl_skip new file mode 100644 index 000000000000..93fcb7e89886 --- /dev/null +++ b/runtest/syscalls_morello_musl_skip @@ -0,0 +1,18 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +# MUSL - Depends on Musl release >= morello-release-1.6.0 +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01 diff --git a/runtest/syscalls_morello_purecap_skip b/runtest/syscalls_morello_purecap_skip new file mode 100644 index 000000000000..4f60d9aff005 --- /dev/null +++ b/runtest/syscalls_morello_purecap_skip @@ -0,0 +1,12 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in purecap.
+# io_uring is not currently supported in purecap, +# this test is expected to fail. +io_uring01
+# These tests share capabilities between proccesses through shared memory. +# This is explicitly prevented by the PCuABI, so there's no point running them. +process_vm_readv02 +process_vm_readv03 +process_vm_writev02 diff --git a/runtest/syscalls_morello_skip b/runtest/syscalls_morello_skip new file mode 100644 index 000000000000..bfa60570cb2f --- /dev/null +++ b/runtest/syscalls_morello_skip @@ -0,0 +1,173 @@ +# This skip file contains all the tests skipped from the syscalls command file +# because of either the build steup for Morello or the system the tests run on.
s/steup/setup/
LGTM otherwise.
Kevin
+# KERN - *: depends on Morello Linux kernel release >= morello-release-1.6.0
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build tests under kernel/security yet, skip them for now. +prot_hsymlinks +dirtyc0w +dirtyc0w_shmem +dirtypipe
+# We do not build *_16 tests, skip them. +chown01_16 +chown02_16 +chown03_16 +chown04_16 +chown05_16 +fchown01_16 +fchown02_16 +fchown03_16 +fchown04_16 +fchown05_16 +getegid01_16 +getegid02_16 +geteuid01_16 +geteuid02_16 +getgid01_16 +getgid03_16 +getgroups01_16 +getgroups03_16 +getresgid01_16 +getresgid02_16 +getresgid03_16 +getresuid01_16 +getresuid02_16 +getresuid03_16 +getuid01_16 +getuid03_16 +lchown01_16 +lchown02_16 +lchown03_16 +setfsgid01_16 +setfsgid02_16 +setfsgid03_16 +setfsuid01_16 +setfsuid02_16 +setfsuid03_16 +setfsuid04_16 +setgid01_16 +setgid02_16 +setgid03_16 +setgroups01_16 +setgroups02_16 +setgroups03_16 +setgroups04_16 +setregid01_16 +setregid02_16 +setregid03_16 +setregid04_16 +setresgid01_16 +setresgid02_16 +setresgid03_16 +setresgid04_16 +setresuid01_16 +setresuid02_16 +setresuid03_16 +setresuid04_16 +setresuid05_16 +setreuid01_16 +setreuid02_16 +setreuid03_16 +setreuid04_16 +setreuid05_16 +setreuid06_16 +setreuid07_16 +setuid01_16 +setuid03_16 +setuid04_16
+# We do not build *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +stat04_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64
On Fri, Mar 31, 2023 at 05:54:45PM +0200, Kevin Brodsky wrote:
On 30/03/2023 18:16, Teo Couprie Diaz wrote:
Add a set of skip files for running the syscalls set of tests in compat or purecap:
- syscalls_morello_musl_skip skips all Musl related failures,
- syscalls_morello_skip is for tests skipped and not built for Morello,
- syscalls_morello_purecap_skip is for tests skipped in purecap.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v6:
- Clarified wording on build setup and version compatibility
- Added two missing tests to the relevant skip lists
v5:
- Clarified the wording around syscalls_morello_skip thanks to Kevin's comments.
- Added command line example for purecap skip files.
v4:
- Some README rephrasing to address Beata's comments
- Changed kernel version dependency comment in the musl skip file
- Renamed all skip files to accomodate for a purecap one
- Added purecap skip file
- Updated README and commit message for purecap skip file
v3:
- Added *_16 tests to the skip list
- Added tests from kernel/security as we don't build them yet
- Mentioned Morello 1.6 release
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 30 +++++ runtest/syscalls_morello_musl_skip | 18 +++ runtest/syscalls_morello_purecap_skip | 12 ++ runtest/syscalls_morello_skip | 173 ++++++++++++++++++++++++++ 4 files changed, 233 insertions(+) create mode 100644 runtest/syscalls_morello_musl_skip create mode 100644 runtest/syscalls_morello_purecap_skip create mode 100644 runtest/syscalls_morello_skip
diff --git a/README.rst b/README.rst index 8b6e3a7769d0..e8293e24401d 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,36 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_morello_musl_skip``: Skips all tests failing on a regular AArch64 Musl based system.
- ``syscalls_morello_skip``: Skips all tests not considered due to the Morello build setup,
- or the run environment.
- ``syscalls_morello_purecap_skip``: Skips all tests not considered while testing with purecap binaries.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl}_skip )
+To use all relevant skip files relevant for plain AArch64.
Double 'relevant', should be enough to keep one :)
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl,_purecap}_skip )
+To use all relevant skip files relevant for pure-capability.
Same as above. Also, it might be better to have the lines order inverted: first explanation that the code block.
+These skip files are comprehensive only for morello release 1.6 and above. Developers playground
diff --git a/runtest/syscalls_morello_musl_skip b/runtest/syscalls_morello_musl_skip new file mode 100644 index 000000000000..93fcb7e89886 --- /dev/null +++ b/runtest/syscalls_morello_musl_skip @@ -0,0 +1,18 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +# MUSL - Depends on Musl release >= morello-release-1.6.0 +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01 diff --git a/runtest/syscalls_morello_purecap_skip b/runtest/syscalls_morello_purecap_skip new file mode 100644 index 000000000000..4f60d9aff005 --- /dev/null +++ b/runtest/syscalls_morello_purecap_skip @@ -0,0 +1,12 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in purecap.
+# io_uring is not currently supported in purecap, +# this test is expected to fail. +io_uring01
+# These tests share capabilities between proccesses through shared memory. +# This is explicitly prevented by the PCuABI, so there's no point running them. +process_vm_readv02 +process_vm_readv03 +process_vm_writev02 diff --git a/runtest/syscalls_morello_skip b/runtest/syscalls_morello_skip new file mode 100644 index 000000000000..bfa60570cb2f --- /dev/null +++ b/runtest/syscalls_morello_skip @@ -0,0 +1,173 @@ +# This skip file contains all the tests skipped from the syscalls command file +# because of either the build steup for Morello or the system the tests run on.
s/steup/setup/
LGTM otherwise.
I can make this change with the ones mentioned above while applying the patch, if you are ok with it.
--- BR B.
Kevin
+# KERN - *: depends on Morello Linux kernel release >= morello-release-1.6.0
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build tests under kernel/security yet, skip them for now. +prot_hsymlinks +dirtyc0w +dirtyc0w_shmem +dirtypipe
+# We do not build *_16 tests, skip them. +chown01_16 +chown02_16 +chown03_16 +chown04_16 +chown05_16 +fchown01_16 +fchown02_16 +fchown03_16 +fchown04_16 +fchown05_16 +getegid01_16 +getegid02_16 +geteuid01_16 +geteuid02_16 +getgid01_16 +getgid03_16 +getgroups01_16 +getgroups03_16 +getresgid01_16 +getresgid02_16 +getresgid03_16 +getresuid01_16 +getresuid02_16 +getresuid03_16 +getuid01_16 +getuid03_16 +lchown01_16 +lchown02_16 +lchown03_16 +setfsgid01_16 +setfsgid02_16 +setfsgid03_16 +setfsuid01_16 +setfsuid02_16 +setfsuid03_16 +setfsuid04_16 +setgid01_16 +setgid02_16 +setgid03_16 +setgroups01_16 +setgroups02_16 +setgroups03_16 +setgroups04_16 +setregid01_16 +setregid02_16 +setregid03_16 +setregid04_16 +setresgid01_16 +setresgid02_16 +setresgid03_16 +setresgid04_16 +setresuid01_16 +setresuid02_16 +setresuid03_16 +setresuid04_16 +setresuid05_16 +setreuid01_16 +setreuid02_16 +setreuid03_16 +setreuid04_16 +setreuid05_16 +setreuid06_16 +setreuid07_16 +setuid01_16 +setuid03_16 +setuid04_16
+# We do not build *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +stat04_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
On 04/04/2023 10:37, Beata Michalska wrote:
On Fri, Mar 31, 2023 at 05:54:45PM +0200, Kevin Brodsky wrote:
On 30/03/2023 18:16, Teo Couprie Diaz wrote:
Add a set of skip files for running the syscalls set of tests in compat or purecap:
- syscalls_morello_musl_skip skips all Musl related failures,
- syscalls_morello_skip is for tests skipped and not built for Morello,
- syscalls_morello_purecap_skip is for tests skipped in purecap.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v6:
- Clarified wording on build setup and version compatibility
- Added two missing tests to the relevant skip lists
v5:
- Clarified the wording around syscalls_morello_skip thanks to Kevin's comments.
- Added command line example for purecap skip files.
v4:
- Some README rephrasing to address Beata's comments
- Changed kernel version dependency comment in the musl skip file
- Renamed all skip files to accomodate for a purecap one
- Added purecap skip file
- Updated README and commit message for purecap skip file
v3:
- Added *_16 tests to the skip list
- Added tests from kernel/security as we don't build them yet
- Mentioned Morello 1.6 release
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 30 +++++ runtest/syscalls_morello_musl_skip | 18 +++ runtest/syscalls_morello_purecap_skip | 12 ++ runtest/syscalls_morello_skip | 173 ++++++++++++++++++++++++++ 4 files changed, 233 insertions(+) create mode 100644 runtest/syscalls_morello_musl_skip create mode 100644 runtest/syscalls_morello_purecap_skip create mode 100644 runtest/syscalls_morello_skip
diff --git a/README.rst b/README.rst index 8b6e3a7769d0..e8293e24401d 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,36 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_morello_musl_skip``: Skips all tests failing on a regular AArch64 Musl based system.
- ``syscalls_morello_skip``: Skips all tests not considered due to the Morello build setup,
- or the run environment.
- ``syscalls_morello_purecap_skip``: Skips all tests not considered while testing with purecap binaries.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl}_skip )
+To use all relevant skip files relevant for plain AArch64.
Double 'relevant', should be enough to keep one :)
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl,_purecap}_skip )
+To use all relevant skip files relevant for pure-capability.
Same as above. Also, it might be better to have the lines order inverted: first explanation that the code block.
+These skip files are comprehensive only for morello release 1.6 and above. Developers playground
diff --git a/runtest/syscalls_morello_musl_skip b/runtest/syscalls_morello_musl_skip new file mode 100644 index 000000000000..93fcb7e89886 --- /dev/null +++ b/runtest/syscalls_morello_musl_skip @@ -0,0 +1,18 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +# MUSL - Depends on Musl release >= morello-release-1.6.0 +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01 diff --git a/runtest/syscalls_morello_purecap_skip b/runtest/syscalls_morello_purecap_skip new file mode 100644 index 000000000000..4f60d9aff005 --- /dev/null +++ b/runtest/syscalls_morello_purecap_skip @@ -0,0 +1,12 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in purecap.
+# io_uring is not currently supported in purecap, +# this test is expected to fail. +io_uring01
+# These tests share capabilities between proccesses through shared memory. +# This is explicitly prevented by the PCuABI, so there's no point running them. +process_vm_readv02 +process_vm_readv03 +process_vm_writev02 diff --git a/runtest/syscalls_morello_skip b/runtest/syscalls_morello_skip new file mode 100644 index 000000000000..bfa60570cb2f --- /dev/null +++ b/runtest/syscalls_morello_skip @@ -0,0 +1,173 @@ +# This skip file contains all the tests skipped from the syscalls command file +# because of either the build steup for Morello or the system the tests run on.
s/steup/setup/
LGTM otherwise.
I can make this change with the ones mentioned above while applying the patch, if you are ok with it.
I agree with all Kevin's comments, happy for you to make them when applying the patch.
Thanks both ! Best regards, Téo
BR B.
Kevin
+# KERN - *: depends on Morello Linux kernel release >= morello-release-1.6.0
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build tests under kernel/security yet, skip them for now. +prot_hsymlinks +dirtyc0w +dirtyc0w_shmem +dirtypipe
+# We do not build *_16 tests, skip them. +chown01_16 +chown02_16 +chown03_16 +chown04_16 +chown05_16 +fchown01_16 +fchown02_16 +fchown03_16 +fchown04_16 +fchown05_16 +getegid01_16 +getegid02_16 +geteuid01_16 +geteuid02_16 +getgid01_16 +getgid03_16 +getgroups01_16 +getgroups03_16 +getresgid01_16 +getresgid02_16 +getresgid03_16 +getresuid01_16 +getresuid02_16 +getresuid03_16 +getuid01_16 +getuid03_16 +lchown01_16 +lchown02_16 +lchown03_16 +setfsgid01_16 +setfsgid02_16 +setfsgid03_16 +setfsuid01_16 +setfsuid02_16 +setfsuid03_16 +setfsuid04_16 +setgid01_16 +setgid02_16 +setgid03_16 +setgroups01_16 +setgroups02_16 +setgroups03_16 +setgroups04_16 +setregid01_16 +setregid02_16 +setregid03_16 +setregid04_16 +setresgid01_16 +setresgid02_16 +setresgid03_16 +setresgid04_16 +setresuid01_16 +setresuid02_16 +setresuid03_16 +setresuid04_16 +setresuid05_16 +setreuid01_16 +setreuid02_16 +setreuid03_16 +setreuid04_16 +setreuid05_16 +setreuid06_16 +setreuid07_16 +setuid01_16 +setuid03_16 +setuid04_16
+# We do not build *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +stat04_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
Just landed on next.
--- BR B. On Tue, Apr 04, 2023 at 11:08:25AM +0100, Teo Couprie Diaz wrote:
On 04/04/2023 10:37, Beata Michalska wrote:
On Fri, Mar 31, 2023 at 05:54:45PM +0200, Kevin Brodsky wrote:
On 30/03/2023 18:16, Teo Couprie Diaz wrote:
Add a set of skip files for running the syscalls set of tests in compat or purecap:
- syscalls_morello_musl_skip skips all Musl related failures,
- syscalls_morello_skip is for tests skipped and not built for Morello,
- syscalls_morello_purecap_skip is for tests skipped in purecap.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v6:
- Clarified wording on build setup and version compatibility
- Added two missing tests to the relevant skip lists
v5:
- Clarified the wording around syscalls_morello_skip thanks to Kevin's comments.
- Added command line example for purecap skip files.
v4:
- Some README rephrasing to address Beata's comments
- Changed kernel version dependency comment in the musl skip file
- Renamed all skip files to accomodate for a purecap one
- Added purecap skip file
- Updated README and commit message for purecap skip file
v3:
- Added *_16 tests to the skip list
- Added tests from kernel/security as we don't build them yet
- Mentioned Morello 1.6 release
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 30 +++++ runtest/syscalls_morello_musl_skip | 18 +++ runtest/syscalls_morello_purecap_skip | 12 ++ runtest/syscalls_morello_skip | 173 ++++++++++++++++++++++++++ 4 files changed, 233 insertions(+) create mode 100644 runtest/syscalls_morello_musl_skip create mode 100644 runtest/syscalls_morello_purecap_skip create mode 100644 runtest/syscalls_morello_skip
diff --git a/README.rst b/README.rst index 8b6e3a7769d0..e8293e24401d 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,36 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_morello_musl_skip``: Skips all tests failing on a regular AArch64 Musl based system.
- ``syscalls_morello_skip``: Skips all tests not considered due to the Morello build setup,
- or the run environment.
- ``syscalls_morello_purecap_skip``: Skips all tests not considered while testing with purecap binaries.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl}_skip )
+To use all relevant skip files relevant for plain AArch64.
Double 'relevant', should be enough to keep one :)
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl,_purecap}_skip )
+To use all relevant skip files relevant for pure-capability.
Same as above. Also, it might be better to have the lines order inverted: first explanation that the code block.
+These skip files are comprehensive only for morello release 1.6 and above. Developers playground
diff --git a/runtest/syscalls_morello_musl_skip b/runtest/syscalls_morello_musl_skip new file mode 100644 index 000000000000..93fcb7e89886 --- /dev/null +++ b/runtest/syscalls_morello_musl_skip @@ -0,0 +1,18 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +# MUSL - Depends on Musl release >= morello-release-1.6.0 +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01 diff --git a/runtest/syscalls_morello_purecap_skip b/runtest/syscalls_morello_purecap_skip new file mode 100644 index 000000000000..4f60d9aff005 --- /dev/null +++ b/runtest/syscalls_morello_purecap_skip @@ -0,0 +1,12 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in purecap.
+# io_uring is not currently supported in purecap, +# this test is expected to fail. +io_uring01
+# These tests share capabilities between proccesses through shared memory. +# This is explicitly prevented by the PCuABI, so there's no point running them. +process_vm_readv02 +process_vm_readv03 +process_vm_writev02 diff --git a/runtest/syscalls_morello_skip b/runtest/syscalls_morello_skip new file mode 100644 index 000000000000..bfa60570cb2f --- /dev/null +++ b/runtest/syscalls_morello_skip @@ -0,0 +1,173 @@ +# This skip file contains all the tests skipped from the syscalls command file +# because of either the build steup for Morello or the system the tests run on.
s/steup/setup/
LGTM otherwise.
I can make this change with the ones mentioned above while applying the patch, if you are ok with it.
I agree with all Kevin's comments, happy for you to make them when applying the patch.
Thanks both ! Best regards, Téo
BR B.
Kevin
+# KERN - *: depends on Morello Linux kernel release >= morello-release-1.6.0
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build tests under kernel/security yet, skip them for now. +prot_hsymlinks +dirtyc0w +dirtyc0w_shmem +dirtypipe
+# We do not build *_16 tests, skip them. +chown01_16 +chown02_16 +chown03_16 +chown04_16 +chown05_16 +fchown01_16 +fchown02_16 +fchown03_16 +fchown04_16 +fchown05_16 +getegid01_16 +getegid02_16 +geteuid01_16 +geteuid02_16 +getgid01_16 +getgid03_16 +getgroups01_16 +getgroups03_16 +getresgid01_16 +getresgid02_16 +getresgid03_16 +getresuid01_16 +getresuid02_16 +getresuid03_16 +getuid01_16 +getuid03_16 +lchown01_16 +lchown02_16 +lchown03_16 +setfsgid01_16 +setfsgid02_16 +setfsgid03_16 +setfsuid01_16 +setfsuid02_16 +setfsuid03_16 +setfsuid04_16 +setgid01_16 +setgid02_16 +setgid03_16 +setgroups01_16 +setgroups02_16 +setgroups03_16 +setgroups04_16 +setregid01_16 +setregid02_16 +setregid03_16 +setregid04_16 +setresgid01_16 +setresgid02_16 +setresgid03_16 +setresgid04_16 +setresuid01_16 +setresuid02_16 +setresuid03_16 +setresuid04_16 +setresuid05_16 +setreuid01_16 +setreuid02_16 +setreuid03_16 +setreuid04_16 +setreuid05_16 +setreuid06_16 +setreuid07_16 +setuid01_16 +setuid03_16 +setuid04_16
+# We do not build *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +stat04_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
linux-morello-ltp@op-lists.linaro.org
-
Beata Michalska -
Kevin Brodsky -
Teo Couprie Diaz