Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com --- v2: - Removed mentions of COMPAT - Reworded comment around Alpine - Added bpf_prog02 to the musl skips after further testing README.rst | 22 +++++++ runtest/syscalls_aarch64_pcuabi.skip | 96 ++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 17 +++++ 3 files changed, 135 insertions(+) create mode 100644 runtest/syscalls_aarch64_pcuabi.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..0c1646f46 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,28 @@ Running tests:
runltp -f morello_transitional,morello_transitional_extended
+Skipping tests: +--------------- + +In certain cases, you might want to skip tests that are known to be failing or are yet to be supported. + +This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``. + +We currently provide the following skip files under ``runtest/``: + - ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system. + - ``syscalls_aarch64_pcuabi.skip``: Skips all tests not considered while testing plain Aarch64 + with a PCuABI kernel. + +``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution. + +For example, in plain Aarch64 on a Musl system with a PCuABI kernel to test all syscalls but skip all +failures unrelated to the PCuABI kernel, one could run the following command: + +.. code-block:: + + ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip )
Developers playground --------------------- diff --git a/runtest/syscalls_aarch64_pcuabi.skip b/runtest/syscalls_aarch64_pcuabi.skip new file mode 100644 index 000000000..be9712906 --- /dev/null +++ b/runtest/syscalls_aarch64_pcuabi.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in plain Aarch64. + +# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01 + +# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64 diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..016d2728c --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,17 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01
On 28/02/2023 16:55, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 22 +++++++ runtest/syscalls_aarch64_pcuabi.skip | 96 ++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 17 +++++ 3 files changed, 135 insertions(+) create mode 100644 runtest/syscalls_aarch64_pcuabi.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..0c1646f46 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,28 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system.
- ``syscalls_aarch64_pcuabi.skip``: Skips all tests not considered while testing plain Aarch64
- with a PCuABI kernel.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+For example, in plain Aarch64 on a Musl system with a PCuABI kernel to test all syscalls but skip all +failures unrelated to the PCuABI kernel, one could run the following command:
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip )
This doesn't seem to work for me:
awk: fatal: cannot open file `/dev/fd/63' for reading: No such file or directory
Directly running awk with the process substitution works fine, so I'm not sure exactly what's happening here.
Kevin
Developers playground
diff --git a/runtest/syscalls_aarch64_pcuabi.skip b/runtest/syscalls_aarch64_pcuabi.skip new file mode 100644 index 000000000..be9712906 --- /dev/null +++ b/runtest/syscalls_aarch64_pcuabi.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in plain Aarch64.
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64 diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..016d2728c --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,17 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01
On 01/03/2023 12:02, Kevin Brodsky wrote:
On 28/02/2023 16:55, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 22 +++++++ runtest/syscalls_aarch64_pcuabi.skip | 96 ++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 17 +++++ 3 files changed, 135 insertions(+) create mode 100644 runtest/syscalls_aarch64_pcuabi.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..0c1646f46 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,28 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system.
- ``syscalls_aarch64_pcuabi.skip``: Skips all tests not considered while testing plain Aarch64
- with a PCuABI kernel.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+For example, in plain Aarch64 on a Musl system with a PCuABI kernel to test all syscalls but skip all +failures unrelated to the PCuABI kernel, one could run the following command:
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip )
This doesn't seem to work for me:
awk: fatal: cannot open file `/dev/fd/63' for reading: No such file or directory
Directly running awk with the process substitution works fine, so I'm not sure exactly what's happening here.
That's a bit weird indeed, especially if it did generate the name pipe and works outside the script...
I haven't been able to reproduce it, either with bash, zsh, ksh or fish (using the slightly different syntax for process substitution there), so if you manage to reproduce it outside of your own environment I would be interested !
Thanks for bringing it up, Best regards Téo
Kevin
Developers playground
diff --git a/runtest/syscalls_aarch64_pcuabi.skip b/runtest/syscalls_aarch64_pcuabi.skip new file mode 100644 index 000000000..be9712906 --- /dev/null +++ b/runtest/syscalls_aarch64_pcuabi.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in plain Aarch64.
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64 diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..016d2728c --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,17 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01
On 06/03/2023 12:58, Teo Couprie Diaz wrote:
+ ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip )
This doesn't seem to work for me:
awk: fatal: cannot open file `/dev/fd/63' for reading: No such file or directory
Directly running awk with the process substitution works fine, so I'm not sure exactly what's happening here.
That's a bit weird indeed, especially if it did generate the name pipe and works outside the script...
I haven't been able to reproduce it, either with bash, zsh, ksh or fish (using the slightly different syntax for process substitution there), so if you manage to reproduce it outside of your own environment I would be interested !
Alright so actually I know why this didn't work for me: I'm running runltp with sudo, and process substitution is one of those things that doesn't play well with sudo. It's a general limitation, nothing specific to runltp, so feel free to ignore and sorry for the noise.
Kevin
On 15/03/2023 08:52, Kevin Brodsky wrote:
On 06/03/2023 12:58, Teo Couprie Diaz wrote:
+ ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip )
This doesn't seem to work for me:
awk: fatal: cannot open file `/dev/fd/63' for reading: No such file or directory
Directly running awk with the process substitution works fine, so I'm not sure exactly what's happening here.
That's a bit weird indeed, especially if it did generate the name pipe and works outside the script...
I haven't been able to reproduce it, either with bash, zsh, ksh or fish (using the slightly different syntax for process substitution there), so if you manage to reproduce it outside of your own environment I would be interested !
Alright so actually I know why this didn't work for me: I'm running runltp with sudo, and process substitution is one of those things that doesn't play well with sudo. It's a general limitation, nothing specific to runltp, so feel free to ignore and sorry for the noise.
Kevin
Oh right, that would make sense indeed ! No worries, on the contrary it's good to know possible limitations and to be sure that I didn't miss something will testing that.
Thanks for testing, Téo
On 28/02/2023 16:55, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 22 +++++++ runtest/syscalls_aarch64_pcuabi.skip | 96 ++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 17 +++++ 3 files changed, 135 insertions(+) create mode 100644 runtest/syscalls_aarch64_pcuabi.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..0c1646f46 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,28 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system.
- ``syscalls_aarch64_pcuabi.skip``: Skips all tests not considered while testing plain Aarch64
- with a PCuABI kernel.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+For example, in plain Aarch64 on a Musl system with a PCuABI kernel to test all syscalls but skip all +failures unrelated to the PCuABI kernel, one could run the following command:
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip )
Developers playground
diff --git a/runtest/syscalls_aarch64_pcuabi.skip b/runtest/syscalls_aarch64_pcuabi.skip new file mode 100644 index 000000000..be9712906 --- /dev/null +++ b/runtest/syscalls_aarch64_pcuabi.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in plain Aarch64.
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64
It looks like this list misses lstat01_64. Also, what about *_16? We do not build these either.
In fact there are a few more tests in syscalls that we do not build yet because they are under testcases/kernel/security: prot_hsymlinks, dirtyc0w, dirtypipe. I guess we would need to either build them or exclude them to run the whole syscalls suite.
Kevin
diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..016d2728c --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,17 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01
On Wed, Mar 01, 2023 at 01:48:40PM +0100, Kevin Brodsky wrote:
On 28/02/2023 16:55, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 22 +++++++ runtest/syscalls_aarch64_pcuabi.skip | 96 ++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 17 +++++ 3 files changed, 135 insertions(+) create mode 100644 runtest/syscalls_aarch64_pcuabi.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..0c1646f46 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,28 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system.
- ``syscalls_aarch64_pcuabi.skip``: Skips all tests not considered while testing plain Aarch64
- with a PCuABI kernel.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+For example, in plain Aarch64 on a Musl system with a PCuABI kernel to test all syscalls but skip all +failures unrelated to the PCuABI kernel, one could run the following command:
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip )
Developers playground
diff --git a/runtest/syscalls_aarch64_pcuabi.skip b/runtest/syscalls_aarch64_pcuabi.skip new file mode 100644 index 000000000..be9712906 --- /dev/null +++ b/runtest/syscalls_aarch64_pcuabi.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in plain Aarch64.
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64
It looks like this list misses lstat01_64. Also, what about *_16? We do not build these either.
In fact there are a few more tests in syscalls that we do not build yet because they are under testcases/kernel/security: prot_hsymlinks, dirtyc0w, dirtypipe. I guess we would need to either build them or exclude them to run the whole syscalls suite.
Kevin
On top of that - we should explicitly state that the skip list assumes we are running with the new release of Musl, so required Musl release is > 1.5 , otherwise the skip list is not covering all currently failing tests that await Musl update (and potentially kernel release ?) So adding smth like we have in morello-specific runtest files: #MUSL - depends on Musl release > morello-release-1.5.0 --- BR B.
diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..016d2728c --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,17 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
On 01/03/2023 16:32, Beata Michalska wrote:
On Wed, Mar 01, 2023 at 01:48:40PM +0100, Kevin Brodsky wrote:
On 28/02/2023 16:55, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 22 +++++++ runtest/syscalls_aarch64_pcuabi.skip | 96 ++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 17 +++++ 3 files changed, 135 insertions(+) create mode 100644 runtest/syscalls_aarch64_pcuabi.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..0c1646f46 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,28 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system.
- ``syscalls_aarch64_pcuabi.skip``: Skips all tests not considered while testing plain Aarch64
- with a PCuABI kernel.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+For example, in plain Aarch64 on a Musl system with a PCuABI kernel to test all syscalls but skip all +failures unrelated to the PCuABI kernel, one could run the following command:
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip )
Developers playground
diff --git a/runtest/syscalls_aarch64_pcuabi.skip b/runtest/syscalls_aarch64_pcuabi.skip new file mode 100644 index 000000000..be9712906 --- /dev/null +++ b/runtest/syscalls_aarch64_pcuabi.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in plain Aarch64.
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64
It looks like this list misses lstat01_64. Also, what about *_16? We do not build these either.
In fact there are a few more tests in syscalls that we do not build yet because they are under testcases/kernel/security: prot_hsymlinks, dirtyc0w, dirtypipe. I guess we would need to either build them or exclude them to run the whole syscalls suite.
Kevin
On top of that - we should explicitly state that the skip list assumes we are running with the new release of Musl, so required Musl release is > 1.5 , otherwise the skip list is not covering all currently failing tests that await Musl update (and potentially kernel release ?) So adding smth like we have in morello-specific runtest files: #MUSL - depends on Musl release > morello-release-1.5.0
Agreed, added some comments in the skip files as well as some info in the README.
Thanks for bringing this up ! Best regards, Téo
BR B.
diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..016d2728c --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,17 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
On 01/03/2023 12:48, Kevin Brodsky wrote:
On 28/02/2023 16:55, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 22 +++++++ runtest/syscalls_aarch64_pcuabi.skip | 96 ++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 17 +++++ 3 files changed, 135 insertions(+) create mode 100644 runtest/syscalls_aarch64_pcuabi.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..0c1646f46 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,28 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system.
- ``syscalls_aarch64_pcuabi.skip``: Skips all tests not considered while testing plain Aarch64
- with a PCuABI kernel.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+For example, in plain Aarch64 on a Musl system with a PCuABI kernel to test all syscalls but skip all +failures unrelated to the PCuABI kernel, one could run the following command:
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_{aarch64_pcuabi,musl}.skip )
Developers playground
diff --git a/runtest/syscalls_aarch64_pcuabi.skip b/runtest/syscalls_aarch64_pcuabi.skip new file mode 100644 index 000000000..be9712906 --- /dev/null +++ b/runtest/syscalls_aarch64_pcuabi.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in plain Aarch64.
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64
It looks like this list misses lstat01_64. Also, what about *_16? We do not build these either.
So, I found out that for some reason I had some of the *_64 tests built for compat and in my test directory, so I didn't include them in the skip list. But they really aren't built, so they should stay in. *_16 will be added in v3.
In fact there are a few more tests in syscalls that we do not build yet because they are under testcases/kernel/security: prot_hsymlinks, dirtyc0w, dirtypipe. I guess we would need to either build them or exclude them to run the whole syscalls suite.
As we don't build them I elected to include them to the skip list.
Investigating with Beata I now understand how I missed a lot of the tests to include in the skip list: I haven't really paid attention to the command line output of runltp, only the output file listing the failed tests. As tests that are not available are not counted as failed, they don't show up in the list of failed tests, which is how I missed them.
I re-ran some tests and used the command line output to check. With the skip lists in v3 they _should_ be all accounted for, and running `syscalls` with the two skip lists results in "all tests PASS".
Kevin
Thanks for testing, Best regards Téo
diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..016d2728c --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,17 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01
linux-morello-ltp@op-lists.linaro.org
-
Beata Michalska -
Kevin Brodsky -
Teo Couprie Diaz