Add a set of skip files for running the syscalls set of tests in compat or purecap: - syscalls_morello_musl_skip skips all Musl related failures, - syscalls_morello_skip is for skipped and not built tests in compat, - syscalls_morello_purecap_skip is for tests skipped in purecap.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com --- v5: - Clarified the wording around syscalls_morello_skip thanks to Kevin's comments. - Added command line example for purecap skip files. v4: - Some README rephrasing to address Beata's comments - Changed kernel version dependency comment in the musl skip file - Renamed all skip files to accomodate for a purecap one - Added purecap skip file - Updated README and commit message for purecap skip file v3: - Added *_16 tests to the skip list - Added tests from kernel/security as we don't build them yet - Mentioned Morello 1.6 release v2: - Removed mentions of COMPAT - Reworded comment around Alpine - Added bpf_prog02 to the musl skips after further testing
README.rst | 30 +++++ runtest/syscalls_morello_musl_skip | 18 +++ runtest/syscalls_morello_purecap_skip | 8 ++ runtest/syscalls_morello_skip | 172 ++++++++++++++++++++++++++ 4 files changed, 228 insertions(+) create mode 100644 runtest/syscalls_morello_musl_skip create mode 100644 runtest/syscalls_morello_purecap_skip create mode 100644 runtest/syscalls_morello_skip
diff --git a/README.rst b/README.rst index 8b6e3a7769d0..73edda443cc4 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,36 @@ Running tests:
runltp -f morello_transitional,morello_transitional_extended
+Skipping tests: +--------------- + +In certain cases, you might want to skip tests that are known to be failing or are yet to be supported. + +This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``. + +We currently provide the following skip files under ``runtest/``: + - ``syscalls_morello_musl_skip``: Skips all tests failing on a regular AArch64 Musl based system. + - ``syscalls_morello_skip``: Skips all tests not considered due to the Morello build, or the run environment. + - ``syscalls_morello_purecap_skip``: Skips all tests not considered while testing with purecap binaries. + +``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution. + +.. code-block:: + + ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl}_skip ) + +To use all relevant skip files relevant for plain AArch64. + +.. code-block:: + + ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl,_purecap}_skip ) + +To use all relevant skip files relevant for plain pure-capability. + +In order for the skip files to successfully filter-out expected/known failures in AArch64, +morello-release-1.6.0 release, or newer is required.
Developers playground --------------------- diff --git a/runtest/syscalls_morello_musl_skip b/runtest/syscalls_morello_musl_skip new file mode 100644 index 000000000000..aa3dff6c9449 --- /dev/null +++ b/runtest/syscalls_morello_musl_skip @@ -0,0 +1,18 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +# MUSL - Depends on Musl release > morello-release-1.5.0 +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01 diff --git a/runtest/syscalls_morello_purecap_skip b/runtest/syscalls_morello_purecap_skip new file mode 100644 index 000000000000..e4d66ee1b27f --- /dev/null +++ b/runtest/syscalls_morello_purecap_skip @@ -0,0 +1,8 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in purecap. + +# These tests share capabilities between proccesses through shared memory. +# This is explicitly prevented by the PCuABI, so there's no point running them. +process_vm_readv02 +process_vm_readv03 +process_vm_writev02 diff --git a/runtest/syscalls_morello_skip b/runtest/syscalls_morello_skip new file mode 100644 index 000000000000..b9c0a2479ca9 --- /dev/null +++ b/runtest/syscalls_morello_skip @@ -0,0 +1,172 @@ +# This skip file contains all the tests skipped from the syscalls command file +# because of either the build for Morello or the system the tests are run with. +# KERN - *: depends on Morello Linux kernel release > morello-release-1.5.0 + +# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01 + +# We do not build tests under kernel/security yet, skip them for now. +prot_hsymlinks +dirtyc0w +dirtypipe + +# We do not build *_16 tests, skip them. +chown01_16 +chown02_16 +chown03_16 +chown04_16 +chown05_16 +fchown01_16 +fchown02_16 +fchown03_16 +fchown04_16 +fchown05_16 +getegid01_16 +getegid02_16 +geteuid01_16 +geteuid02_16 +getgid01_16 +getgid03_16 +getgroups01_16 +getgroups03_16 +getresgid01_16 +getresgid02_16 +getresgid03_16 +getresuid01_16 +getresuid02_16 +getresuid03_16 +getuid01_16 +getuid03_16 +lchown01_16 +lchown02_16 +lchown03_16 +setfsgid01_16 +setfsgid02_16 +setfsgid03_16 +setfsuid01_16 +setfsuid02_16 +setfsuid03_16 +setfsuid04_16 +setgid01_16 +setgid02_16 +setgid03_16 +setgroups01_16 +setgroups02_16 +setgroups03_16 +setgroups04_16 +setregid01_16 +setregid02_16 +setregid03_16 +setregid04_16 +setresgid01_16 +setresgid02_16 +setresgid03_16 +setresgid04_16 +setresuid01_16 +setresuid02_16 +setresuid03_16 +setresuid04_16 +setresuid05_16 +setreuid01_16 +setreuid02_16 +setreuid03_16 +setreuid04_16 +setreuid05_16 +setreuid06_16 +setreuid07_16 +setuid01_16 +setuid03_16 +setuid04_16 + +# We do not build *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +stat04_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64
On 27/03/2023 17:05, Teo Couprie Diaz wrote:
Add a set of skip files for running the syscalls set of tests in compat or purecap:
- syscalls_morello_musl_skip skips all Musl related failures,
- syscalls_morello_skip is for skipped and not built tests in compat,
Should be updated too I suppose.
- syscalls_morello_purecap_skip is for tests skipped in purecap.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v5:
- Clarified the wording around syscalls_morello_skip thanks to Kevin's comments.
- Added command line example for purecap skip files.
v4:
- Some README rephrasing to address Beata's comments
- Changed kernel version dependency comment in the musl skip file
- Renamed all skip files to accomodate for a purecap one
- Added purecap skip file
- Updated README and commit message for purecap skip file
v3:
- Added *_16 tests to the skip list
- Added tests from kernel/security as we don't build them yet
- Mentioned Morello 1.6 release
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 30 +++++ runtest/syscalls_morello_musl_skip | 18 +++ runtest/syscalls_morello_purecap_skip | 8 ++ runtest/syscalls_morello_skip | 172 ++++++++++++++++++++++++++ 4 files changed, 228 insertions(+) create mode 100644 runtest/syscalls_morello_musl_skip create mode 100644 runtest/syscalls_morello_purecap_skip create mode 100644 runtest/syscalls_morello_skip
diff --git a/README.rst b/README.rst index 8b6e3a7769d0..73edda443cc4 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,36 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_morello_musl_skip``: Skips all tests failing on a regular AArch64 Musl based system.
- ``syscalls_morello_skip``: Skips all tests not considered due to the Morello build, or the run environment.
"Morello build" on its own sounds like there are defects in the build output, when really the issue is simply that we are not building some tests at all. "Morello build setup" maybe?
- ``syscalls_morello_purecap_skip``: Skips all tests not considered while testing with purecap binaries.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl}_skip )
+To use all relevant skip files relevant for plain AArch64.
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl,_purecap}_skip )
+To use all relevant skip files relevant for plain pure-capability.
"plain pure-capability" is new terminology :) Just "pure-capability"?
+In order for the skip files to successfully filter-out expected/known failures in AArch64, +morello-release-1.6.0 release, or newer is required.
I think that one might be confusing because it's unclear which component that requirement applies to, especially since the files we're adding require release 1.5 for the kernel and Musl, not 1.6.
Developers playground
diff --git a/runtest/syscalls_morello_musl_skip b/runtest/syscalls_morello_musl_skip new file mode 100644 index 000000000000..aa3dff6c9449 --- /dev/null +++ b/runtest/syscalls_morello_musl_skip @@ -0,0 +1,18 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +# MUSL - Depends on Musl release > morello-release-1.5.0 +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01 diff --git a/runtest/syscalls_morello_purecap_skip b/runtest/syscalls_morello_purecap_skip new file mode 100644 index 000000000000..e4d66ee1b27f --- /dev/null +++ b/runtest/syscalls_morello_purecap_skip @@ -0,0 +1,8 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in purecap.
+# These tests share capabilities between proccesses through shared memory. +# This is explicitly prevented by the PCuABI, so there's no point running them. +process_vm_readv02 +process_vm_readv03 +process_vm_writev02 diff --git a/runtest/syscalls_morello_skip b/runtest/syscalls_morello_skip new file mode 100644 index 000000000000..b9c0a2479ca9 --- /dev/null +++ b/runtest/syscalls_morello_skip @@ -0,0 +1,172 @@ +# This skip file contains all the tests skipped from the syscalls command file +# because of either the build for Morello or the system the tests are run with.
Ditto here, would say "build setup".
Other than these superficial comments, the patch looks good and I confirmed locally that all the expected tests are passing.
Kevin
On 30/03/2023 13:30, Kevin Brodsky wrote:
On 27/03/2023 17:05, Teo Couprie Diaz wrote:
Add a set of skip files for running the syscalls set of tests in compat or purecap:
- syscalls_morello_musl_skip skips all Musl related failures,
- syscalls_morello_skip is for skipped and not built tests in compat,
Should be updated too I suppose.
Right, I missed it when I checked but it definitely needs a change.
- syscalls_morello_purecap_skip is for tests skipped in purecap.
Update the README to reflect those additions and explain how to use the skip files.
This is preparation for CI testing in Morello release 1.6 and later.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
v5:
- Clarified the wording around syscalls_morello_skip thanks to Kevin's comments.
- Added command line example for purecap skip files.
v4:
- Some README rephrasing to address Beata's comments
- Changed kernel version dependency comment in the musl skip file
- Renamed all skip files to accomodate for a purecap one
- Added purecap skip file
- Updated README and commit message for purecap skip file
v3:
- Added *_16 tests to the skip list
- Added tests from kernel/security as we don't build them yet
- Mentioned Morello 1.6 release
v2:
- Removed mentions of COMPAT
- Reworded comment around Alpine
- Added bpf_prog02 to the musl skips after further testing
README.rst | 30 +++++ runtest/syscalls_morello_musl_skip | 18 +++ runtest/syscalls_morello_purecap_skip | 8 ++ runtest/syscalls_morello_skip | 172 ++++++++++++++++++++++++++ 4 files changed, 228 insertions(+) create mode 100644 runtest/syscalls_morello_musl_skip create mode 100644 runtest/syscalls_morello_purecap_skip create mode 100644 runtest/syscalls_morello_skip
diff --git a/README.rst b/README.rst index 8b6e3a7769d0..73edda443cc4 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,36 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_morello_musl_skip``: Skips all tests failing on a regular AArch64 Musl based system.
- ``syscalls_morello_skip``: Skips all tests not considered due to the Morello build, or the run environment.
"Morello build" on its own sounds like there are defects in the build output, when really the issue is simply that we are not building some tests at all. "Morello build setup" maybe?
Yeah agreed, it makes it a bit less weird.
- ``syscalls_morello_purecap_skip``: Skips all tests not considered while testing with purecap binaries.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl}_skip )
+To use all relevant skip files relevant for plain AArch64.
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_morello{,_musl,_purecap}_skip )
+To use all relevant skip files relevant for plain pure-capability.
"plain pure-capability" is new terminology :) Just "pure-capability"?
That's copy-paste terminology – who needs AI to make stuff up ! My bad indeed, just "pure-capability" :)
+In order for the skip files to successfully filter-out expected/known failures in AArch64, +morello-release-1.6.0 release, or newer is required.
I think that one might be confusing because it's unclear which component that requirement applies to, especially since the files we're adding require release 1.5 for the kernel and Musl, not 1.6.
Mh I'm not sure I understand correctly. There are multiple tests that require either Musl or Linux 1.6 (or, at any rate, newer than 1.5) to run successfully. The files themselves don't really need anything, for sure, but for the test suite to pass it would.
But I agree that my phrasing is unclear, slightly voluntarily. Indeed it would be correct only in compat, as you confirmed that the io_uring changes wouldn't make it in the 1.6 release, and so in purecap io_uring01 will fail even with the skip list until the following release.
I wanted to set some kind of expectation with regards to versions and expected success. Maybe it's not needed ? Maybe being more explicit would be needed now that we know precisely which tests would or wouldn't be fixed compared to 1.5 ?
Developers playground
diff --git a/runtest/syscalls_morello_musl_skip b/runtest/syscalls_morello_musl_skip new file mode 100644 index 000000000000..aa3dff6c9449 --- /dev/null +++ b/runtest/syscalls_morello_musl_skip @@ -0,0 +1,18 @@ +# All the following tests from the syscalls test list are failing in Aarch64 +# Musl-based systems (Alpine, Debian...). Thus they depend on Musl or LTP, +# rather than the kernel, to be fixed. +# MUSL - Depends on Musl release > morello-release-1.5.0 +bpf_prog02 +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01 diff --git a/runtest/syscalls_morello_purecap_skip b/runtest/syscalls_morello_purecap_skip new file mode 100644 index 000000000000..e4d66ee1b27f --- /dev/null +++ b/runtest/syscalls_morello_purecap_skip @@ -0,0 +1,8 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in purecap.
+# These tests share capabilities between proccesses through shared memory. +# This is explicitly prevented by the PCuABI, so there's no point running them. +process_vm_readv02 +process_vm_readv03 +process_vm_writev02 diff --git a/runtest/syscalls_morello_skip b/runtest/syscalls_morello_skip new file mode 100644 index 000000000000..b9c0a2479ca9 --- /dev/null +++ b/runtest/syscalls_morello_skip @@ -0,0 +1,172 @@ +# This skip file contains all the tests skipped from the syscalls command file +# because of either the build for Morello or the system the tests are run with.
Ditto here, would say "build setup".
Makes sense.
Other than these superficial comments, the patch looks good and I confirmed locally that all the expected tests are passing.
Thanks for the review Kevin.
Kevin
On 30/03/2023 16:35, Teo Couprie Diaz wrote:
+In order for the skip files to successfully filter-out expected/known failures in AArch64, +morello-release-1.6.0 release, or newer is required.
I think that one might be confusing because it's unclear which component that requirement applies to, especially since the files we're adding require release 1.5 for the kernel and Musl, not 1.6.
Mh I'm not sure I understand correctly. There are multiple tests that require either Musl or Linux 1.6 (or, at any rate, newer than 1.5) to run successfully. The files themselves don't really need anything, for sure, but for the test suite to pass it would.
But I agree that my phrasing is unclear, slightly voluntarily. Indeed it would be correct only in compat, as you confirmed that the io_uring changes wouldn't make it in the 1.6 release, and so in purecap io_uring01 will fail even with the skip list until the following release.
I wanted to set some kind of expectation with regards to versions and expected success. Maybe it's not needed ? Maybe being more explicit would be needed now that we know precisely which tests would or wouldn't be fixed compared to 1.5 ?
Right, I do understand that some tests that we don't mention anywhere will be failing on 1.5. I don't think we need to be more specific, we are likely to get it wrong without much benefit. I guess what I don't understand at this point is what the KERN and MUSL comments in the skip files mean, as they mention 1.5. Or is it that they say "strictly greater" (> 1.5) and therefore 1.6 or above? If so I would just say ">= 1.6" which is much less likely to be misinterpreted.
Kevin
On 30/03/2023 15:43, Kevin Brodsky wrote:
On 30/03/2023 16:35, Teo Couprie Diaz wrote:
+In order for the skip files to successfully filter-out expected/known failures in AArch64, +morello-release-1.6.0 release, or newer is required.
I think that one might be confusing because it's unclear which component that requirement applies to, especially since the files we're adding require release 1.5 for the kernel and Musl, not 1.6.
Mh I'm not sure I understand correctly. There are multiple tests that require either Musl or Linux 1.6 (or, at any rate, newer than 1.5) to run successfully. The files themselves don't really need anything, for sure, but for the test suite to pass it would.
But I agree that my phrasing is unclear, slightly voluntarily. Indeed it would be correct only in compat, as you confirmed that the io_uring changes wouldn't make it in the 1.6 release, and so in purecap io_uring01 will fail even with the skip list until the following release.
I wanted to set some kind of expectation with regards to versions and expected success. Maybe it's not needed ? Maybe being more explicit would be needed now that we know precisely which tests would or wouldn't be fixed compared to 1.5 ?
Right, I do understand that some tests that we don't mention anywhere will be failing on 1.5. I don't think we need to be more specific, we are likely to get it wrong without much benefit. I guess what I don't understand at this point is what the KERN and MUSL comments in the skip files mean, as they mention 1.5. Or is it that they say "strictly greater" (> 1.5) and therefore 1.6 or above? If so I would just say ">= 1.6" which is much less likely to be misinterpreted.
Oh, yes that was I was meaning to convey. Will update with ">= 1.6" then. Should I simply remove the mention in the README then ?
Téo
Kevin
On 30/03/2023 17:05, Teo Couprie Diaz wrote:
On 30/03/2023 15:43, Kevin Brodsky wrote:
On 30/03/2023 16:35, Teo Couprie Diaz wrote:
+In order for the skip files to successfully filter-out expected/known failures in AArch64, +morello-release-1.6.0 release, or newer is required.
I think that one might be confusing because it's unclear which component that requirement applies to, especially since the files we're adding require release 1.5 for the kernel and Musl, not 1.6.
Mh I'm not sure I understand correctly. There are multiple tests that require either Musl or Linux 1.6 (or, at any rate, newer than 1.5) to run successfully. The files themselves don't really need anything, for sure, but for the test suite to pass it would.
But I agree that my phrasing is unclear, slightly voluntarily. Indeed it would be correct only in compat, as you confirmed that the io_uring changes wouldn't make it in the 1.6 release, and so in purecap io_uring01 will fail even with the skip list until the following release.
I wanted to set some kind of expectation with regards to versions and expected success. Maybe it's not needed ? Maybe being more explicit would be needed now that we know precisely which tests would or wouldn't be fixed compared to 1.5 ?
Right, I do understand that some tests that we don't mention anywhere will be failing on 1.5. I don't think we need to be more specific, we are likely to get it wrong without much benefit. I guess what I don't understand at this point is what the KERN and MUSL comments in the skip files mean, as they mention 1.5. Or is it that they say "strictly greater" (> 1.5) and therefore 1.6 or above? If so I would just say ">= 1.6" which is much less likely to be misinterpreted.
Oh, yes that was I was meaning to convey. Will update with ">= 1.6" then. Should I simply remove the mention in the README then ?
Sounds good. It doesn't hurt to keep this paragraph in the README, the comments in the skip files are easily missed. While at it, might be good to clarify it as right now it talks about successfully filtering out, which is going to happen in any case :) Maybe say something along the lines that the skip files are only comprehensive for release 1.6 and above?
Kevin
On 30/03/2023 16:22, Kevin Brodsky wrote:
On 30/03/2023 17:05, Teo Couprie Diaz wrote:
On 30/03/2023 15:43, Kevin Brodsky wrote:
On 30/03/2023 16:35, Teo Couprie Diaz wrote:
+In order for the skip files to successfully filter-out expected/known failures in AArch64, +morello-release-1.6.0 release, or newer is required.
I think that one might be confusing because it's unclear which component that requirement applies to, especially since the files we're adding require release 1.5 for the kernel and Musl, not 1.6.
Mh I'm not sure I understand correctly. There are multiple tests that require either Musl or Linux 1.6 (or, at any rate, newer than 1.5) to run successfully. The files themselves don't really need anything, for sure, but for the test suite to pass it would.
But I agree that my phrasing is unclear, slightly voluntarily. Indeed it would be correct only in compat, as you confirmed that the io_uring changes wouldn't make it in the 1.6 release, and so in purecap io_uring01 will fail even with the skip list until the following release.
I wanted to set some kind of expectation with regards to versions and expected success. Maybe it's not needed ? Maybe being more explicit would be needed now that we know precisely which tests would or wouldn't be fixed compared to 1.5 ?
Right, I do understand that some tests that we don't mention anywhere will be failing on 1.5. I don't think we need to be more specific, we are likely to get it wrong without much benefit. I guess what I don't understand at this point is what the KERN and MUSL comments in the skip files mean, as they mention 1.5. Or is it that they say "strictly greater" (> 1.5) and therefore 1.6 or above? If so I would just say ">= 1.6" which is much less likely to be misinterpreted.
Oh, yes that was I was meaning to convey. Will update with ">= 1.6" then. Should I simply remove the mention in the README then ?
Sounds good. It doesn't hurt to keep this paragraph in the README, the comments in the skip files are easily missed. While at it, might be good to clarify it as right now it talks about successfully filtering out, which is going to happen in any case :) Maybe say something along the lines that the skip files are only comprehensive for release 1.6 and above?
Yeah that's fair, I like your phrasing ! Will send v6 with fixes for your comments and the added tests to the skip list.
Thanks for the comments ! Téo
Kevin
linux-morello-ltp@op-lists.linaro.org
-
Kevin Brodsky -
Teo Couprie Diaz