Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com --- README.rst | 21 ++++++++ runtest/syscalls_compat.skip | 96 ++++++++++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 16 ++++++ 3 files changed, 133 insertions(+) create mode 100644 runtest/syscalls_compat.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..4efa6d38f 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,27 @@ Running tests:
runltp -f morello_transitional,morello_transitional_extended
+Skipping tests: +--------------- + +In certain cases, you might want to skip tests that are known to be failing or are yet to be supported. + +This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``. + +We currently provide the following skip files under ``runtest/``: + - ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system. + - ``syscalls_compat.skip``: Skips all tests not considered for COMPAT64 with a PCuABI kernel. + +``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution. + +For example, in COMPAT64 on a Musl system to test all syscalls but skip all known failures, one +could run the following command: + +.. code-block:: + + ./runltp -f syscalls -S <( cat runtest/syscalls_{compat,musl}.skip )
Developers playground --------------------- diff --git a/runtest/syscalls_compat.skip b/runtest/syscalls_compat.skip new file mode 100644 index 000000000..b9af14a3a --- /dev/null +++ b/runtest/syscalls_compat.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in COMPAT64. + +# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01 + +# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64 diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..92c952dac --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,16 @@ +# All the following tests from the syscalls test list are failing in a standard +# Aarch64 Musl-based distribution (Alpine Linux). Thus they depend on Musl +# or LTP, rather than the kernel, to be fixed. +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01 +sigwaitinfo01
Hi Teo On Tue, Feb 21, 2023 at 04:43:33PM +0000, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
README.rst | 21 ++++++++ runtest/syscalls_compat.skip | 96 ++++++++++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 16 ++++++ 3 files changed, 133 insertions(+) create mode 100644 runtest/syscalls_compat.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..4efa6d38f 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,27 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system.
- ``syscalls_compat.skip``: Skips all tests not considered for COMPAT64 with a PCuABI kernel.
I think it would be better to replace all COMPAT64 mentions here (as this is kernel specific term) and use 'plain aarch64' instead (that might include renaming the skip file as well)
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+For example, in COMPAT64 on a Musl system to test all syscalls but skip all known failures, one +could run the following command:
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_{compat,musl}.skip )
Developers playground
diff --git a/runtest/syscalls_compat.skip b/runtest/syscalls_compat.skip new file mode 100644 index 000000000..b9af14a3a --- /dev/null +++ b/runtest/syscalls_compat.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in COMPAT64.
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64 diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..92c952dac --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,16 @@ +# All the following tests from the syscalls test list are failing in a standard +# Aarch64 Musl-based distribution (Alpine Linux). Thus they depend on Musl
Is Alpine Linux a standard distribution? Are those known failures with reported bugs against either ?
--- BR B.
+# or LTP, rather than the kernel, to be fixed. +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01
+sigwaitinfo01
2.25.1
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
Hi Beata,
On 23/02/2023 11:05, Beata Michalska wrote:
Hi Teo On Tue, Feb 21, 2023 at 04:43:33PM +0000, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
README.rst | 21 ++++++++ runtest/syscalls_compat.skip | 96 ++++++++++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 16 ++++++ 3 files changed, 133 insertions(+) create mode 100644 runtest/syscalls_compat.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..4efa6d38f 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,27 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system.
- ``syscalls_compat.skip``: Skips all tests not considered for COMPAT64 with a PCuABI kernel.
I think it would be better to replace all COMPAT64 mentions here (as this is kernel specific term) and use 'plain aarch64' instead (that might include renaming the skip file as well)
I see, that does make sense. Will update in a v2.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+For example, in COMPAT64 on a Musl system to test all syscalls but skip all known failures, one +could run the following command:
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_{compat,musl}.skip )
Developers playground
diff --git a/runtest/syscalls_compat.skip b/runtest/syscalls_compat.skip new file mode 100644 index 000000000..b9af14a3a --- /dev/null +++ b/runtest/syscalls_compat.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in COMPAT64.
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64 diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..92c952dac --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,16 @@ +# All the following tests from the syscalls test list are failing in a standard +# Aarch64 Musl-based distribution (Alpine Linux). Thus they depend on Musl
Is Alpine Linux a standard distribution?
I would say it is quite well known: it's the second most downloaded Docker image for a Linux distribution (Behind Ubuntu) and a lot of others use it as a base, for example. It's also somewhat supported by LTP as it's part of the build CI. But I guess it really depends on what you consider standard ?
Are those known failures with reported bugs against either ?
On the other hand, Musl is not officially supported in LTP. There doesn't seem to be anything related to those issues on the LTP side, and not much more on the Alpine side either. LTP is available as a package there but I couldn't find the failing tests on the issue tracker, although there was some recent activity from one of the LTP maintainers who fixed some lint issues for the Alpine package.
BR B.
Thanks for having a look !
Téo
+# or LTP, rather than the kernel, to be fixed. +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01
+sigwaitinfo01
2.25.1
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
On Thu, Feb 23, 2023 at 05:36:32PM +0000, Teo Couprie Diaz wrote:
Hi Beata,
On 23/02/2023 11:05, Beata Michalska wrote:
Hi Teo On Tue, Feb 21, 2023 at 04:43:33PM +0000, Teo Couprie Diaz wrote:
Add two skip files for running the syscalls set of tests in compat. One skips the tests failing because of Musl, the second the tests that we skip or don't build at all.
Update the README to reflect those additions and explain how to use the skip files.
Signed-off-by: Teo Couprie Diaz teo.coupriediaz@arm.com
README.rst | 21 ++++++++ runtest/syscalls_compat.skip | 96 ++++++++++++++++++++++++++++++++++++ runtest/syscalls_musl.skip | 16 ++++++ 3 files changed, 133 insertions(+) create mode 100644 runtest/syscalls_compat.skip create mode 100644 runtest/syscalls_musl.skip
diff --git a/README.rst b/README.rst index 8b6e3a776..4efa6d38f 100644 --- a/README.rst +++ b/README.rst @@ -136,6 +136,27 @@ Running tests: runltp -f morello_transitional,morello_transitional_extended +Skipping tests: +---------------
+In certain cases, you might want to skip tests that are known to be failing or are yet to be supported.
+This is done by passing a file containing the names of the tests to be skip with the option ``-S`` to +``runltp``.
+We currently provide the following skip files under ``runtest/``:
- ``syscalls_musl.skip``: Skips all tests failing on a regular Aarch64 Musl based system.
- ``syscalls_compat.skip``: Skips all tests not considered for COMPAT64 with a PCuABI kernel.
I think it would be better to replace all COMPAT64 mentions here (as this is kernel specific term) and use 'plain aarch64' instead (that might include renaming the skip file as well)
I see, that does make sense. Will update in a v2.
+``runltp`` does not support passing multiple skip files, however this can be circumvented by using +process substitution.
+For example, in COMPAT64 on a Musl system to test all syscalls but skip all known failures, one +could run the following command:
+.. code-block::
- ./runltp -f syscalls -S <( cat runtest/syscalls_{compat,musl}.skip ) Developers playground
diff --git a/runtest/syscalls_compat.skip b/runtest/syscalls_compat.skip new file mode 100644 index 000000000..b9af14a3a --- /dev/null +++ b/runtest/syscalls_compat.skip @@ -0,0 +1,96 @@ +# This skip file contains all the tests skipped from the syscalls command file +# for testing the PCuABI kernel in COMPAT64.
+# leapsec01 is skipped as systemd-timesyncd interfers with it and it doesn't +# test any worthwhile functionality. +leapsec01
+# We do not build most *_64 tests, skip them. +posix_fadvise01_64 +posix_fadvise02_64 +posix_fadvise03_64 +posix_fadvise04_64 +fcntl01_64 +fcntl02_64 +fcntl03_64 +fcntl04_64 +fcntl05_64 +fcntl06_64 +fcntl07_64 +fcntl08_64 +fcntl09_64 +fcntl10_64 +fcntl11_64 +fcntl12_64 +fcntl13_64 +fcntl14_64 +fcntl15_64 +fcntl16_64 +fcntl17_64 +fcntl18_64 +fcntl19_64 +fcntl20_64 +fcntl21_64 +fcntl22_64 +fcntl23_64 +fcntl24_64 +fcntl25_64 +fcntl26_64 +fcntl27_64 +fcntl28_64 +fcntl29_64 +fcntl30_64 +fcntl31_64 +fcntl32_64 +fcntl33_64 +fcntl34_64 +fcntl35_64 +fcntl36_64 +fcntl37_64 +fcntl38_64 +fcntl39_64 +fstat02_64 +fstat03_64 +fstatfs01_64 +fstatfs02_64 +ftruncate01_64 +ftruncate03_64 +ftruncate04_64 +lstat01A_64 +lstat02_64 +pread01_64 +pread02_64 +preadv01_64 +preadv02_64 +preadv03_64 +preadv201_64 +preadv202_64 +preadv203_64 +pselect01_64 +pselect02_64 +pselect03_64 +pwrite01_64 +pwrite02_64 +pwrite03_64 +pwrite04_64 +pwritev01_64 +pwritev02_64 +pwritev03_64 +pwritev201_64 +pwritev202_64 +sendfile02_64 +sendfile03_64 +sendfile04_64 +sendfile05_64 +sendfile06_64 +sendfile07_64 +sendfile08_64 +sendfile09_64 +stat01_64 +stat02_64 +stat03_64 +statfs01_64 +statfs02_64 +statfs03_64 +truncate02_64 +truncate03_64 diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..92c952dac --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,16 @@ +# All the following tests from the syscalls test list are failing in a standard +# Aarch64 Musl-based distribution (Alpine Linux). Thus they depend on Musl
Is Alpine Linux a standard distribution?
I would say it is quite well known: it's the second most downloaded Docker image for a Linux distribution (Behind Ubuntu) and a lot of others use it as a base, for example. It's also somewhat supported by LTP as it's part of the build CI. But I guess it really depends on what you consider standard ?
I think I got caught up with exactly that. Note I am not debating that Alpine is a well-established distribution. How about rephrasing it a bit as of: "are failing on standard Aarch64 Linux Musl-based distribution' and skipping the mentioning of Alpine at all ? Those are to be affected by Musl implementation, in the end ?
Are those known failures with reported bugs against either ?
On the other hand, Musl is not officially supported in LTP. There doesn't seem to be anything related to those issues on the LTP side, and not much more on the Alpine side either. LTP is available as a package there but I couldn't find the failing tests on the issue tracker, although there was some recent activity from one of the LTP maintainers who fixed some lint issues for the Alpine package.
Was asking out of curiosity as if there were any issues raised against Musl/LTP it would be nice to track those. Thinking at loud (and no need to follow up on this one): maybe it would be possible to change the ci.yml file and try at least to run the tests on Apline + Musl ? LTP CI is using official docker images and there is an aarch64 flavor there: https://github.com/alpinelinux/docker-alpine/blob/d8ed1701dac37e1b6db026bec0...
--- BR B.
BR B.
Thanks for having a look !
Téo
+# or LTP, rather than the kernel, to be fixed. +gethostbyname_r01 +mq_open01 +nftw01 +nftw6401 +open13 +perf_event_open02 +recvmmsg01 +sbrk01 +sendmsg01 +setgroups03 +sigrelse01 +sigtimedwait01
+sigwaitinfo01
2.25.1
linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
On 24/02/2023 13:58, Beata Michalska wrote:
diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..92c952dac --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,16 @@ +# All the following tests from the syscalls test list are failing in a standard +# Aarch64 Musl-based distribution (Alpine Linux). Thus they depend on Musl
Is Alpine Linux a standard distribution?
I would say it is quite well known: it's the second most downloaded Docker image for a Linux distribution (Behind Ubuntu) and a lot of others use it as a base, for example. It's also somewhat supported by LTP as it's part of the build CI. But I guess it really depends on what you consider standard ?
I think I got caught up with exactly that. Note I am not debating that Alpine is a well-established distribution. How about rephrasing it a bit as of: "are failing on standard Aarch64 Linux Musl-based distribution' and skipping the mentioning of Alpine at all ? Those are to be affected by Musl implementation, in the end ?
Might be worth pointing out the existence of ci/alpine.sh, linked from doc/supported-kernel-libc-versions.txt. So I'd say mentioning Alpine is relevant, though indeed it is really about Musl in the end.
Kevin
On Mon, Feb 27, 2023 at 05:03:44PM +0100, Kevin Brodsky wrote:
On 24/02/2023 13:58, Beata Michalska wrote:
diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..92c952dac --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,16 @@ +# All the following tests from the syscalls test list are failing in a standard +# Aarch64 Musl-based distribution (Alpine Linux). Thus they depend on Musl
Is Alpine Linux a standard distribution?
I would say it is quite well known: it's the second most downloaded Docker image for a Linux distribution (Behind Ubuntu) and a lot of others use it as a base, for example. It's also somewhat supported by LTP as it's part of the build CI. But I guess it really depends on what you consider standard ?
I think I got caught up with exactly that. Note I am not debating that Alpine is a well-established distribution. How about rephrasing it a bit as of: "are failing on standard Aarch64 Linux Musl-based distribution' and skipping the mentioning of Alpine at all ? Those are to be affected by Musl implementation, in the end ?
Might be worth pointing out the existence of ci/alpine.sh, linked from doc/supported-kernel-libc-versions.txt. So I'd say mentioning Alpine is relevant, though indeed it is really about Musl in the end.
The list of tests from that file does not actually overlap with the skip list proposed (?), so I am not convinced that it is a good argument to mention Alpine explicitly. Also there seems to be no trace (?) of those tests failing on a setup Alpine + Musl (which is why I was wondering if there are any reports). If Musl is the culprit for those failures than why attributing that to any specific distribution ? Especially that Alpine is not the only one being used for testing , and not the only one (?) where those tests fail ? I do not mind keeping mentioning Alpine, but I'd rather make the message clear. Or am I completely missing the point here which might very much be the case, so bare with me , please.
--- BR B.
Kevin
On 27/02/2023 19:25, Beata Michalska wrote:
On Mon, Feb 27, 2023 at 05:03:44PM +0100, Kevin Brodsky wrote:
On 24/02/2023 13:58, Beata Michalska wrote:
diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip new file mode 100644 index 000000000..92c952dac --- /dev/null +++ b/runtest/syscalls_musl.skip @@ -0,0 +1,16 @@ +# All the following tests from the syscalls test list are failing in a standard +# Aarch64 Musl-based distribution (Alpine Linux). Thus they depend on Musl
Is Alpine Linux a standard distribution?
I would say it is quite well known: it's the second most downloaded Docker image for a Linux distribution (Behind Ubuntu) and a lot of others use it as a base, for example. It's also somewhat supported by LTP as it's part of the build CI. But I guess it really depends on what you consider standard ?
I think I got caught up with exactly that. Note I am not debating that Alpine is a well-established distribution. How about rephrasing it a bit as of: "are failing on standard Aarch64 Linux Musl-based distribution' and skipping the mentioning of Alpine at all ? Those are to be affected by Musl implementation, in the end ?
Might be worth pointing out the existence of ci/alpine.sh, linked from doc/supported-kernel-libc-versions.txt. So I'd say mentioning Alpine is relevant, though indeed it is really about Musl in the end.
The list of tests from that file does not actually overlap with the skip list proposed (?), so I am not convinced that it is a good argument to mention Alpine explicitly. Also there seems to be no trace (?) of those tests failing on a setup Alpine + Musl (which is why I was wondering if there are any reports). If Musl is the culprit for those failures than why attributing that to any specific distribution ? Especially that Alpine is not the only one being used for testing , and not the only one (?) where those tests fail ? I do not mind keeping mentioning Alpine, but I'd rather make the message clear. Or am I completely missing the point here which might very much be the case, so bare with me , please.
No strong opinion either way, I'm just saying that LTP being influenced by a number of factors besides the libc (as we've figured out with the cgroup config for instance), it is completely possible that some failures may be different on another Musl-based distro. Ideally we would know exactly why all of these fail and mentioning the environment would be irrelevant, but since we precisely haven't investigated these failures (yet) more information doesn't seem to hurt.
In any case this really doesn't matter in the grand scheme of things so I'll be happy when this is merged, whether Alpine is mentioned or not :)
Kevin
Hi Beata, Kevin, (Not sure if it's correct etiquette to reply to both people from a down-thread reply ?)
On 28/02/2023 08:00, Kevin Brodsky wrote:
On 27/02/2023 19:25, Beata Michalska wrote:
On Mon, Feb 27, 2023 at 05:03:44PM +0100, Kevin Brodsky wrote:
On 24/02/2023 13:58, Beata Michalska wrote:
> diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip > new file mode 100644 > index 000000000..92c952dac > --- /dev/null > +++ b/runtest/syscalls_musl.skip > @@ -0,0 +1,16 @@ > +# All the following tests from the syscalls test list are failing in a standard > +# Aarch64 Musl-based distribution (Alpine Linux). Thus they depend on Musl Is Alpine Linux a standard distribution?
I would say it is quite well known: it's the second most downloaded Docker image for a Linux distribution (Behind Ubuntu) and a lot of others use it as a base, for example. It's also somewhat supported by LTP as it's part of the build CI. But I guess it really depends on what you consider standard ?
I think I got caught up with exactly that. Note I am not debating that Alpine is a well-established distribution. How about rephrasing it a bit as of: "are failing on standard Aarch64 Linux Musl-based distribution' and skipping the mentioning of Alpine at all ? Those are to be affected by Musl implementation, in the end ?
Might be worth pointing out the existence of ci/alpine.sh, linked from doc/supported-kernel-libc-versions.txt. So I'd say mentioning Alpine is relevant, though indeed it is really about Musl in the end.
The list of tests from that file does not actually overlap with the skip list proposed (?),[...]
Indeed, ci/alpine.sh lists tests that _do not compile_, rather than tests that fail. I had to remove them manually for my testing on Alpine as they did fail to compile, but interestingly they build fine for us.
[...] so I am not convinced that it is a good argument to mention Alpine explicitly. Also there seems to be no trace (?) of those tests failing on a setup Alpine + Musl (which is why I was wondering if there are any reports). If Musl is the culprit for those failures than why attributing that to any specific distribution ? Especially that Alpine is not the only one being used for testing , and not the only one (?) where those tests fail ?
You're right that it also clearly fails on our Debian distribution using Musl, which is the whole point of having this list in the first place. My reasoning for mentioning it was more in the lines of "This is what we used to make sure this wasn't just our kernel having issues", but it might not be necessary, as you highlight.
I do not mind keeping mentioning Alpine, but I'd rather make the message clear. Or am I completely missing the point here which might very much be the case, so bare with me , please.
No strong opinion either way, I'm just saying that LTP being influenced by a number of factors besides the libc (as we've figured out with the cgroup config for instance), it is completely possible that some failures may be different on another Musl-based distro. Ideally we would know exactly why all of these fail and mentioning the environment would be irrelevant, but since we precisely haven't investigated these failures (yet) more information doesn't seem to hurt.
To be clear, I'm fine either way as well ! I just tried to have as much information as possible, but maybe it's too much, or maybe it's not clear enough.
Maybe being explicit on the mention would help then ? Something explaining that this test failures are shared between our Debian+PCuABI kernel+Musl system and a regular Alpine system, which is based on Musl. Would it be relevant for me to build LTP with Musl on my own machine (Ubuntu, will have to statically link the tests otherwise they would call into glibc) and check if the failures are consistent with what I already observed ? I don't feel like it would be very useful but if you want an extra layer of validation I'll take care of it.
In any case this really doesn't matter in the grand scheme of things so I'll be happy when this is merged, whether Alpine is mentioned or not :)
Kevin
Thanks both for the discussion, Téo
On Tue, Feb 28, 2023 at 10:43:57AM +0000, Teo Couprie Diaz wrote:
Hi Beata, Kevin, (Not sure if it's correct etiquette to reply to both people from a down-thread reply ?)
On 28/02/2023 08:00, Kevin Brodsky wrote:
On 27/02/2023 19:25, Beata Michalska wrote:
On Mon, Feb 27, 2023 at 05:03:44PM +0100, Kevin Brodsky wrote:
On 24/02/2023 13:58, Beata Michalska wrote:
> > diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip > > new file mode 100644 > > index 000000000..92c952dac > > --- /dev/null > > +++ b/runtest/syscalls_musl.skip > > @@ -0,0 +1,16 @@ > > +# All the following tests from the syscalls test list are failing in a standard > > +# Aarch64 Musl-based distribution (Alpine Linux). Thus they depend on Musl > Is Alpine Linux a standard distribution? I would say it is quite well known: it's the second most downloaded Docker image for a Linux distribution (Behind Ubuntu) and a lot of others use it as a base, for example. It's also somewhat supported by LTP as it's part of the build CI. But I guess it really depends on what you consider standard ?
I think I got caught up with exactly that. Note I am not debating that Alpine is a well-established distribution. How about rephrasing it a bit as of: "are failing on standard Aarch64 Linux Musl-based distribution' and skipping the mentioning of Alpine at all ? Those are to be affected by Musl implementation, in the end ?
Might be worth pointing out the existence of ci/alpine.sh, linked from doc/supported-kernel-libc-versions.txt. So I'd say mentioning Alpine is relevant, though indeed it is really about Musl in the end.
The list of tests from that file does not actually overlap with the skip list proposed (?),[...]
Indeed, ci/alpine.sh lists tests that _do not compile_, rather than tests that fail. I had to remove them manually for my testing on Alpine as they did fail to compile, but interestingly they build fine for us.
[...] so I am not convinced that it is a good argument to mention Alpine explicitly. Also there seems to be no trace (?) of those tests failing on a setup Alpine + Musl (which is why I was wondering if there are any reports). If Musl is the culprit for those failures than why attributing that to any specific distribution ? Especially that Alpine is not the only one being used for testing , and not the only one (?) where those tests fail ?
You're right that it also clearly fails on our Debian distribution using Musl, which is the whole point of having this list in the first place. My reasoning for mentioning it was more in the lines of "This is what we used to make sure this wasn't just our kernel having issues", but it might not be necessary, as you highlight.
I do not mind keeping mentioning Alpine, but I'd rather make the message clear. Or am I completely missing the point here which might very much be the case, so bare with me , please.
No strong opinion either way, I'm just saying that LTP being influenced by a number of factors besides the libc (as we've figured out with the cgroup config for instance), it is completely possible that some failures may be different on another Musl-based distro. Ideally we would know exactly why all of these fail and mentioning the environment would be irrelevant, but since we precisely haven't investigated these failures (yet) more information doesn't seem to hurt.
To be clear, I'm fine either way as well ! I just tried to have as much information as possible, but maybe it's too much, or maybe it's not clear enough.
Maybe being explicit on the mention would help then ? Something explaining that this test failures are shared between our Debian+PCuABI kernel+Musl system and a regular Alpine system, which is based on Musl. Would it be relevant for me to build LTP with Musl on my own machine (Ubuntu, will have to statically link the tests otherwise they would call into glibc) and check if the failures are consistent with what I already observed ? I don't feel like it would be very useful but if you want an extra layer of validation I'll take care of it.
I do not think it is necessary to perform yet another validation. Let's make a compromise here and mention both Alpine and Debian so that it is obvious the failures are not directly related to the first (just toss Debian next to Alpine in the brackets in your comment)
--- BR B.
In any case this really doesn't matter in the grand scheme of things so I'll be happy when this is merged, whether Alpine is mentioned or not :)
Kevin
Thanks both for the discussion, Téo _______________________________________________ linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
On 28/02/2023 11:45, Beata Michalska wrote:
On Tue, Feb 28, 2023 at 10:43:57AM +0000, Teo Couprie Diaz wrote:
Hi Beata, Kevin, (Not sure if it's correct etiquette to reply to both people from a down-thread reply ?)
On 28/02/2023 08:00, Kevin Brodsky wrote:
On 27/02/2023 19:25, Beata Michalska wrote:
On Mon, Feb 27, 2023 at 05:03:44PM +0100, Kevin Brodsky wrote:
On 24/02/2023 13:58, Beata Michalska wrote:
>>> diff --git a/runtest/syscalls_musl.skip b/runtest/syscalls_musl.skip >>> new file mode 100644 >>> index 000000000..92c952dac >>> --- /dev/null >>> +++ b/runtest/syscalls_musl.skip >>> @@ -0,0 +1,16 @@ >>> +# All the following tests from the syscalls test list are failing in a standard >>> +# Aarch64 Musl-based distribution (Alpine Linux). Thus they depend on Musl >> Is Alpine Linux a standard distribution? > I would say it is quite well known: it's the second most downloaded Docker > image for a Linux distribution (Behind Ubuntu) and a lot of others use it as > a base, for example. It's also somewhat supported by LTP as it's part of the > build CI. > But I guess it really depends on what you consider standard ? > I think I got caught up with exactly that. Note I am not debating that Alpine is a well-established distribution. How about rephrasing it a bit as of: "are failing on standard Aarch64 Linux Musl-based distribution' and skipping the mentioning of Alpine at all ? Those are to be affected by Musl implementation, in the end ?
Might be worth pointing out the existence of ci/alpine.sh, linked from doc/supported-kernel-libc-versions.txt. So I'd say mentioning Alpine is relevant, though indeed it is really about Musl in the end.
The list of tests from that file does not actually overlap with the skip list proposed (?),[...]
Indeed, ci/alpine.sh lists tests that _do not compile_, rather than tests that fail. I had to remove them manually for my testing on Alpine as they did fail to compile, but interestingly they build fine for us.
[...] so I am not convinced that it is a good argument to mention Alpine explicitly. Also there seems to be no trace (?) of those tests failing on a setup Alpine + Musl (which is why I was wondering if there are any reports). If Musl is the culprit for those failures than why attributing that to any specific distribution ? Especially that Alpine is not the only one being used for testing , and not the only one (?) where those tests fail ?
You're right that it also clearly fails on our Debian distribution using Musl, which is the whole point of having this list in the first place. My reasoning for mentioning it was more in the lines of "This is what we used to make sure this wasn't just our kernel having issues", but it might not be necessary, as you highlight.
I do not mind keeping mentioning Alpine, but I'd rather make the message clear. Or am I completely missing the point here which might very much be the case, so bare with me , please.
No strong opinion either way, I'm just saying that LTP being influenced by a number of factors besides the libc (as we've figured out with the cgroup config for instance), it is completely possible that some failures may be different on another Musl-based distro. Ideally we would know exactly why all of these fail and mentioning the environment would be irrelevant, but since we precisely haven't investigated these failures (yet) more information doesn't seem to hurt.
To be clear, I'm fine either way as well ! I just tried to have as much information as possible, but maybe it's too much, or maybe it's not clear enough.
Maybe being explicit on the mention would help then ? Something explaining that this test failures are shared between our Debian+PCuABI kernel+Musl system and a regular Alpine system, which is based on Musl. Would it be relevant for me to build LTP with Musl on my own machine (Ubuntu, will have to statically link the tests otherwise they would call into glibc) and check if the failures are consistent with what I already observed ? I don't feel like it would be very useful but if you want an extra layer of validation I'll take care of it.
I do not think it is necessary to perform yet another validation. Let's make a compromise here and mention both Alpine and Debian so that it is obvious the failures are not directly related to the first (just toss Debian next to Alpine in the brackets in your comment)
Added a mention of Debian and slightly reworded to be a bit broader.
Before you replied I also started the tests to run while having lunch on another Musl-based distribution (VoidLinux) and can confirm what we already had in the skip files. There are a few other failures which PASS on our side, but the same goes for Alpine, which matches Kevin's intuition.
Thanks again for the review, I sent out the v2. Best regards, Téo
BR B.
In any case this really doesn't matter in the grand scheme of things so I'll be happy when this is merged, whether Alpine is mentioned or not :)
Kevin
Thanks both for the discussion, T�o _______________________________________________ linux-morello-ltp mailing list -- linux-morello-ltp@op-lists.linaro.org To unsubscribe send an email to linux-morello-ltp-leave@op-lists.linaro.org
linux-morello-ltp@op-lists.linaro.org
-
Beata Michalska -
Kevin Brodsky -
Teo Couprie Diaz