June 2024 - ldcg-ai-ml - op-lists.linaro.org

Build failed in Jenkins: Tensorflow Python package (manylinux2014 nightly) #1205

by ci_notify＠linaro.org

See <https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/1205/dis…> Changes: ------------------------------------------ [...truncated 1.57 MiB...] Stats over 12 runs: max = 28.4s, min = 8.2s, avg = 17.5s, dev = 6.4s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 24.6s Stats over 12 runs: max = 24.6s, min = 7.1s, avg = 13.8s, dev = 5.6s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 18.7s Stats over 12 runs: max = 18.7s, min = 4.9s, avg = 8.4s, dev = 3.9s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 13.9s Stats over 12 runs: max = 13.9s, min = 4.4s, avg = 6.9s, dev = 2.7s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 28.0s Stats over 12 runs: max = 28.0s, min = 12.6s, avg = 16.2s, dev = 4.1s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 27.4s Stats over 12 runs: max = 27.4s, min = 13.9s, avg = 17.5s, dev = 3.5s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 32.2s Stats over 15 runs: max = 32.2s, min = 12.6s, avg = 18.5s, dev = 4.4s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 146.9s Stats over 15 runs: max = 146.9s, min = 109.3s, avg = 127.4s, dev = 10.3s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 62.9s Stats over 15 runs: max = 62.9s, min = 8.3s, avg = 16.9s, dev = 14.2s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 16.9s Stats over 16 runs: max = 16.9s, min = 5.1s, avg = 10.6s, dev = 3.5s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 47.3s Stats over 16 runs: max = 47.3s, min = 12.6s, avg = 27.4s, dev = 9.9s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 28.0s Stats over 16 runs: max = 28.0s, min = 6.1s, avg = 10.8s, dev = 5.0s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 10.7s Stats over 16 runs: max = 10.7s, min = 4.5s, avg = 6.4s, dev = 1.9s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 20.7s Stats over 16 runs: max = 20.7s, min = 8.4s, avg = 12.5s, dev = 3.3s //tensorflow/python/ops:image_ops_test_cpu PASSED in 26.3s Stats over 16 runs: max = 26.3s, min = 8.5s, avg = 16.1s, dev = 4.2s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 81.1s Stats over 17 runs: max = 81.1s, min = 10.7s, avg = 35.5s, dev = 21.9s //tensorflow/python/data/kernel_tests:map_test PASSED in 54.9s Stats over 19 runs: max = 54.9s, min = 28.8s, avg = 38.8s, dev = 6.9s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 11.2s Stats over 20 runs: max = 11.2s, min = 4.6s, avg = 5.4s, dev = 1.4s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 11.1s Stats over 20 runs: max = 11.1s, min = 4.3s, avg = 5.0s, dev = 1.4s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 18.5s Stats over 20 runs: max = 18.5s, min = 3.6s, avg = 5.5s, dev = 3.2s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 12.2s Stats over 20 runs: max = 12.2s, min = 3.6s, avg = 5.2s, dev = 2.1s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 14.3s Stats over 20 runs: max = 14.3s, min = 5.1s, avg = 6.0s, dev = 1.9s //tensorflow/dtensor/python/tests:rng_test_cpu PASSED in 16.7s Stats over 20 runs: max = 16.7s, min = 7.1s, avg = 8.7s, dev = 2.0s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 26.4s Stats over 20 runs: max = 26.4s, min = 15.5s, avg = 18.5s, dev = 2.6s //tensorflow/python/kernel_tests:metrics_test PASSED in 53.6s Stats over 20 runs: max = 53.6s, min = 7.3s, avg = 22.6s, dev = 12.3s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 10.3s Stats over 20 runs: max = 10.3s, min = 4.3s, avg = 5.6s, dev = 1.4s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 14.8s Stats over 20 runs: max = 14.8s, min = 4.3s, avg = 7.0s, dev = 3.1s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 49.5s Stats over 20 runs: max = 49.5s, min = 4.6s, avg = 15.8s, dev = 15.4s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 119.4s Stats over 20 runs: max = 119.4s, min = 28.7s, avg = 55.5s, dev = 21.3s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 13.2s Stats over 20 runs: max = 13.2s, min = 6.1s, avg = 7.8s, dev = 1.7s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 19.6s Stats over 20 runs: max = 19.6s, min = 7.8s, avg = 12.5s, dev = 3.1s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 319.1s Stats over 20 runs: max = 319.1s, min = 43.5s, avg = 123.1s, dev = 78.0s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 26.1s Stats over 20 runs: max = 26.1s, min = 5.2s, avg = 13.4s, dev = 7.7s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 23.9s Stats over 20 runs: max = 23.9s, min = 7.6s, avg = 14.9s, dev = 5.7s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 25.1s Stats over 20 runs: max = 25.1s, min = 15.4s, avg = 19.1s, dev = 2.2s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 65.4s Stats over 20 runs: max = 65.4s, min = 7.9s, avg = 32.0s, dev = 21.3s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 29.8s Stats over 20 runs: max = 29.8s, min = 12.6s, avg = 17.2s, dev = 4.1s //tensorflow/compiler/tests:unary_ops_test_cpu FAILED in 3 out of 22 in 17.0s Stats over 22 runs: max = 17.0s, min = 4.2s, avg = 8.6s, dev = 4.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu/shard_18_of_20/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu/shard_18_of_20/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu/shard_18_of_20/test_attempts/attempt_2.log //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test FAILED in 3 out of 22 in 19.2s Stats over 22 runs: max = 19.2s, min = 3.7s, avg = 9.4s, dev = 5.0s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu_mlir_bridge_test/shard_18_of_20/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu_mlir_bridge_test/shard_18_of_20/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu_mlir_bridge_test/shard_18_of_20/test_attempts/attempt_2.log //tensorflow/python/data/kernel_tests:interleave_test PASSED in 41.4s Stats over 24 runs: max = 41.4s, min = 11.5s, avg = 23.7s, dev = 8.4s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 36.6s Stats over 24 runs: max = 36.6s, min = 5.3s, avg = 12.7s, dev = 8.1s //tensorflow/dtensor/python/tests:multi_device_spmd_test_cpu PASSED in 36.3s Stats over 25 runs: max = 36.3s, min = 26.3s, avg = 30.3s, dev = 2.5s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 23.1s Stats over 30 runs: max = 23.1s, min = 4.6s, avg = 9.9s, dev = 4.9s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 23.4s Stats over 32 runs: max = 23.4s, min = 5.0s, avg = 11.0s, dev = 5.1s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 25.3s Stats over 32 runs: max = 25.3s, min = 4.3s, avg = 11.8s, dev = 5.3s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 216.2s Stats over 35 runs: max = 216.2s, min = 5.5s, avg = 24.5s, dev = 37.0s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 220.9s Stats over 35 runs: max = 220.9s, min = 5.4s, avg = 24.6s, dev = 37.8s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.5s Stats over 48 runs: max = 1.5s, min = 0.4s, avg = 0.6s, dev = 0.2s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 78.8s Stats over 50 runs: max = 78.8s, min = 42.9s, avg = 58.5s, dev = 9.0s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 16.7s Stats over 50 runs: max = 16.7s, min = 3.8s, avg = 9.9s, dev = 3.5s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 16.4s Stats over 50 runs: max = 16.4s, min = 4.0s, avg = 9.5s, dev = 3.4s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 69.1s Stats over 50 runs: max = 69.1s, min = 44.5s, avg = 53.9s, dev = 6.2s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 21.8s Stats over 50 runs: max = 21.8s, min = 5.6s, avg = 11.4s, dev = 3.9s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_onednn_grad_test PASSED in 20.5s Stats over 50 runs: max = 20.5s, min = 5.6s, avg = 11.4s, dev = 3.6s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 28.9s Stats over 50 runs: max = 28.9s, min = 8.4s, avg = 16.0s, dev = 5.6s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 11.6s Stats over 50 runs: max = 11.6s, min = 3.9s, avg = 4.8s, dev = 1.3s Executed 3085 out of 3087 tests: 3083 tests pass, 2 fail to build and 2 fail locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. stdout_lines: <omitted> ...ignoring TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages] *** Tuesday 18 June 2024 22:59:24 +0100 (2:15:05.506) 3:49:11.879 ********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save list of installed Python packages] *** Tuesday 18 June 2024 22:59:26 +0100 (0:00:01.868) 3:49:13.747 ********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save tests output to file] ******** Tuesday 18 June 2024 22:59:27 +0100 (0:00:00.695) 3:49:14.442 ********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save tests stderr output to file] *** Tuesday 18 June 2024 22:59:28 +0100 (0:00:00.697) 3:49:15.139 ********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) Extract number of tests run] ****** Tuesday 18 June 2024 22:59:28 +0100 (0:00:00.704) 3:49:15.844 ********** ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) Show number of tests run] ********* Tuesday 18 June 2024 22:59:28 +0100 (0:00:00.081) 3:49:15.926 ********** ok: [localhost] => ut_tests_line: 'Executed 3085 out of 3087 tests: 3083 tests pass, 2 fail to build and 2 fail locally.' TASK [tensorflow_in_container : (Python 3.9) Extract failed tests] ************* Tuesday 18 June 2024 22:59:28 +0100 (0:00:00.039) 3:49:15.965 ********** ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) Show which tests are flaky] ******* Tuesday 18 June 2024 22:59:29 +0100 (0:00:00.161) 3:49:16.126 ********** ok: [localhost] => tests_flaky: - //tensorflow/python/kernel_tests/linalg:matrix_triangular_solve_op_test_cpu FLAKY, failed in 1 out of 4 in 900.4s TASK [tensorflow_in_container : (Python 3.9) Show which tests failed] ********** Tuesday 18 June 2024 22:59:29 +0100 (0:00:00.038) 3:49:16.165 ********** ok: [localhost] => tests_failed: - 'FAILED: //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test (Summary)' - 'FAILED: //tensorflow/compiler/tests:unary_ops_test_cpu (Summary)' - 'FAILED: Build did NOT complete successfully' TASK [tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one] *** Tuesday 18 June 2024 22:59:29 +0100 (0:00:00.038) 3:49:16.203 ********** fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- + for wheel in /tf/pkg/*.whl + echo 'Checking and renaming /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl...' + python3 -m auditwheel repair --plat manylinux2014_aarch64 /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl --wheel-dir /tf/pkg + tee check.txt real 2m30.267s user 2m14.570s sys 0m15.596s ++ grep --extended-regexp --only-matching '/tf/pkg/\S+.whl' check.txt + new_wheel=/tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + [[ /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl != \/\t\f\/\p\k\g\/\t\e\n\s\o\r\f\l\o\w\_\a\a\r\c\h\6\4\-\2\.\1\8\.\0\-\c\p\3\9\-\c\p\3\9\-\l\i\n\u\x\_\a\a\r\c\h\6\4\.\w\h\l ]] + rm /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl + wheel=/tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + TF_WHEEL=/tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + bats /usertools/wheel_verification.bats --timing stderr_lines: <omitted> stdout: |- Checking and renaming /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl... INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl INFO:auditwheel.wheeltools:Previous filename tags: linux_aarch64 INFO:auditwheel.wheeltools:New filename tags: manylinux_2_17_aarch64, manylinux2014_aarch64 INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp39-cp39-linux_aarch64 INFO:auditwheel.wheeltools:New WHEEL info tags: cp39-cp39-manylinux_2_17_aarch64, cp39-cp39-manylinux2014_aarch64 INFO:auditwheel.main_repair: Fixed-up wheel written to /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 1..5 ok 1 Wheel is manylinux2014 (manylinux_2_17) compliant in 68696ms # Size of /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is 263 / 580 megabytes. ok 2 Wheel conforms to upstream size limitations in 19ms ok 3 Wheel is installable in 57389ms ok 4 TensorFlow is importable in 4073ms not ok 5 TensorFlow has Keras in 3756ms # (in test file /usertools/wheel_verification.bats, line 72) # `python3 -c 'import sys; import tensorflow as tf; sys.exit(0 if "keras" in tf.keras.__name__ else 1)'' failed # Traceback (most recent call last): # File "/tf/venv/lib/python3.9/site-packages/tensorflow/python/util/lazy_loader.py", line 147, in _initialize # import keras # pylint: disable=g-import-not-at-top # File "/tf/venv/lib/python3.9/site-packages/keras/__init__.py", line 23, in <module> # from keras.api import _tf_keras # ImportError: cannot import name '_tf_keras' from 'keras.api' (/tf/venv/lib/python3.9/site-packages/keras/api/__init__.py) # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "<string>", line 1, in <module> # File "/tf/venv/lib/python3.9/site-packages/tensorflow/python/util/lazy_loader.py", line 181, in __getattr__ # self._initialize() # File "/tf/venv/lib/python3.9/site-packages/tensorflow/python/util/lazy_loader.py", line 161, in _initialize # raise ImportError( # pylint: disable=raise-missing-from # ImportError: Keras cannot be imported. Check that it is installed. stdout_lines: <omitted> PLAY RECAP ********************************************************************* localhost : ok=39 changed=26 unreachable=0 failed=1 skipped=7 rescued=0 ignored=1 Tuesday 18 June 2024 23:04:17 +0100 (0:04:48.629) 3:54:04.833 ********** =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow git tests ------ 8105.51s tensorflow_in_container : (Python 3.9) build TensorFlow git ---------- 4448.16s tensorflow_in_container : (Python 3.9) start build container --------- 1058.84s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one - 288.63s Cleanup after previous build ------------------------------------------- 93.14s tensorflow : Fetch TensorFlow from git --------------------------------- 22.83s tensorflow : Remove TensorFlow git sources from previous build ---------- 4.22s Gathering Facts --------------------------------------------------------- 1.95s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 1.89s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 1.87s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.58s Create cache dirs ------------------------------------------------------- 1.42s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.21s tensorflow : Save git commit hash to file ------------------------------- 1.15s tensorflow_in_container : (Python 3.9) Copy built wheel ----------------- 1.07s tensorflow_in_container : (Python 3.9) cleanup Bazel install directory --- 0.89s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.84s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.82s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.82s tensorflow_in_container : (Python 3.9) save tests stderr output to file --- 0.70s Build step 'Execute shell' marked build as failure

5 months, 3 weeks

1
11
0 0

Build failed in Jenkins: Tensorflow oneDNN Python package (manylinux2014 nightly) #835

by ci_notify＠linaro.org

See <https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/8…> Changes: ------------------------------------------ [...truncated 1.37 MiB...] Stats over 10 runs: max = 63.3s, min = 32.7s, avg = 39.6s, dev = 8.8s //tensorflow/compiler/tests:fft_test_cpu PASSED in 20.4s Stats over 12 runs: max = 20.4s, min = 7.2s, avg = 14.7s, dev = 4.8s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 16.4s Stats over 12 runs: max = 16.4s, min = 4.1s, avg = 9.5s, dev = 3.9s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 15.8s Stats over 12 runs: max = 15.8s, min = 5.2s, avg = 7.3s, dev = 2.8s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 10.8s Stats over 12 runs: max = 10.8s, min = 4.4s, avg = 6.5s, dev = 1.9s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 23.4s Stats over 12 runs: max = 23.4s, min = 11.4s, avg = 14.4s, dev = 3.1s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 22.7s Stats over 12 runs: max = 22.7s, min = 11.3s, avg = 15.2s, dev = 2.8s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 28.5s Stats over 15 runs: max = 28.5s, min = 12.1s, avg = 17.1s, dev = 4.9s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 176.0s Stats over 15 runs: max = 176.0s, min = 135.5s, avg = 150.3s, dev = 11.0s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 54.7s Stats over 15 runs: max = 54.7s, min = 6.6s, avg = 13.4s, dev = 12.3s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 11.9s Stats over 16 runs: max = 11.9s, min = 4.2s, avg = 7.8s, dev = 2.7s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 35.2s Stats over 16 runs: max = 35.2s, min = 12.0s, avg = 19.6s, dev = 6.0s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 28.1s Stats over 16 runs: max = 28.1s, min = 5.8s, avg = 9.2s, dev = 5.3s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 16.8s Stats over 16 runs: max = 16.8s, min = 4.3s, avg = 7.6s, dev = 3.0s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 17.0s Stats over 16 runs: max = 17.0s, min = 8.4s, avg = 11.5s, dev = 2.1s //tensorflow/python/ops:image_ops_test_cpu PASSED in 26.9s Stats over 16 runs: max = 26.9s, min = 8.6s, avg = 15.0s, dev = 4.5s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 57.9s Stats over 17 runs: max = 57.9s, min = 8.3s, avg = 25.3s, dev = 15.6s //tensorflow/python/data/kernel_tests:map_test PASSED in 76.9s Stats over 19 runs: max = 76.9s, min = 27.9s, avg = 40.2s, dev = 10.5s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 13.9s Stats over 20 runs: max = 13.9s, min = 4.2s, avg = 5.6s, dev = 2.0s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 12.4s Stats over 20 runs: max = 12.4s, min = 4.3s, avg = 5.9s, dev = 1.9s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 11.3s Stats over 20 runs: max = 11.3s, min = 4.1s, avg = 5.7s, dev = 1.8s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 12.2s Stats over 20 runs: max = 12.2s, min = 4.4s, avg = 5.9s, dev = 2.1s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 12.0s Stats over 20 runs: max = 12.0s, min = 5.1s, avg = 6.0s, dev = 1.4s //tensorflow/dtensor/python/tests:rng_test_cpu PASSED in 14.2s Stats over 20 runs: max = 14.2s, min = 6.5s, avg = 7.9s, dev = 1.6s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 29.1s Stats over 20 runs: max = 29.1s, min = 14.0s, avg = 16.8s, dev = 3.1s //tensorflow/python/kernel_tests:metrics_test PASSED in 44.4s Stats over 20 runs: max = 44.4s, min = 7.9s, avg = 20.0s, dev = 10.2s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 10.5s Stats over 20 runs: max = 10.5s, min = 3.8s, avg = 4.8s, dev = 1.3s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 12.9s Stats over 20 runs: max = 12.9s, min = 4.1s, avg = 6.3s, dev = 2.9s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 56.6s Stats over 20 runs: max = 56.6s, min = 5.1s, avg = 17.5s, dev = 15.9s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 136.5s Stats over 20 runs: max = 136.5s, min = 42.0s, avg = 76.5s, dev = 27.9s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 14.0s Stats over 20 runs: max = 14.0s, min = 6.4s, avg = 9.1s, dev = 2.2s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 19.2s Stats over 20 runs: max = 19.2s, min = 8.1s, avg = 12.7s, dev = 3.2s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 153.2s Stats over 20 runs: max = 153.2s, min = 42.6s, avg = 82.8s, dev = 32.0s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 25.5s Stats over 20 runs: max = 25.5s, min = 5.0s, avg = 12.1s, dev = 6.5s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 23.1s Stats over 20 runs: max = 23.1s, min = 8.7s, avg = 14.3s, dev = 3.7s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 34.8s Stats over 20 runs: max = 34.8s, min = 21.2s, avg = 26.1s, dev = 3.7s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 81.5s Stats over 20 runs: max = 81.5s, min = 10.0s, avg = 38.1s, dev = 24.7s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 42.4s Stats over 20 runs: max = 42.4s, min = 11.8s, avg = 19.0s, dev = 7.2s //tensorflow/compiler/tests:unary_ops_test_cpu FAILED in 3 out of 22 in 25.0s Stats over 22 runs: max = 25.0s, min = 4.2s, avg = 10.4s, dev = 6.5s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu/shard_18_of_20/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu/shard_18_of_20/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu/shard_18_of_20/test_attempts/attempt_2.log //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test FAILED in 3 out of 22 in 27.0s Stats over 22 runs: max = 27.0s, min = 3.8s, avg = 11.0s, dev = 6.5s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu_mlir_bridge_test/shard_18_of_20/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu_mlir_bridge_test/shard_18_of_20/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu_mlir_bridge_test/shard_18_of_20/test_attempts/attempt_2.log //tensorflow/python/data/kernel_tests:interleave_test PASSED in 31.1s Stats over 24 runs: max = 31.1s, min = 9.2s, avg = 17.2s, dev = 6.3s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 20.4s Stats over 24 runs: max = 20.4s, min = 4.6s, avg = 9.8s, dev = 4.8s //tensorflow/dtensor/python/tests:multi_device_spmd_test_cpu PASSED in 31.1s Stats over 25 runs: max = 31.1s, min = 23.1s, avg = 27.2s, dev = 2.0s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 43.9s Stats over 30 runs: max = 43.9s, min = 4.4s, avg = 15.1s, dev = 10.3s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 17.4s Stats over 32 runs: max = 17.4s, min = 4.4s, avg = 8.7s, dev = 3.3s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 19.6s Stats over 32 runs: max = 19.6s, min = 4.2s, avg = 9.9s, dev = 3.9s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 215.6s Stats over 35 runs: max = 215.6s, min = 4.5s, avg = 22.8s, dev = 37.1s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 218.8s Stats over 35 runs: max = 218.8s, min = 4.3s, avg = 23.7s, dev = 37.5s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.7s Stats over 48 runs: max = 1.7s, min = 0.4s, avg = 0.5s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 63.3s Stats over 50 runs: max = 63.3s, min = 31.5s, avg = 43.2s, dev = 8.0s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 21.2s Stats over 50 runs: max = 21.2s, min = 3.9s, avg = 11.7s, dev = 4.5s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 18.3s Stats over 50 runs: max = 18.3s, min = 4.2s, avg = 10.9s, dev = 4.0s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 83.4s Stats over 50 runs: max = 83.4s, min = 48.2s, avg = 60.8s, dev = 7.9s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 21.4s Stats over 50 runs: max = 21.4s, min = 6.2s, avg = 11.4s, dev = 3.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_onednn_grad_test PASSED in 25.8s Stats over 50 runs: max = 25.8s, min = 5.5s, avg = 11.9s, dev = 4.2s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 39.1s Stats over 50 runs: max = 39.1s, min = 10.5s, avg = 21.3s, dev = 7.2s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 12.3s Stats over 50 runs: max = 12.3s, min = 4.3s, avg = 6.3s, dev = 1.6s Executed 3086 out of 3086 tests: 3084 tests pass and 2 fail locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. stdout_lines: <omitted> ...ignoring TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages] *** Tuesday 18 June 2024 04:41:58 +0100 (2:05:25.924) 3:31:53.188 ********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save list of installed Python packages] *** Tuesday 18 June 2024 04:42:00 +0100 (0:00:02.409) 3:31:55.597 ********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save tests output to file] ******** Tuesday 18 June 2024 04:42:01 +0100 (0:00:00.691) 3:31:56.288 ********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save tests stderr output to file] *** Tuesday 18 June 2024 04:42:02 +0100 (0:00:00.699) 3:31:56.988 ********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) Extract number of tests run] ****** Tuesday 18 June 2024 04:42:02 +0100 (0:00:00.704) 3:31:57.692 ********** ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) Show number of tests run] ********* Tuesday 18 June 2024 04:42:03 +0100 (0:00:00.080) 3:31:57.772 ********** ok: [localhost] => ut_tests_line: 'Executed 3086 out of 3086 tests: 3084 tests pass and 2 fail locally.' TASK [tensorflow_in_container : (Python 3.9) Extract failed tests] ************* Tuesday 18 June 2024 04:42:03 +0100 (0:00:00.040) 3:31:57.813 ********** ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) Show which tests are flaky] ******* Tuesday 18 June 2024 04:42:03 +0100 (0:00:00.156) 3:31:57.969 ********** ok: [localhost] => tests_flaky: [] TASK [tensorflow_in_container : (Python 3.9) Show which tests failed] ********** Tuesday 18 June 2024 04:42:03 +0100 (0:00:00.039) 3:31:58.009 ********** ok: [localhost] => tests_failed: - 'FAILED: //tensorflow/compiler/tests:unary_ops_test_cpu (Summary)' - 'FAILED: //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test (Summary)' TASK [tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one] *** Tuesday 18 June 2024 04:42:03 +0100 (0:00:00.040) 3:31:58.050 ********** fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- + for wheel in /tf/pkg/*.whl + echo 'Checking and renaming /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl...' + python3 -m auditwheel repair --plat manylinux2014_aarch64 /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl --wheel-dir /tf/pkg + tee check.txt real 2m38.362s user 2m11.527s sys 0m20.972s ++ grep --extended-regexp --only-matching '/tf/pkg/\S+.whl' check.txt + new_wheel=/tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + [[ /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl != \/\t\f\/\p\k\g\/\t\e\n\s\o\r\f\l\o\w\_\a\a\r\c\h\6\4\-\2\.\1\8\.\0\-\c\p\3\9\-\c\p\3\9\-\l\i\n\u\x\_\a\a\r\c\h\6\4\.\w\h\l ]] + rm /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl + wheel=/tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + TF_WHEEL=/tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + bats /usertools/wheel_verification.bats --timing stderr_lines: <omitted> stdout: |- Checking and renaming /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl... INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl INFO:auditwheel.wheeltools:Previous filename tags: linux_aarch64 INFO:auditwheel.wheeltools:New filename tags: manylinux_2_17_aarch64, manylinux2014_aarch64 INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp39-cp39-linux_aarch64 INFO:auditwheel.wheeltools:New WHEEL info tags: cp39-cp39-manylinux_2_17_aarch64, cp39-cp39-manylinux2014_aarch64 INFO:auditwheel.main_repair: Fixed-up wheel written to /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 1..5 ok 1 Wheel is manylinux2014 (manylinux_2_17) compliant in 75030ms # Size of /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is 209 / 580 megabytes. ok 2 Wheel conforms to upstream size limitations in 18ms ok 3 Wheel is installable in 58309ms ok 4 TensorFlow is importable in 4345ms not ok 5 TensorFlow has Keras in 3732ms # (in test file /usertools/wheel_verification.bats, line 72) # `python3 -c 'import sys; import tensorflow as tf; sys.exit(0 if "keras" in tf.keras.__name__ else 1)'' failed # Traceback (most recent call last): # File "/tf/venv/lib/python3.9/site-packages/tensorflow/python/util/lazy_loader.py", line 147, in _initialize # import keras # pylint: disable=g-import-not-at-top # File "/tf/venv/lib/python3.9/site-packages/keras/__init__.py", line 23, in <module> # from keras.api import _tf_keras # ImportError: cannot import name '_tf_keras' from 'keras.api' (/tf/venv/lib/python3.9/site-packages/keras/api/__init__.py) # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "<string>", line 1, in <module> # File "/tf/venv/lib/python3.9/site-packages/tensorflow/python/util/lazy_loader.py", line 181, in __getattr__ # self._initialize() # File "/tf/venv/lib/python3.9/site-packages/tensorflow/python/util/lazy_loader.py", line 161, in _initialize # raise ImportError( # pylint: disable=raise-missing-from # ImportError: Keras cannot be imported. Check that it is installed. stdout_lines: <omitted> PLAY RECAP ********************************************************************* localhost : ok=39 changed=26 unreachable=0 failed=1 skipped=7 rescued=0 ignored=1 Tuesday 18 June 2024 04:47:05 +0100 (0:05:02.388) 3:37:00.438 ********** =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow-oneDNN-ACL git tests 7525.92s tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git 5080.08s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one - 302.39s Cleanup after previous build ------------------------------------------- 49.44s tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 23.05s tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 5.28s tensorflow_in_container : (Python 3.9) start build container ------------ 4.74s tensorflow_in_container : (Python 3.9) cleanup Bazel install directory --- 3.00s Gathering Facts --------------------------------------------------------- 2.72s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 2.53s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 2.41s tensorflow_in_container : (Python 3.9) Copy built wheel ----------------- 2.23s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.70s Create cache dirs ------------------------------------------------------- 1.44s tensorflow_in_container : (Python 3.9) create script to run tests ------- 1.37s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.22s tensorflow : Save git commit hash to file ------------------------------- 1.15s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.85s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.83s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.80s Build step 'Execute shell' marked build as failure

6 months

1
3
0 0

Build failed in Jenkins: Tensorflow oneDNN Python package (manylinux2014 nightly) #833

by ci_notify＠linaro.org

See <https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/8…> Changes: ------------------------------------------ [...truncated 8.40 KiB...] Sunday 16 June 2024 01:11:38 +0100 (0:00:00.079) 0:01:33.266 *********** changed: [localhost] TASK [tensorflow : Use nightly package requirements] *************************** Sunday 16 June 2024 01:11:39 +0100 (0:00:00.691) 0:01:33.957 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) copy upper constraints for hermetic builds] *** Sunday 16 June 2024 01:11:39 +0100 (0:00:00.645) 0:01:34.603 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) build Python versioned container] *** Sunday 16 June 2024 01:11:40 +0100 (0:00:00.687) 0:01:35.291 *********** ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) start build container] ************ Sunday 16 June 2024 01:11:42 +0100 (0:00:01.734) 0:01:37.026 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir] *** Sunday 16 June 2024 01:11:47 +0100 (0:00:05.454) 0:01:42.481 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for bazel cache] ****** Sunday 16 June 2024 01:11:49 +0100 (0:00:01.235) 0:01:43.716 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for bazelisk cache] *** Sunday 16 June 2024 01:11:49 +0100 (0:00:00.899) 0:01:44.616 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for pip cache] ******** Sunday 16 June 2024 01:11:50 +0100 (0:00:00.861) 0:01:45.478 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build] *** Sunday 16 June 2024 01:11:51 +0100 (0:00:00.857) 0:01:46.336 *********** changed: [localhost] TASK [tensorflow_in_container : Show list of installed Python packages for build] *** Sunday 16 June 2024 01:11:54 +0100 (0:00:03.279) 0:01:49.615 *********** ok: [localhost] => pip_list_output.stdout: |- Package Version Location Installer ------------------- -------------------- -------------------------------------- --------- auditwheel 5.4.0 /usr/local/lib/python3.9/dist-packages pip certifi 2019.11.28 /usr/lib/python3/dist-packages cffi 1.16.0 /usr/local/lib/python3.9/dist-packages pip chardet 3.0.4 /usr/lib/python3/dist-packages cryptography 42.0.5 /usr/local/lib/python3.9/dist-packages pip Cython 3.0.9 /usr/local/lib/python3.9/dist-packages pip dbus-python 1.2.16 /usr/lib/python3/dist-packages docutils 0.20.1 /usr/local/lib/python3.9/dist-packages pip idna 2.8 /usr/lib/python3/dist-packages importlib-metadata 7.0.1 /usr/local/lib/python3.9/dist-packages pip jaraco.classes 3.3.1 /usr/local/lib/python3.9/dist-packages pip jeepney 0.8.0 /usr/local/lib/python3.9/dist-packages pip keyring 24.3.1 /usr/local/lib/python3.9/dist-packages pip markdown-it-py 3.0.0 /usr/local/lib/python3.9/dist-packages pip mdurl 0.1.2 /usr/local/lib/python3.9/dist-packages pip more-itertools 10.2.0 /usr/local/lib/python3.9/dist-packages pip nh3 0.2.15 /usr/local/lib/python3.9/dist-packages pip pip 24.0 /usr/local/lib/python3.9/dist-packages pip pkginfo 1.10.0 /usr/local/lib/python3.9/dist-packages pip portpicker 1.6.0 /usr/local/lib/python3.9/dist-packages pip psutil 5.9.8 /usr/local/lib/python3.9/dist-packages pip pycparser 2.21 /usr/local/lib/python3.9/dist-packages pip pyelftools 0.30 /usr/local/lib/python3.9/dist-packages pip Pygments 2.17.2 /usr/local/lib/python3.9/dist-packages pip PyGObject 3.36.0 /usr/lib/python3/dist-packages python-apt 2.0.1+ubuntu0.20.4.1 /usr/lib/python3/dist-packages readme_renderer 43.0 /usr/local/lib/python3.9/dist-packages pip requests 2.22.0 /usr/lib/python3/dist-packages requests-toolbelt 1.0.0 /usr/local/lib/python3.9/dist-packages pip requests-unixsocket 0.2.0 /usr/lib/python3/dist-packages rfc3986 2.0.0 /usr/local/lib/python3.9/dist-packages pip rich 13.7.1 /usr/local/lib/python3.9/dist-packages pip SecretStorage 3.3.3 /usr/local/lib/python3.9/dist-packages pip setuptools 69.1.1 /usr/local/lib/python3.9/dist-packages pip six 1.14.0 /usr/lib/python3/dist-packages twine 4.0.2 /usr/local/lib/python3.9/dist-packages pip urllib3 2.2.1 /usr/local/lib/python3.9/dist-packages pip wheel 0.42.0 /usr/local/lib/python3.9/dist-packages pip zipp 3.17.0 /usr/local/lib/python3.9/dist-packages pip TASK [tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git] *** Sunday 16 June 2024 01:11:54 +0100 (0:00:00.037) 0:01:49.653 *********** fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- Extracting Bazel installation... Starting local Bazel server and connecting to it... INFO: Invocation ID: d8965d29-38fd-466e-a52f-f1776b7bd719 INFO: Reading 'startup' options from /tf/tensorflow/.bazelrc: --windows_enable_symlinks INFO: Options provided by the client: Inherited 'common' options: --isatty=0 --terminal_columns=80 INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc: Inherited 'common' options: --experimental_repo_remote_exec INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc: 'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility INFO: Reading rc options for 'build' from /usertools/aarch64_clang.bazelrc: 'build' options: --action_env=CACHEBUSTER=20220325 --action_env PYTHON_BIN_PATH=/usr/local/bin/python3 --python_path=/usr/local/bin/python3 --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 --linkopt=-fuse-ld=lld --linkopt=-lm --linkopt=-Wl,--undefined-version --host_linkopt=-fuse-ld=lld --host_linkopt=-lm --host_linkopt=-Wl,--undefined-version --distinct_host_configuration=false --copt=-Wno-gnu-offsetof-extensions --profile=/tf/pkg/profile.json.gz --crosstool_top=@ml2014_clang_aarch64_config_aarch64//crosstool:toolchain --copt=-mtune=generic --copt=-march=armv8-a --copt=-O3 INFO: Found applicable config definition build:short_logs in file /tf/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING INFO: Found applicable config definition build:v2 in file /tf/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 INFO: Found applicable config definition build:sigbuild_local_cache in file /usertools/aarch64_clang.bazelrc: --disk_cache=/tf/cache INFO: Found applicable config definition build:mkl_aarch64_threadpool in file /tf/tensorflow/.bazelrc: --define=build_with_mkl_aarch64=true -c opt INFO: Found applicable config definition build:linux in file /tf/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes INFO: Found applicable config definition build:dynamic_kernels in file /tf/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: INFO: Repository llvm-raw instantiated at: /tf/tensorflow/WORKSPACE:91:14: in <toplevel> /tf/tensorflow/tensorflow/workspace3.bzl:63:9: in workspace /tf/tensorflow/third_party/llvm/workspace.bzl:10:20: in repo /tf/tensorflow/third_party/repo.bzl:136:21: in tf_http_archive Repository rule _tf_http_archive defined at: /tf/tensorflow/third_party/repo.bzl:89:35: in <toplevel> WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-p… failed: class java.io.FileNotFoundException GET returned 404 Not Found ERROR: An error occurred during the fetch of repository 'llvm-raw': Traceback (most recent call last): File "/tf/tensorflow/third_party/repo.bzl", line 83, column 30, in _tf_http_archive_impl ctx.patch(patch_file, strip = 1) Error in patch: Error applying patch /tf/tensorflow/third_party/llvm/typo.patch: Incorrect Chunk: the chunk content doesn't match the target **Original Position**: 1051 **Original Content**: ":CAPIDebugObjects", ":CAPIIRObjects", ":CAPIInterfacesObjects", ":CAPITransformObjects", ], ) **Revised Content**: ":CAPIDebugObjects", ":CAPIIRObjects", ":CAPIInterfacesObjects", ":CAPITransformsObjects", ], ) ERROR: /tf/tensorflow/WORKSPACE:91:14: fetching _tf_http_archive rule //external:llvm-raw: Traceback (most recent call last): File "/tf/tensorflow/third_party/repo.bzl", line 83, column 30, in _tf_http_archive_impl ctx.patch(patch_file, strip = 1) Error in patch: Error applying patch /tf/tensorflow/third_party/llvm/typo.patch: Incorrect Chunk: the chunk content doesn't match the target **Original Position**: 1051 **Original Content**: ":CAPIDebugObjects", ":CAPIIRObjects", ":CAPIInterfacesObjects", ":CAPITransformObjects", ], ) **Revised Content**: ":CAPIDebugObjects", ":CAPIIRObjects", ":CAPIInterfacesObjects", ":CAPITransformsObjects", ], ) ERROR: Error computing the main repository mapping: no such package '@llvm-raw//utils/bazel': Error applying patch /tf/tensorflow/third_party/llvm/typo.patch: Incorrect Chunk: the chunk content doesn't match the target **Original Position**: 1051 **Original Content**: ":CAPIDebugObjects", ":CAPIIRObjects", ":CAPIInterfacesObjects", ":CAPITransformObjects", ], ) **Revised Content**: ":CAPIDebugObjects", ":CAPIIRObjects", ":CAPIInterfacesObjects", ":CAPITransformsObjects", ], ) stderr_lines: <omitted> stdout: '' stdout_lines: <omitted> PLAY RECAP ********************************************************************* localhost : ok=25 changed=17 unreachable=0 failed=1 skipped=6 rescued=0 ignored=0 Sunday 16 June 2024 01:13:21 +0100 (0:01:26.392) 0:03:16.045 *********** =============================================================================== tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git -- 86.39s Cleanup after previous build ------------------------------------------- 57.36s tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 22.00s tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 6.88s tensorflow_in_container : (Python 3.9) start build container ------------ 5.45s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 3.28s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.73s Gathering Facts --------------------------------------------------------- 1.70s Create cache dirs ------------------------------------------------------- 1.45s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.24s tensorflow : Save git commit hash to file ------------------------------- 1.14s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.90s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.86s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.86s tensorflow : Create linklibs shell script ------------------------------- 0.69s tensorflow_in_container : (Python 3.9) copy upper constraints for hermetic builds --- 0.69s tensorflow_in_container : Reset checkout to clean state ----------------- 0.66s tensorflow : Use nightly package requirements --------------------------- 0.65s tensorflow : Get oneline version of git log for this checkout ----------- 0.58s Build TensorFlow for each Python version (dual abi) --------------------- 0.43s Build step 'Execute shell' marked build as failure

6 months, 1 week

1
1
0 0

Build failed in Jenkins: Tensorflow oneDNN Python package (manylinux2014 nightly) #818

by ci_notify＠linaro.org

See <https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/8…> Changes: ------------------------------------------ [...truncated 7.02 KiB...] TASK [tensorflow : Record git commit hash to variable] ************************* Saturday 01 June 2024 01:11:14 +0100 (0:00:00.036) 0:01:10.344 ********* ok: [localhost] TASK [tensorflow : Save git commit hash to file] ******************************* Saturday 01 June 2024 01:11:14 +0100 (0:00:00.058) 0:01:10.403 ********* changed: [localhost] TASK [Build TensorFlow for each Python version (dual abi)] ********************* Saturday 01 June 2024 01:11:15 +0100 (0:00:01.103) 0:01:11.506 ********* TASK [tensorflow_in_container : Reset checkout to clean state] ***************** Saturday 01 June 2024 01:11:16 +0100 (0:00:00.411) 0:01:11.917 ********* changed: [localhost] TASK [tensorflow_in_container : Remove TensorFlow-oneDNN-ACL git sources from previous build] *** Saturday 01 June 2024 01:11:16 +0100 (0:00:00.690) 0:01:12.608 ********* skipping: [localhost] TASK [Fetch TensorFlow] ******************************************************** Saturday 01 June 2024 01:11:16 +0100 (0:00:00.029) 0:01:12.638 ********* skipping: [localhost] TASK [(Python 3.9) patch TensorFlow tree] ************************************** Saturday 01 June 2024 01:11:17 +0100 (0:00:00.030) 0:01:12.668 ********* TASK [tensorflow : Create linklibs shell script] ******************************* Saturday 01 June 2024 01:11:17 +0100 (0:00:00.082) 0:01:12.751 ********* changed: [localhost] TASK [tensorflow : Use nightly package requirements] *************************** Saturday 01 June 2024 01:11:17 +0100 (0:00:00.695) 0:01:13.446 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) copy upper constraints for hermetic builds] *** Saturday 01 June 2024 01:11:18 +0100 (0:00:00.638) 0:01:14.084 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) build Python versioned container] *** Saturday 01 June 2024 01:11:19 +0100 (0:00:00.688) 0:01:14.773 ********* ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) start build container] ************ Saturday 01 June 2024 01:11:20 +0100 (0:00:01.629) 0:01:16.403 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir] *** Saturday 01 June 2024 01:11:22 +0100 (0:00:02.163) 0:01:18.567 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for bazel cache] ****** Saturday 01 June 2024 01:11:24 +0100 (0:00:01.345) 0:01:19.912 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for bazelisk cache] *** Saturday 01 June 2024 01:11:25 +0100 (0:00:00.905) 0:01:20.818 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for pip cache] ******** Saturday 01 June 2024 01:11:26 +0100 (0:00:00.843) 0:01:21.661 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build] *** Saturday 01 June 2024 01:11:26 +0100 (0:00:00.865) 0:01:22.526 ********* changed: [localhost] TASK [tensorflow_in_container : Show list of installed Python packages for build] *** Saturday 01 June 2024 01:11:29 +0100 (0:00:02.342) 0:01:24.869 ********* ok: [localhost] => pip_list_output.stdout: |- Package Version Location Installer ------------------- -------------------- -------------------------------------- --------- auditwheel 5.4.0 /usr/local/lib/python3.9/dist-packages pip certifi 2019.11.28 /usr/lib/python3/dist-packages cffi 1.16.0 /usr/local/lib/python3.9/dist-packages pip chardet 3.0.4 /usr/lib/python3/dist-packages cryptography 42.0.5 /usr/local/lib/python3.9/dist-packages pip Cython 3.0.9 /usr/local/lib/python3.9/dist-packages pip dbus-python 1.2.16 /usr/lib/python3/dist-packages docutils 0.20.1 /usr/local/lib/python3.9/dist-packages pip idna 2.8 /usr/lib/python3/dist-packages importlib-metadata 7.0.1 /usr/local/lib/python3.9/dist-packages pip jaraco.classes 3.3.1 /usr/local/lib/python3.9/dist-packages pip jeepney 0.8.0 /usr/local/lib/python3.9/dist-packages pip keyring 24.3.1 /usr/local/lib/python3.9/dist-packages pip markdown-it-py 3.0.0 /usr/local/lib/python3.9/dist-packages pip mdurl 0.1.2 /usr/local/lib/python3.9/dist-packages pip more-itertools 10.2.0 /usr/local/lib/python3.9/dist-packages pip nh3 0.2.15 /usr/local/lib/python3.9/dist-packages pip pip 24.0 /usr/local/lib/python3.9/dist-packages pip pkginfo 1.10.0 /usr/local/lib/python3.9/dist-packages pip portpicker 1.6.0 /usr/local/lib/python3.9/dist-packages pip psutil 5.9.8 /usr/local/lib/python3.9/dist-packages pip pycparser 2.21 /usr/local/lib/python3.9/dist-packages pip pyelftools 0.30 /usr/local/lib/python3.9/dist-packages pip Pygments 2.17.2 /usr/local/lib/python3.9/dist-packages pip PyGObject 3.36.0 /usr/lib/python3/dist-packages python-apt 2.0.1+ubuntu0.20.4.1 /usr/lib/python3/dist-packages readme_renderer 43.0 /usr/local/lib/python3.9/dist-packages pip requests 2.22.0 /usr/lib/python3/dist-packages requests-toolbelt 1.0.0 /usr/local/lib/python3.9/dist-packages pip requests-unixsocket 0.2.0 /usr/lib/python3/dist-packages rfc3986 2.0.0 /usr/local/lib/python3.9/dist-packages pip rich 13.7.1 /usr/local/lib/python3.9/dist-packages pip SecretStorage 3.3.3 /usr/local/lib/python3.9/dist-packages pip setuptools 69.1.1 /usr/local/lib/python3.9/dist-packages pip six 1.14.0 /usr/lib/python3/dist-packages twine 4.0.2 /usr/local/lib/python3.9/dist-packages pip urllib3 2.2.1 /usr/local/lib/python3.9/dist-packages pip wheel 0.42.0 /usr/local/lib/python3.9/dist-packages pip zipp 3.17.0 /usr/local/lib/python3.9/dist-packages pip TASK [tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git] *** Saturday 01 June 2024 01:11:29 +0100 (0:00:00.037) 0:01:24.907 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) build Python package] ************* Saturday 01 June 2024 02:20:32 +0100 (1:09:02.754) 1:10:27.662 ********* skipping: [localhost] TASK [tensorflow_in_container : (Python 3.9) Copy built wheel] ***************** Saturday 01 June 2024 02:20:32 +0100 (0:00:00.040) 1:10:27.702 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) cleanup Bazel install directory] *** Saturday 01 June 2024 02:20:33 +0100 (0:00:01.579) 1:10:29.282 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create script to run tests] ******* Saturday 01 June 2024 02:20:36 +0100 (0:00:02.703) 1:10:31.985 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) run TensorFlow-oneDNN-ACL git tests] *** Saturday 01 June 2024 02:20:37 +0100 (0:00:00.966) 1:10:32.952 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages] *** Saturday 01 June 2024 04:41:53 +0100 (2:21:16.641) 3:31:49.593 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save list of installed Python packages] *** Saturday 01 June 2024 04:41:55 +0100 (0:00:01.870) 3:31:51.463 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save tests output to file] ******** Saturday 01 June 2024 04:41:56 +0100 (0:00:00.690) 3:31:52.154 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save tests stderr output to file] *** Saturday 01 June 2024 04:41:57 +0100 (0:00:00.698) 3:31:52.853 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) Extract number of tests run] ****** Saturday 01 June 2024 04:41:57 +0100 (0:00:00.704) 3:31:53.557 ********* ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) Show number of tests run] ********* Saturday 01 June 2024 04:41:57 +0100 (0:00:00.076) 3:31:53.634 ********* ok: [localhost] => ut_tests_line: 'Executed 3086 out of 3086 tests: 3086 tests pass.' TASK [tensorflow_in_container : (Python 3.9) Extract failed tests] ************* Saturday 01 June 2024 04:41:58 +0100 (0:00:00.039) 3:31:53.674 ********* ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) Show which tests are flaky] ******* Saturday 01 June 2024 04:41:58 +0100 (0:00:00.127) 3:31:53.802 ********* ok: [localhost] => tests_flaky: - //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test FLAKY, failed in 1 out of 2 in 0.3s TASK [tensorflow_in_container : (Python 3.9) Show which tests failed] ********** Saturday 01 June 2024 04:41:58 +0100 (0:00:00.039) 3:31:53.842 ********* ok: [localhost] => tests_failed: [] TASK [tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one] *** Saturday 01 June 2024 04:41:58 +0100 (0:00:00.039) 3:31:53.881 ********* fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- + for wheel in /tf/pkg/*.whl + echo 'Checking and renaming /tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl...' + python3 -m auditwheel repair --plat manylinux2014_aarch64 /tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl --wheel-dir /tf/pkg + tee check.txt real 1m27.249s user 1m11.201s sys 0m15.533s stderr_lines: <omitted> stdout: |- Checking and renaming /tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl... INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/auditwheel/repair.py", line 80, in repair_wheel raise ValueError( ValueError: Cannot repair wheel, because required library "_pywrap_tensorflow_internal.so" could not be located During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.9/dist-packages/auditwheel/__main__.py", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.9/dist-packages/auditwheel/main.py", line 59, in main rval = args.func(args, p) File "/usr/local/lib/python3.9/dist-packages/auditwheel/main_repair.py", line 175, in execute out_wheel = repair_wheel( File "/usr/local/lib/python3.9/dist-packages/auditwheel/repair.py", line 118, in repair_wheel strip_symbols(itertools.chain(libs_to_strip, extensions)) File "/usr/local/lib/python3.9/dist-packages/auditwheel/wheeltools.py", line 137, in __exit__ dir2zip(self.name, self.out_wheel, date_time) File "/usr/local/lib/python3.9/dist-packages/auditwheel/tools.py", line 76, in dir2zip with zipfile.ZipFile(zip_fname, "w", compression=compression) as z: File "/usr/lib/python3.9/zipfile.py", line 1248, in __init__ self.fp = io.open(file, filemode) PermissionError: [Errno 13] Permission denied: '/tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl' stdout_lines: <omitted> PLAY RECAP ********************************************************************* localhost : ok=39 changed=26 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0 Saturday 01 June 2024 04:43:26 +0100 (0:01:28.052) 3:33:21.934 ********* =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow-oneDNN-ACL git tests 8476.64s tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git 4142.75s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one -- 88.05s Cleanup after previous build ------------------------------------------- 38.04s tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 22.84s tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 4.94s tensorflow_in_container : (Python 3.9) cleanup Bazel install directory --- 2.70s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 2.34s tensorflow_in_container : (Python 3.9) start build container ------------ 2.16s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 1.87s Gathering Facts --------------------------------------------------------- 1.69s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.63s tensorflow_in_container : (Python 3.9) Copy built wheel ----------------- 1.58s Create cache dirs ------------------------------------------------------- 1.43s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.35s tensorflow : Save git commit hash to file ------------------------------- 1.10s tensorflow_in_container : (Python 3.9) create script to run tests ------- 0.97s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.91s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.87s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.84s Build step 'Execute shell' marked build as failure

6 months, 2 weeks

1
3
0 0

Build failed in Jenkins: Tensorflow Python package (manylinux2014 nightly) #1188

by ci_notify＠linaro.org

See <https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/1188/dis…> Changes: ------------------------------------------ [...truncated 6.63 KiB...] git_log.stdout: f2535930fb518a3996cac26d59c3dd2bf9776e77 [XLA:GPU] Clang-tidy cleanup for xla/service/gpu/fusions/mlir/mlir_fusion_emitter.h TASK [tensorflow : Record git commit hash to variable] ************************* Saturday 01 June 2024 19:11:14 +0100 (0:00:00.037) 0:01:09.722 ********* ok: [localhost] TASK [tensorflow : Save git commit hash to file] ******************************* Saturday 01 June 2024 19:11:14 +0100 (0:00:00.055) 0:01:09.778 ********* changed: [localhost] TASK [Build TensorFlow for each Python version (dual abi)] ********************* Saturday 01 June 2024 19:11:15 +0100 (0:00:01.137) 0:01:10.915 ********* TASK [tensorflow_in_container : Reset checkout to clean state] ***************** Saturday 01 June 2024 19:11:15 +0100 (0:00:00.412) 0:01:11.328 ********* changed: [localhost] TASK [tensorflow_in_container : Remove TensorFlow git sources from previous build] *** Saturday 01 June 2024 19:11:16 +0100 (0:00:00.646) 0:01:11.974 ********* skipping: [localhost] TASK [Fetch TensorFlow] ******************************************************** Saturday 01 June 2024 19:11:16 +0100 (0:00:00.029) 0:01:12.004 ********* skipping: [localhost] TASK [(Python 3.9) patch TensorFlow tree] ************************************** Saturday 01 June 2024 19:11:16 +0100 (0:00:00.030) 0:01:12.034 ********* TASK [tensorflow : Create linklibs shell script] ******************************* Saturday 01 June 2024 19:11:16 +0100 (0:00:00.081) 0:01:12.116 ********* changed: [localhost] TASK [tensorflow : Use nightly package requirements] *************************** Saturday 01 June 2024 19:11:17 +0100 (0:00:00.685) 0:01:12.802 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) copy upper constraints for hermetic builds] *** Saturday 01 June 2024 19:11:17 +0100 (0:00:00.626) 0:01:13.429 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) build Python versioned container] *** Saturday 01 June 2024 19:11:18 +0100 (0:00:00.680) 0:01:14.109 ********* ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) start build container] ************ Saturday 01 June 2024 19:11:19 +0100 (0:00:01.486) 0:01:15.596 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir] *** Saturday 01 June 2024 19:27:00 +0100 (0:15:40.311) 0:16:55.907 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for bazel cache] ****** Saturday 01 June 2024 19:27:01 +0100 (0:00:01.205) 0:16:57.113 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for bazelisk cache] *** Saturday 01 June 2024 19:27:02 +0100 (0:00:00.823) 0:16:57.936 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for pip cache] ******** Saturday 01 June 2024 19:27:03 +0100 (0:00:00.866) 0:16:58.803 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build] *** Saturday 01 June 2024 19:27:03 +0100 (0:00:00.800) 0:16:59.603 ********* changed: [localhost] TASK [tensorflow_in_container : Show list of installed Python packages for build] *** Saturday 01 June 2024 19:27:05 +0100 (0:00:01.857) 0:17:01.461 ********* ok: [localhost] => pip_list_output.stdout: |- Package Version Location Installer ------------------- -------------------- -------------------------------------- --------- auditwheel 5.4.0 /usr/local/lib/python3.9/dist-packages pip certifi 2019.11.28 /usr/lib/python3/dist-packages cffi 1.16.0 /usr/local/lib/python3.9/dist-packages pip chardet 3.0.4 /usr/lib/python3/dist-packages cryptography 42.0.5 /usr/local/lib/python3.9/dist-packages pip Cython 3.0.9 /usr/local/lib/python3.9/dist-packages pip dbus-python 1.2.16 /usr/lib/python3/dist-packages docutils 0.20.1 /usr/local/lib/python3.9/dist-packages pip idna 2.8 /usr/lib/python3/dist-packages importlib-metadata 7.0.1 /usr/local/lib/python3.9/dist-packages pip jaraco.classes 3.3.1 /usr/local/lib/python3.9/dist-packages pip jeepney 0.8.0 /usr/local/lib/python3.9/dist-packages pip keyring 24.3.1 /usr/local/lib/python3.9/dist-packages pip markdown-it-py 3.0.0 /usr/local/lib/python3.9/dist-packages pip mdurl 0.1.2 /usr/local/lib/python3.9/dist-packages pip more-itertools 10.2.0 /usr/local/lib/python3.9/dist-packages pip nh3 0.2.15 /usr/local/lib/python3.9/dist-packages pip pip 24.0 /usr/local/lib/python3.9/dist-packages pip pkginfo 1.10.0 /usr/local/lib/python3.9/dist-packages pip portpicker 1.6.0 /usr/local/lib/python3.9/dist-packages pip psutil 5.9.8 /usr/local/lib/python3.9/dist-packages pip pycparser 2.21 /usr/local/lib/python3.9/dist-packages pip pyelftools 0.30 /usr/local/lib/python3.9/dist-packages pip Pygments 2.17.2 /usr/local/lib/python3.9/dist-packages pip PyGObject 3.36.0 /usr/lib/python3/dist-packages python-apt 2.0.1+ubuntu0.20.4.1 /usr/lib/python3/dist-packages readme_renderer 43.0 /usr/local/lib/python3.9/dist-packages pip requests 2.22.0 /usr/lib/python3/dist-packages requests-toolbelt 1.0.0 /usr/local/lib/python3.9/dist-packages pip requests-unixsocket 0.2.0 /usr/lib/python3/dist-packages rfc3986 2.0.0 /usr/local/lib/python3.9/dist-packages pip rich 13.7.1 /usr/local/lib/python3.9/dist-packages pip SecretStorage 3.3.3 /usr/local/lib/python3.9/dist-packages pip setuptools 69.1.1 /usr/local/lib/python3.9/dist-packages pip six 1.14.0 /usr/lib/python3/dist-packages twine 4.0.2 /usr/local/lib/python3.9/dist-packages pip urllib3 2.2.1 /usr/local/lib/python3.9/dist-packages pip wheel 0.42.0 /usr/local/lib/python3.9/dist-packages pip zipp 3.17.0 /usr/local/lib/python3.9/dist-packages pip TASK [tensorflow_in_container : (Python 3.9) build TensorFlow git] ************* Saturday 01 June 2024 19:27:05 +0100 (0:00:00.037) 0:17:01.498 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) build Python package] ************* Saturday 01 June 2024 20:32:04 +0100 (1:04:58.846) 1:22:00.345 ********* skipping: [localhost] TASK [tensorflow_in_container : (Python 3.9) Copy built wheel] ***************** Saturday 01 June 2024 20:32:04 +0100 (0:00:00.031) 1:22:00.376 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) cleanup Bazel install directory] *** Saturday 01 June 2024 20:32:05 +0100 (0:00:00.909) 1:22:01.285 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create script to run tests] ******* Saturday 01 June 2024 20:32:06 +0100 (0:00:00.869) 1:22:02.155 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) run TensorFlow git tests] ********* Saturday 01 June 2024 20:32:07 +0100 (0:00:00.703) 1:22:02.858 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages] *** Saturday 01 June 2024 22:37:19 +0100 (2:05:11.868) 3:27:14.727 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save list of installed Python packages] *** Saturday 01 June 2024 22:37:21 +0100 (0:00:02.366) 3:27:17.093 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save tests output to file] ******** Saturday 01 June 2024 22:37:22 +0100 (0:00:00.679) 3:27:17.773 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save tests stderr output to file] *** Saturday 01 June 2024 22:37:22 +0100 (0:00:00.694) 3:27:18.467 ********* changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) Extract number of tests run] ****** Saturday 01 June 2024 22:37:23 +0100 (0:00:00.700) 3:27:19.167 ********* ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) Show number of tests run] ********* Saturday 01 June 2024 22:37:23 +0100 (0:00:00.075) 3:27:19.243 ********* ok: [localhost] => ut_tests_line: 'Executed 3087 out of 3087 tests: 3087 tests pass.' TASK [tensorflow_in_container : (Python 3.9) Extract failed tests] ************* Saturday 01 June 2024 22:37:23 +0100 (0:00:00.062) 3:27:19.305 ********* ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) Show which tests are flaky] ******* Saturday 01 June 2024 22:37:23 +0100 (0:00:00.123) 3:27:19.429 ********* ok: [localhost] => tests_flaky: [] TASK [tensorflow_in_container : (Python 3.9) Show which tests failed] ********** Saturday 01 June 2024 22:37:23 +0100 (0:00:00.038) 3:27:19.467 ********* ok: [localhost] => tests_failed: [] TASK [tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one] *** Saturday 01 June 2024 22:37:23 +0100 (0:00:00.039) 3:27:19.507 ********* fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- + for wheel in /tf/pkg/*.whl + echo 'Checking and renaming /tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl...' + python3 -m auditwheel repair --plat manylinux2014_aarch64 /tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl --wheel-dir /tf/pkg + tee check.txt real 1m22.581s user 1m8.234s sys 0m13.796s stderr_lines: <omitted> stdout: |- Checking and renaming /tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl... INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/auditwheel/repair.py", line 80, in repair_wheel raise ValueError( ValueError: Cannot repair wheel, because required library "_pywrap_tensorflow_internal.so" could not be located During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.9/dist-packages/auditwheel/__main__.py", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.9/dist-packages/auditwheel/main.py", line 59, in main rval = args.func(args, p) File "/usr/local/lib/python3.9/dist-packages/auditwheel/main_repair.py", line 175, in execute out_wheel = repair_wheel( File "/usr/local/lib/python3.9/dist-packages/auditwheel/repair.py", line 118, in repair_wheel strip_symbols(itertools.chain(libs_to_strip, extensions)) File "/usr/local/lib/python3.9/dist-packages/auditwheel/wheeltools.py", line 137, in __exit__ dir2zip(self.name, self.out_wheel, date_time) File "/usr/local/lib/python3.9/dist-packages/auditwheel/tools.py", line 76, in dir2zip with zipfile.ZipFile(zip_fname, "w", compression=compression) as z: File "/usr/lib/python3.9/zipfile.py", line 1248, in __init__ self.fp = io.open(file, filemode) PermissionError: [Errno 13] Permission denied: '/tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl' stdout_lines: <omitted> PLAY RECAP ********************************************************************* localhost : ok=39 changed=26 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0 Saturday 01 June 2024 22:38:47 +0100 (0:01:23.357) 3:28:42.864 ********* =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow git tests ------ 7511.87s tensorflow_in_container : (Python 3.9) build TensorFlow git ---------- 3898.85s tensorflow_in_container : (Python 3.9) start build container ---------- 940.31s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one -- 83.36s Cleanup after previous build ------------------------------------------- 36.54s tensorflow : Fetch TensorFlow from git --------------------------------- 23.26s tensorflow : Remove TensorFlow git sources from previous build ---------- 4.43s Gathering Facts --------------------------------------------------------- 2.63s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 2.37s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 1.86s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.49s Create cache dirs ------------------------------------------------------- 1.41s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.21s tensorflow : Save git commit hash to file ------------------------------- 1.14s tensorflow_in_container : (Python 3.9) Copy built wheel ----------------- 0.91s tensorflow_in_container : (Python 3.9) cleanup Bazel install directory --- 0.87s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.87s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.82s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.80s tensorflow_in_container : (Python 3.9) create script to run tests ------- 0.70s Build step 'Execute shell' marked build as failure

6 months, 2 weeks

1
2
0 0

2024

2023

2022

2021

ldcg-ai-ml June 2024