See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/835/display/redirect
Changes:
------------------------------------------ [...truncated 1.37 MiB...] Stats over 10 runs: max = 63.3s, min = 32.7s, avg = 39.6s, dev = 8.8s //tensorflow/compiler/tests:fft_test_cpu PASSED in 20.4s Stats over 12 runs: max = 20.4s, min = 7.2s, avg = 14.7s, dev = 4.8s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 16.4s Stats over 12 runs: max = 16.4s, min = 4.1s, avg = 9.5s, dev = 3.9s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 15.8s Stats over 12 runs: max = 15.8s, min = 5.2s, avg = 7.3s, dev = 2.8s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 10.8s Stats over 12 runs: max = 10.8s, min = 4.4s, avg = 6.5s, dev = 1.9s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 23.4s Stats over 12 runs: max = 23.4s, min = 11.4s, avg = 14.4s, dev = 3.1s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 22.7s Stats over 12 runs: max = 22.7s, min = 11.3s, avg = 15.2s, dev = 2.8s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 28.5s Stats over 15 runs: max = 28.5s, min = 12.1s, avg = 17.1s, dev = 4.9s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 176.0s Stats over 15 runs: max = 176.0s, min = 135.5s, avg = 150.3s, dev = 11.0s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 54.7s Stats over 15 runs: max = 54.7s, min = 6.6s, avg = 13.4s, dev = 12.3s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 11.9s Stats over 16 runs: max = 11.9s, min = 4.2s, avg = 7.8s, dev = 2.7s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 35.2s Stats over 16 runs: max = 35.2s, min = 12.0s, avg = 19.6s, dev = 6.0s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 28.1s Stats over 16 runs: max = 28.1s, min = 5.8s, avg = 9.2s, dev = 5.3s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 16.8s Stats over 16 runs: max = 16.8s, min = 4.3s, avg = 7.6s, dev = 3.0s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 17.0s Stats over 16 runs: max = 17.0s, min = 8.4s, avg = 11.5s, dev = 2.1s //tensorflow/python/ops:image_ops_test_cpu PASSED in 26.9s Stats over 16 runs: max = 26.9s, min = 8.6s, avg = 15.0s, dev = 4.5s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 57.9s Stats over 17 runs: max = 57.9s, min = 8.3s, avg = 25.3s, dev = 15.6s //tensorflow/python/data/kernel_tests:map_test PASSED in 76.9s Stats over 19 runs: max = 76.9s, min = 27.9s, avg = 40.2s, dev = 10.5s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 13.9s Stats over 20 runs: max = 13.9s, min = 4.2s, avg = 5.6s, dev = 2.0s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 12.4s Stats over 20 runs: max = 12.4s, min = 4.3s, avg = 5.9s, dev = 1.9s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 11.3s Stats over 20 runs: max = 11.3s, min = 4.1s, avg = 5.7s, dev = 1.8s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 12.2s Stats over 20 runs: max = 12.2s, min = 4.4s, avg = 5.9s, dev = 2.1s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 12.0s Stats over 20 runs: max = 12.0s, min = 5.1s, avg = 6.0s, dev = 1.4s //tensorflow/dtensor/python/tests:rng_test_cpu PASSED in 14.2s Stats over 20 runs: max = 14.2s, min = 6.5s, avg = 7.9s, dev = 1.6s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 29.1s Stats over 20 runs: max = 29.1s, min = 14.0s, avg = 16.8s, dev = 3.1s //tensorflow/python/kernel_tests:metrics_test PASSED in 44.4s Stats over 20 runs: max = 44.4s, min = 7.9s, avg = 20.0s, dev = 10.2s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 10.5s Stats over 20 runs: max = 10.5s, min = 3.8s, avg = 4.8s, dev = 1.3s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 12.9s Stats over 20 runs: max = 12.9s, min = 4.1s, avg = 6.3s, dev = 2.9s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 56.6s Stats over 20 runs: max = 56.6s, min = 5.1s, avg = 17.5s, dev = 15.9s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 136.5s Stats over 20 runs: max = 136.5s, min = 42.0s, avg = 76.5s, dev = 27.9s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 14.0s Stats over 20 runs: max = 14.0s, min = 6.4s, avg = 9.1s, dev = 2.2s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 19.2s Stats over 20 runs: max = 19.2s, min = 8.1s, avg = 12.7s, dev = 3.2s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 153.2s Stats over 20 runs: max = 153.2s, min = 42.6s, avg = 82.8s, dev = 32.0s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 25.5s Stats over 20 runs: max = 25.5s, min = 5.0s, avg = 12.1s, dev = 6.5s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 23.1s Stats over 20 runs: max = 23.1s, min = 8.7s, avg = 14.3s, dev = 3.7s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 34.8s Stats over 20 runs: max = 34.8s, min = 21.2s, avg = 26.1s, dev = 3.7s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 81.5s Stats over 20 runs: max = 81.5s, min = 10.0s, avg = 38.1s, dev = 24.7s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 42.4s Stats over 20 runs: max = 42.4s, min = 11.8s, avg = 19.0s, dev = 7.2s //tensorflow/compiler/tests:unary_ops_test_cpu FAILED in 3 out of 22 in 25.0s Stats over 22 runs: max = 25.0s, min = 4.2s, avg = 10.4s, dev = 6.5s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu/shard_18_of_20/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu/shard_18_of_20/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu/shard_18_of_20/test_attempts/attempt_2.log //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test FAILED in 3 out of 22 in 27.0s Stats over 22 runs: max = 27.0s, min = 3.8s, avg = 11.0s, dev = 6.5s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu_mlir_bridge_test/shard_18_of_20/test.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu_mlir_bridge_test/shard_18_of_20/test_attempts/attempt_1.log /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/compiler/tests/unary_ops_test_cpu_mlir_bridge_test/shard_18_of_20/test_attempts/attempt_2.log //tensorflow/python/data/kernel_tests:interleave_test PASSED in 31.1s Stats over 24 runs: max = 31.1s, min = 9.2s, avg = 17.2s, dev = 6.3s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 20.4s Stats over 24 runs: max = 20.4s, min = 4.6s, avg = 9.8s, dev = 4.8s //tensorflow/dtensor/python/tests:multi_device_spmd_test_cpu PASSED in 31.1s Stats over 25 runs: max = 31.1s, min = 23.1s, avg = 27.2s, dev = 2.0s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 43.9s Stats over 30 runs: max = 43.9s, min = 4.4s, avg = 15.1s, dev = 10.3s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 17.4s Stats over 32 runs: max = 17.4s, min = 4.4s, avg = 8.7s, dev = 3.3s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 19.6s Stats over 32 runs: max = 19.6s, min = 4.2s, avg = 9.9s, dev = 3.9s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 215.6s Stats over 35 runs: max = 215.6s, min = 4.5s, avg = 22.8s, dev = 37.1s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 218.8s Stats over 35 runs: max = 218.8s, min = 4.3s, avg = 23.7s, dev = 37.5s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.7s Stats over 48 runs: max = 1.7s, min = 0.4s, avg = 0.5s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 63.3s Stats over 50 runs: max = 63.3s, min = 31.5s, avg = 43.2s, dev = 8.0s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 21.2s Stats over 50 runs: max = 21.2s, min = 3.9s, avg = 11.7s, dev = 4.5s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 18.3s Stats over 50 runs: max = 18.3s, min = 4.2s, avg = 10.9s, dev = 4.0s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 83.4s Stats over 50 runs: max = 83.4s, min = 48.2s, avg = 60.8s, dev = 7.9s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 21.4s Stats over 50 runs: max = 21.4s, min = 6.2s, avg = 11.4s, dev = 3.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_onednn_grad_test PASSED in 25.8s Stats over 50 runs: max = 25.8s, min = 5.5s, avg = 11.9s, dev = 4.2s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 39.1s Stats over 50 runs: max = 39.1s, min = 10.5s, avg = 21.3s, dev = 7.2s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 12.3s Stats over 50 runs: max = 12.3s, min = 4.3s, avg = 6.3s, dev = 1.6s
Executed 3086 out of 3086 tests: 3084 tests pass and 2 fail locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. stdout_lines: <omitted> ...ignoring
TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages] *** Tuesday 18 June 2024 04:41:58 +0100 (2:05:25.924) 3:31:53.188 ********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save list of installed Python packages] *** Tuesday 18 June 2024 04:42:00 +0100 (0:00:02.409) 3:31:55.597 ********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save tests output to file] ******** Tuesday 18 June 2024 04:42:01 +0100 (0:00:00.691) 3:31:56.288 ********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save tests stderr output to file] *** Tuesday 18 June 2024 04:42:02 +0100 (0:00:00.699) 3:31:56.988 ********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Extract number of tests run] ****** Tuesday 18 June 2024 04:42:02 +0100 (0:00:00.704) 3:31:57.692 ********** ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Show number of tests run] ********* Tuesday 18 June 2024 04:42:03 +0100 (0:00:00.080) 3:31:57.772 ********** ok: [localhost] => ut_tests_line: 'Executed 3086 out of 3086 tests: 3084 tests pass and 2 fail locally.'
TASK [tensorflow_in_container : (Python 3.9) Extract failed tests] ************* Tuesday 18 June 2024 04:42:03 +0100 (0:00:00.040) 3:31:57.813 ********** ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Show which tests are flaky] ******* Tuesday 18 June 2024 04:42:03 +0100 (0:00:00.156) 3:31:57.969 ********** ok: [localhost] => tests_flaky: []
TASK [tensorflow_in_container : (Python 3.9) Show which tests failed] ********** Tuesday 18 June 2024 04:42:03 +0100 (0:00:00.039) 3:31:58.009 ********** ok: [localhost] => tests_failed: - 'FAILED: //tensorflow/compiler/tests:unary_ops_test_cpu (Summary)' - 'FAILED: //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test (Summary)'
TASK [tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one] *** Tuesday 18 June 2024 04:42:03 +0100 (0:00:00.040) 3:31:58.050 ********** fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- + for wheel in /tf/pkg/*.whl + echo 'Checking and renaming /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl...' + python3 -m auditwheel repair --plat manylinux2014_aarch64 /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl --wheel-dir /tf/pkg + tee check.txt
real 2m38.362s user 2m11.527s sys 0m20.972s ++ grep --extended-regexp --only-matching '/tf/pkg/\S+.whl' check.txt + new_wheel=/tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + [[ /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl != /\t\f/\p\k\g/\t\e\n\s\o\r\f\l\o\w_\a\a\r\c\h\6\4-\2.\1\8.\0-\c\p\3\9-\c\p\3\9-\l\i\n\u\x_\a\a\r\c\h\6\4.\w\h\l ]] + rm /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl + wheel=/tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + TF_WHEEL=/tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + bats /usertools/wheel_verification.bats --timing stderr_lines: <omitted> stdout: |- Checking and renaming /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl... INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.18.0-cp39-cp39-linux_aarch64.whl INFO:auditwheel.wheeltools:Previous filename tags: linux_aarch64 INFO:auditwheel.wheeltools:New filename tags: manylinux_2_17_aarch64, manylinux2014_aarch64 INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp39-cp39-linux_aarch64 INFO:auditwheel.wheeltools:New WHEEL info tags: cp39-cp39-manylinux_2_17_aarch64, cp39-cp39-manylinux2014_aarch64 INFO:auditwheel.main_repair: Fixed-up wheel written to /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 1..5 ok 1 Wheel is manylinux2014 (manylinux_2_17) compliant in 75030ms # Size of /tf/pkg/tensorflow_aarch64-2.18.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is 209 / 580 megabytes. ok 2 Wheel conforms to upstream size limitations in 18ms ok 3 Wheel is installable in 58309ms ok 4 TensorFlow is importable in 4345ms not ok 5 TensorFlow has Keras in 3732ms # (in test file /usertools/wheel_verification.bats, line 72) # `python3 -c 'import sys; import tensorflow as tf; sys.exit(0 if "keras" in tf.keras.__name__ else 1)'' failed # Traceback (most recent call last): # File "/tf/venv/lib/python3.9/site-packages/tensorflow/python/util/lazy_loader.py", line 147, in _initialize # import keras # pylint: disable=g-import-not-at-top # File "/tf/venv/lib/python3.9/site-packages/keras/__init__.py", line 23, in <module> # from keras.api import _tf_keras # ImportError: cannot import name '_tf_keras' from 'keras.api' (/tf/venv/lib/python3.9/site-packages/keras/api/__init__.py) # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "<string>", line 1, in <module> # File "/tf/venv/lib/python3.9/site-packages/tensorflow/python/util/lazy_loader.py", line 181, in __getattr__ # self._initialize() # File "/tf/venv/lib/python3.9/site-packages/tensorflow/python/util/lazy_loader.py", line 161, in _initialize # raise ImportError( # pylint: disable=raise-missing-from # ImportError: Keras cannot be imported. Check that it is installed. stdout_lines: <omitted>
PLAY RECAP ********************************************************************* localhost : ok=39 changed=26 unreachable=0 failed=1 skipped=7 rescued=0 ignored=1
Tuesday 18 June 2024 04:47:05 +0100 (0:05:02.388) 3:37:00.438 ********** =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow-oneDNN-ACL git tests 7525.92s tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git 5080.08s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one - 302.39s Cleanup after previous build ------------------------------------------- 49.44s tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 23.05s tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 5.28s tensorflow_in_container : (Python 3.9) start build container ------------ 4.74s tensorflow_in_container : (Python 3.9) cleanup Bazel install directory --- 3.00s Gathering Facts --------------------------------------------------------- 2.72s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 2.53s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 2.41s tensorflow_in_container : (Python 3.9) Copy built wheel ----------------- 2.23s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.70s Create cache dirs ------------------------------------------------------- 1.44s tensorflow_in_container : (Python 3.9) create script to run tests ------- 1.37s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.22s tensorflow : Save git commit hash to file ------------------------------- 1.15s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.85s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.83s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.80s Build step 'Execute shell' marked build as failure