See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/688/display/redirect
Changes:
------------------------------------------ [...truncated 2.22 MB...] //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 36.2s Stats over 10 runs: max = 36.2s, min = 4.3s, avg = 19.4s, dev = 12.2s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 27.1s Stats over 10 runs: max = 27.1s, min = 5.3s, avg = 10.0s, dev = 7.7s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 16.4s Stats over 10 runs: max = 16.4s, min = 5.3s, avg = 7.1s, dev = 3.1s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 12.2s Stats over 10 runs: max = 12.2s, min = 5.0s, avg = 6.8s, dev = 2.4s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 102.6s Stats over 10 runs: max = 102.6s, min = 15.2s, avg = 58.0s, dev = 39.8s //tensorflow/python/ops:special_math_ops_test_cpu PASSED in 49.7s Stats over 10 runs: max = 49.7s, min = 6.6s, avg = 13.8s, dev = 12.4s //tensorflow/python/ops:weak_tensor_special_math_ops_test_cpu PASSED in 12.3s Stats over 10 runs: max = 12.3s, min = 5.0s, avg = 7.6s, dev = 2.2s //tensorflow/python/ops/numpy_ops/tests:np_indexing_test PASSED in 107.5s Stats over 10 runs: max = 107.5s, min = 81.4s, avg = 89.7s, dev = 7.1s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 23.0s Stats over 10 runs: max = 23.0s, min = 15.1s, avg = 17.5s, dev = 2.1s //tensorflow/python/saved_model:load_test_cpu PASSED in 52.1s Stats over 10 runs: max = 52.1s, min = 31.7s, avg = 37.3s, dev = 5.9s //tensorflow/compiler/tests:fft_test_cpu PASSED in 16.4s Stats over 12 runs: max = 16.4s, min = 6.0s, avg = 11.3s, dev = 4.2s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 17.9s Stats over 12 runs: max = 17.9s, min = 4.7s, avg = 9.9s, dev = 4.4s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 16.8s Stats over 12 runs: max = 16.8s, min = 5.2s, avg = 7.9s, dev = 3.1s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 11.5s Stats over 12 runs: max = 11.5s, min = 4.2s, avg = 7.0s, dev = 2.0s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 23.3s Stats over 12 runs: max = 23.3s, min = 12.3s, avg = 14.4s, dev = 2.9s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 23.6s Stats over 12 runs: max = 23.6s, min = 11.5s, avg = 14.8s, dev = 3.1s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 225.2s Stats over 12 runs: max = 225.2s, min = 10.5s, avg = 50.2s, dev = 58.6s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 224.7s Stats over 12 runs: max = 224.7s, min = 10.1s, avg = 50.8s, dev = 58.2s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 24.7s Stats over 15 runs: max = 24.7s, min = 12.4s, avg = 15.8s, dev = 3.4s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 119.1s Stats over 15 runs: max = 119.1s, min = 107.8s, avg = 112.4s, dev = 3.6s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 51.7s Stats over 15 runs: max = 51.7s, min = 5.6s, avg = 12.5s, dev = 11.8s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 20.0s Stats over 16 runs: max = 20.0s, min = 5.4s, avg = 11.9s, dev = 4.7s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 51.0s Stats over 16 runs: max = 51.0s, min = 14.2s, avg = 31.6s, dev = 10.3s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 35.2s Stats over 16 runs: max = 35.2s, min = 6.4s, avg = 10.2s, dev = 6.7s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 11.1s Stats over 16 runs: max = 11.1s, min = 4.6s, avg = 6.4s, dev = 2.1s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 14.4s Stats over 16 runs: max = 14.4s, min = 7.6s, avg = 9.1s, dev = 2.0s //tensorflow/python/ops:image_ops_test_cpu PASSED in 20.2s Stats over 16 runs: max = 20.2s, min = 7.3s, avg = 12.4s, dev = 3.2s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 109.8s Stats over 17 runs: max = 109.8s, min = 12.0s, avg = 42.0s, dev = 29.2s //tensorflow/python/data/kernel_tests:map_test PASSED in 55.8s Stats over 19 runs: max = 55.8s, min = 13.7s, avg = 34.7s, dev = 10.9s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 12.0s Stats over 20 runs: max = 12.0s, min = 4.2s, avg = 5.2s, dev = 1.6s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 11.0s Stats over 20 runs: max = 11.0s, min = 4.4s, avg = 5.2s, dev = 1.4s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 11.1s Stats over 20 runs: max = 11.1s, min = 4.1s, avg = 5.5s, dev = 1.8s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 11.6s Stats over 20 runs: max = 11.6s, min = 3.9s, avg = 5.6s, dev = 2.0s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 11.9s Stats over 20 runs: max = 11.9s, min = 5.1s, avg = 6.4s, dev = 1.5s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 18.6s Stats over 20 runs: max = 18.6s, min = 3.9s, avg = 7.9s, dev = 4.3s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 27.2s Stats over 20 runs: max = 27.2s, min = 4.3s, avg = 9.6s, dev = 6.5s //tensorflow/dtensor/python/tests:rng_test_cpu PASSED in 13.8s Stats over 20 runs: max = 13.8s, min = 7.4s, avg = 8.6s, dev = 1.3s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 26.9s Stats over 20 runs: max = 26.9s, min = 14.0s, avg = 16.5s, dev = 2.8s //tensorflow/python/kernel_tests:metrics_test PASSED in 49.7s Stats over 20 runs: max = 49.7s, min = 7.8s, avg = 19.9s, dev = 11.8s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 10.2s Stats over 20 runs: max = 10.2s, min = 3.8s, avg = 4.6s, dev = 1.3s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 13.2s Stats over 20 runs: max = 13.2s, min = 4.1s, avg = 6.6s, dev = 2.9s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 47.2s Stats over 20 runs: max = 47.2s, min = 4.7s, avg = 14.6s, dev = 14.3s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 117.6s Stats over 20 runs: max = 117.6s, min = 35.9s, avg = 69.2s, dev = 25.8s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 12.6s Stats over 20 runs: max = 12.6s, min = 6.2s, avg = 7.7s, dev = 1.5s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 16.1s Stats over 20 runs: max = 16.1s, min = 7.3s, avg = 12.0s, dev = 2.9s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 235.7s Stats over 20 runs: max = 235.7s, min = 47.1s, avg = 95.7s, dev = 42.5s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 28.3s Stats over 20 runs: max = 28.3s, min = 4.9s, avg = 12.0s, dev = 7.3s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 28.5s Stats over 20 runs: max = 28.5s, min = 8.7s, avg = 15.9s, dev = 5.6s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 31.9s Stats over 20 runs: max = 31.9s, min = 17.2s, avg = 23.2s, dev = 4.1s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 90.0s Stats over 20 runs: max = 90.0s, min = 7.1s, avg = 36.7s, dev = 27.0s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 28.4s Stats over 20 runs: max = 28.4s, min = 10.5s, avg = 12.6s, dev = 3.8s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 42.2s Stats over 24 runs: max = 42.2s, min = 13.6s, avg = 26.2s, dev = 8.4s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 38.9s Stats over 24 runs: max = 38.9s, min = 4.6s, avg = 13.7s, dev = 9.0s //tensorflow/dtensor/python/tests:multi_device_spmd_test_cpu PASSED in 41.3s Stats over 25 runs: max = 41.3s, min = 25.3s, avg = 30.8s, dev = 3.2s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 28.7s Stats over 30 runs: max = 28.7s, min = 4.2s, avg = 9.3s, dev = 5.3s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 25.3s Stats over 32 runs: max = 25.3s, min = 6.2s, avg = 12.3s, dev = 5.9s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test FLAKY, failed in 1 out of 33 in 900.0s Stats over 33 runs: max = 900.0s, min = 4.8s, avg = 39.8s, dev = 152.2s /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/testlogs/tensorflow/python/data/experimental/kernel_tests/service/worker_tags_test/shard_6_of_32/test_attempts/attempt_1.log //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.8s Stats over 48 runs: max = 1.8s, min = 0.4s, avg = 0.5s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 65.1s Stats over 50 runs: max = 65.1s, min = 35.9s, avg = 48.2s, dev = 7.8s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 17.3s Stats over 50 runs: max = 17.3s, min = 3.8s, avg = 10.3s, dev = 3.4s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 17.3s Stats over 50 runs: max = 17.3s, min = 3.9s, avg = 10.4s, dev = 3.8s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 55.3s Stats over 50 runs: max = 55.3s, min = 38.9s, avg = 47.2s, dev = 3.9s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 19.6s Stats over 50 runs: max = 19.6s, min = 6.3s, avg = 11.3s, dev = 3.3s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 34.9s Stats over 50 runs: max = 34.9s, min = 9.7s, avg = 19.5s, dev = 6.7s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 14.1s Stats over 50 runs: max = 14.1s, min = 4.4s, avg = 6.5s, dev = 2.0s
Executed 3059 out of 3059 tests: 3058 tests pass and 1 fails locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. stdout_lines: <omitted> ...ignoring
TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages] *** Wednesday 24 January 2024 03:31:02 +0000 (2:12:10.673) 3:20:57.823 ***** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save list of installed Python packages] *** Wednesday 24 January 2024 03:31:03 +0000 (0:00:01.826) 3:20:59.650 ***** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save tests output to file] ******** Wednesday 24 January 2024 03:31:04 +0000 (0:00:00.721) 3:21:00.372 ***** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save tests stderr output to file] *** Wednesday 24 January 2024 03:31:05 +0000 (0:00:00.707) 3:21:01.079 ***** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Extract number of tests run] ****** Wednesday 24 January 2024 03:31:06 +0000 (0:00:00.706) 3:21:01.785 ***** ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Show number of tests run] ********* Wednesday 24 January 2024 03:31:06 +0000 (0:00:00.106) 3:21:01.892 ***** ok: [localhost] => ut_tests_line: 'Executed 3059 out of 3059 tests: 3058 tests pass and 1 fails locally.'
TASK [tensorflow_in_container : (Python 3.9) Extract failed tests] ************* Wednesday 24 January 2024 03:31:06 +0000 (0:00:00.042) 3:21:01.934 ***** ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Show which tests are flaky] ******* Wednesday 24 January 2024 03:31:06 +0000 (0:00:00.491) 3:21:02.426 ***** ok: [localhost] => tests_flaky: - //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test FLAKY, failed in 1 out of 33 in 900.0s
TASK [tensorflow_in_container : (Python 3.9) Show which tests failed] ********** Wednesday 24 January 2024 03:31:06 +0000 (0:00:00.042) 3:21:02.468 ***** ok: [localhost] => tests_failed: - 'FAILED: //tensorflow/core/grappler/optimizers:mkl_remapper_test (Summary)'
TASK [tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one] *** Wednesday 24 January 2024 03:31:06 +0000 (0:00:00.042) 3:21:02.511 ***** fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- + for wheel in /tf/pkg/*.whl + echo 'Checking and renaming /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl...' + python3 -m auditwheel repair --plat manylinux2014_aarch64 /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl --wheel-dir /tf/pkg + tee check.txt
real 2m35.741s user 2m10.427s sys 0m19.896s ++ grep --extended-regexp --only-matching '/tf/pkg/\S+.whl' check.txt + new_wheel=/tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + [[ /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl != /\t\f/\p\k\g/\t\e\n\s\o\r\f\l\o\w_\a\a\r\c\h\6\4-\2.\1\6.\0-\c\p\3\9-\c\p\3\9-\l\i\n\u\x_\a\a\r\c\h\6\4.\w\h\l ]] + rm /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl + wheel=/tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + TF_WHEEL=/tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + bats /usertools/wheel_verification.bats --timing stderr_lines: <omitted> stdout: |- Checking and renaming /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl... INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl INFO:auditwheel.wheeltools:Previous filename tags: linux_aarch64 INFO:auditwheel.wheeltools:New filename tags: manylinux_2_17_aarch64, manylinux2014_aarch64 INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp39-cp39-linux_aarch64 INFO:auditwheel.wheeltools:New WHEEL info tags: cp39-cp39-manylinux_2_17_aarch64, cp39-cp39-manylinux2014_aarch64 INFO:auditwheel.main_repair: Fixed-up wheel written to /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 1..6 ok 1 Wheel is manylinux2014 (manylinux_2_17) compliant in 72123ms # Size of /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is 209 / 580 megabytes. ok 2 Wheel conforms to upstream size limitations in 17ms ok 3 Wheel is installable in 60733ms ok 4 TensorFlow is importable in 4189ms ok 5 TensorFlow has Keras in 3701ms not ok 6 TensorFlow has Estimator in 3686ms # (in test file /usertools/wheel_verification.bats, line 78) # `python3 -c 'import sys; import tensorflow as tf; sys.exit(0 if "_v2.estimator" in tf.estimator.__name__ else 1)'' failed # Traceback (most recent call last): # File "<string>", line 1, in <module> # AttributeError: module 'tensorflow' has no attribute 'estimator' stdout_lines: <omitted>
PLAY RECAP ********************************************************************* localhost : ok=43 changed=29 unreachable=0 failed=1 skipped=9 rescued=0 ignored=1
Wednesday 24 January 2024 03:36:09 +0000 (0:05:02.882) 3:26:05.393 ***** =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow-oneDNN-ACL git tests 7930.67s tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git 2955.36s tensorflow_in_container : (Python 3.9) start build container ---------- 963.48s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one - 302.88s tensorflow_in_container : (Python 3.9) build Python package ----------- 128.29s Cleanup after previous build ------------------------------------------- 33.80s tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 21.25s tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 4.16s Gathering Facts --------------------------------------------------------- 2.63s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 1.95s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 1.83s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.65s Create cache dirs ------------------------------------------------------- 1.44s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.21s tensorflow : Save git commit hash to file ------------------------------- 1.15s tensorflow_in_container : (Python 3.9) cleanup Bazel install directory --- 0.92s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.81s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.80s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.79s tensorflow_in_container : (Python 3.9) save list of installed Python packages --- 0.72s Build step 'Execute shell' marked build as failure
See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/689/display/redirect
Changes:
------------------------------------------ [...truncated 1.32 MB...] //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 17.8s Stats over 10 runs: max = 17.8s, min = 5.7s, avg = 10.2s, dev = 3.1s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 38.6s Stats over 10 runs: max = 38.6s, min = 5.8s, avg = 17.9s, dev = 10.0s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 46.0s Stats over 10 runs: max = 46.0s, min = 6.3s, avg = 16.0s, dev = 13.8s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 18.8s Stats over 10 runs: max = 18.8s, min = 5.9s, avg = 9.4s, dev = 3.5s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 16.6s Stats over 10 runs: max = 16.6s, min = 5.5s, avg = 9.4s, dev = 3.7s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 146.4s Stats over 10 runs: max = 146.4s, min = 16.1s, avg = 80.6s, dev = 57.1s //tensorflow/python/ops:special_math_ops_test_cpu PASSED in 71.0s Stats over 10 runs: max = 71.0s, min = 7.9s, avg = 18.9s, dev = 17.9s //tensorflow/python/ops:weak_tensor_special_math_ops_test_cpu PASSED in 11.8s Stats over 10 runs: max = 11.8s, min = 5.5s, avg = 9.2s, dev = 2.1s //tensorflow/python/ops/numpy_ops/tests:np_indexing_test PASSED in 104.0s Stats over 10 runs: max = 104.0s, min = 81.3s, avg = 89.7s, dev = 5.8s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 21.9s Stats over 10 runs: max = 21.9s, min = 13.7s, avg = 15.2s, dev = 2.3s //tensorflow/python/saved_model:load_test_cpu PASSED in 58.6s Stats over 10 runs: max = 58.6s, min = 30.0s, avg = 37.1s, dev = 8.0s //tensorflow/compiler/tests:fft_test_cpu PASSED in 19.5s Stats over 12 runs: max = 19.5s, min = 7.0s, avg = 12.2s, dev = 4.3s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 18.1s Stats over 12 runs: max = 18.1s, min = 5.0s, avg = 9.8s, dev = 4.2s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 16.1s Stats over 12 runs: max = 16.1s, min = 4.9s, avg = 8.0s, dev = 3.2s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 11.9s Stats over 12 runs: max = 11.9s, min = 4.3s, avg = 7.1s, dev = 2.5s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 23.9s Stats over 12 runs: max = 23.9s, min = 15.4s, avg = 20.3s, dev = 2.8s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 25.3s Stats over 12 runs: max = 25.3s, min = 12.8s, avg = 16.9s, dev = 3.9s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 225.6s Stats over 12 runs: max = 225.6s, min = 12.4s, avg = 52.2s, dev = 58.1s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 229.6s Stats over 12 runs: max = 229.6s, min = 11.8s, avg = 53.8s, dev = 58.3s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 24.0s Stats over 15 runs: max = 24.0s, min = 12.3s, avg = 15.5s, dev = 2.7s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 166.1s Stats over 15 runs: max = 166.1s, min = 114.7s, avg = 145.4s, dev = 14.8s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 77.9s Stats over 15 runs: max = 77.9s, min = 7.0s, avg = 18.8s, dev = 18.2s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 12.5s Stats over 16 runs: max = 12.5s, min = 4.7s, avg = 8.5s, dev = 2.6s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 30.9s Stats over 16 runs: max = 30.9s, min = 9.5s, avg = 18.5s, dev = 6.0s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 33.0s Stats over 16 runs: max = 33.0s, min = 6.6s, avg = 11.3s, dev = 6.1s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 10.9s Stats over 16 runs: max = 10.9s, min = 5.2s, avg = 7.4s, dev = 1.8s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 21.9s Stats over 16 runs: max = 21.9s, min = 8.5s, avg = 13.1s, dev = 3.6s //tensorflow/python/ops:image_ops_test_cpu PASSED in 23.8s Stats over 16 runs: max = 23.8s, min = 9.4s, avg = 16.3s, dev = 3.9s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 83.2s Stats over 17 runs: max = 83.2s, min = 8.8s, avg = 32.4s, dev = 25.1s //tensorflow/python/data/kernel_tests:map_test PASSED in 53.6s Stats over 19 runs: max = 53.6s, min = 11.5s, avg = 22.9s, dev = 9.1s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 10.9s Stats over 20 runs: max = 10.9s, min = 3.9s, avg = 5.2s, dev = 1.5s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 12.2s Stats over 20 runs: max = 12.2s, min = 4.1s, avg = 5.8s, dev = 1.7s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 11.5s Stats over 20 runs: max = 11.5s, min = 4.3s, avg = 6.2s, dev = 1.8s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 11.9s Stats over 20 runs: max = 11.9s, min = 4.1s, avg = 6.0s, dev = 2.3s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 10.7s Stats over 20 runs: max = 10.7s, min = 5.1s, avg = 6.3s, dev = 1.1s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 16.2s Stats over 20 runs: max = 16.2s, min = 4.2s, avg = 7.9s, dev = 3.9s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 23.9s Stats over 20 runs: max = 23.9s, min = 4.3s, avg = 9.0s, dev = 5.6s //tensorflow/dtensor/python/tests:rng_test_cpu PASSED in 13.4s Stats over 20 runs: max = 13.4s, min = 7.3s, avg = 8.9s, dev = 1.5s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 32.3s Stats over 20 runs: max = 32.3s, min = 13.4s, avg = 16.7s, dev = 3.7s //tensorflow/python/kernel_tests:metrics_test PASSED in 59.8s Stats over 20 runs: max = 59.8s, min = 7.5s, avg = 26.1s, dev = 15.2s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 9.9s Stats over 20 runs: max = 9.9s, min = 3.8s, avg = 4.6s, dev = 1.3s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 12.7s Stats over 20 runs: max = 12.7s, min = 4.2s, avg = 6.6s, dev = 2.7s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 48.1s Stats over 20 runs: max = 48.1s, min = 5.6s, avg = 18.0s, dev = 15.3s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 149.3s Stats over 20 runs: max = 149.3s, min = 35.6s, avg = 74.0s, dev = 30.3s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 11.9s Stats over 20 runs: max = 11.9s, min = 5.9s, avg = 8.2s, dev = 1.4s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 18.6s Stats over 20 runs: max = 18.6s, min = 7.4s, avg = 12.6s, dev = 3.2s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 156.5s Stats over 20 runs: max = 156.5s, min = 38.9s, avg = 79.9s, dev = 31.3s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 28.8s Stats over 20 runs: max = 28.8s, min = 5.2s, avg = 12.5s, dev = 7.2s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 20.2s Stats over 20 runs: max = 20.2s, min = 8.8s, avg = 14.0s, dev = 3.4s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 34.0s Stats over 20 runs: max = 34.0s, min = 20.9s, avg = 24.3s, dev = 3.4s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 87.6s Stats over 20 runs: max = 87.6s, min = 6.8s, avg = 37.8s, dev = 25.6s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 28.9s Stats over 20 runs: max = 28.9s, min = 12.9s, avg = 18.8s, dev = 3.6s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 27.4s Stats over 24 runs: max = 27.4s, min = 9.8s, avg = 17.4s, dev = 4.7s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 20.2s Stats over 24 runs: max = 20.2s, min = 4.7s, avg = 9.4s, dev = 4.7s //tensorflow/dtensor/python/tests:multi_device_spmd_test_cpu PASSED in 32.6s Stats over 25 runs: max = 32.6s, min = 24.2s, avg = 27.8s, dev = 2.2s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 48.7s Stats over 30 runs: max = 48.7s, min = 4.3s, avg = 15.5s, dev = 11.1s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 19.9s Stats over 32 runs: max = 19.9s, min = 4.8s, avg = 9.0s, dev = 3.8s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 22.8s Stats over 32 runs: max = 22.8s, min = 4.6s, avg = 10.5s, dev = 4.4s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.8s Stats over 48 runs: max = 1.8s, min = 0.4s, avg = 0.5s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 66.0s Stats over 50 runs: max = 66.0s, min = 31.4s, avg = 44.6s, dev = 8.7s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 19.6s Stats over 50 runs: max = 19.6s, min = 4.1s, avg = 10.5s, dev = 3.6s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 18.6s Stats over 50 runs: max = 18.6s, min = 4.7s, avg = 10.4s, dev = 3.5s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 84.6s Stats over 50 runs: max = 84.6s, min = 46.4s, avg = 62.2s, dev = 9.3s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 22.2s Stats over 50 runs: max = 22.2s, min = 5.2s, avg = 11.6s, dev = 3.8s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 39.3s Stats over 50 runs: max = 39.3s, min = 9.7s, avg = 20.0s, dev = 6.6s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 15.2s Stats over 50 runs: max = 15.2s, min = 4.3s, avg = 6.7s, dev = 2.3s
Executed 3059 out of 3059 tests: 3058 tests pass and 1 fails locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. stdout_lines: <omitted> ...ignoring
TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages] *** Thursday 25 January 2024 03:16:54 +0000 (1:57:01.294) 3:06:50.152 ****** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save list of installed Python packages] *** Thursday 25 January 2024 03:16:56 +0000 (0:00:01.920) 3:06:52.073 ****** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save tests output to file] ******** Thursday 25 January 2024 03:16:56 +0000 (0:00:00.699) 3:06:52.772 ****** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save tests stderr output to file] *** Thursday 25 January 2024 03:16:57 +0000 (0:00:00.697) 3:06:53.469 ****** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Extract number of tests run] ****** Thursday 25 January 2024 03:16:58 +0000 (0:00:00.710) 3:06:54.179 ****** ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Show number of tests run] ********* Thursday 25 January 2024 03:16:58 +0000 (0:00:00.082) 3:06:54.262 ****** ok: [localhost] => ut_tests_line: 'Executed 3059 out of 3059 tests: 3058 tests pass and 1 fails locally.'
TASK [tensorflow_in_container : (Python 3.9) Extract failed tests] ************* Thursday 25 January 2024 03:16:58 +0000 (0:00:00.043) 3:06:54.306 ****** ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Show which tests are flaky] ******* Thursday 25 January 2024 03:16:58 +0000 (0:00:00.146) 3:06:54.453 ****** ok: [localhost] => tests_flaky: []
TASK [tensorflow_in_container : (Python 3.9) Show which tests failed] ********** Thursday 25 January 2024 03:16:58 +0000 (0:00:00.043) 3:06:54.497 ****** ok: [localhost] => tests_failed: - 'FAILED: //tensorflow/core/grappler/optimizers:mkl_remapper_test (Summary)'
TASK [tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one] *** Thursday 25 January 2024 03:16:58 +0000 (0:00:00.044) 3:06:54.541 ****** fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- + for wheel in /tf/pkg/*.whl + echo 'Checking and renaming /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl...' + python3 -m auditwheel repair --plat manylinux2014_aarch64 /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl --wheel-dir /tf/pkg + tee check.txt
real 2m33.285s user 2m9.906s sys 0m19.618s ++ grep --extended-regexp --only-matching '/tf/pkg/\S+.whl' check.txt + new_wheel=/tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + [[ /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl != /\t\f/\p\k\g/\t\e\n\s\o\r\f\l\o\w_\a\a\r\c\h\6\4-\2.\1\6.\0-\c\p\3\9-\c\p\3\9-\l\i\n\u\x_\a\a\r\c\h\6\4.\w\h\l ]] + rm /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl + wheel=/tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + TF_WHEEL=/tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + bats /usertools/wheel_verification.bats --timing stderr_lines: <omitted> stdout: |- Checking and renaming /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl... INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl INFO:auditwheel.wheeltools:Previous filename tags: linux_aarch64 INFO:auditwheel.wheeltools:New filename tags: manylinux_2_17_aarch64, manylinux2014_aarch64 INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp39-cp39-linux_aarch64 INFO:auditwheel.wheeltools:New WHEEL info tags: cp39-cp39-manylinux_2_17_aarch64, cp39-cp39-manylinux2014_aarch64 INFO:auditwheel.main_repair: Fixed-up wheel written to /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 1..6 ok 1 Wheel is manylinux2014 (manylinux_2_17) compliant in 72531ms # Size of /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is 209 / 580 megabytes. ok 2 Wheel conforms to upstream size limitations in 13ms ok 3 Wheel is installable in 61394ms ok 4 TensorFlow is importable in 3880ms ok 5 TensorFlow has Keras in 3654ms not ok 6 TensorFlow has Estimator in 3642ms # (in test file /usertools/wheel_verification.bats, line 78) # `python3 -c 'import sys; import tensorflow as tf; sys.exit(0 if "_v2.estimator" in tf.estimator.__name__ else 1)'' failed # Traceback (most recent call last): # File "<string>", line 1, in <module> # AttributeError: module 'tensorflow' has no attribute 'estimator' stdout_lines: <omitted>
PLAY RECAP ********************************************************************* localhost : ok=43 changed=29 unreachable=0 failed=1 skipped=9 rescued=0 ignored=1
Thursday 25 January 2024 03:21:59 +0000 (0:05:00.835) 3:11:55.376 ****** =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow-oneDNN-ACL git tests 7021.29s tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git 2939.76s tensorflow_in_container : (Python 3.9) start build container --------- 1042.85s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one - 300.84s tensorflow_in_container : (Python 3.9) build Python package ----------- 127.59s Cleanup after previous build ------------------------------------------- 32.53s tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 21.99s tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 4.16s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 1.93s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 1.92s Gathering Facts --------------------------------------------------------- 1.69s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.63s Create cache dirs ------------------------------------------------------- 1.44s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.20s tensorflow : Save git commit hash to file ------------------------------- 1.15s tensorflow_in_container : (Python 3.9) cleanup Bazel install directory --- 0.93s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.86s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.84s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.81s tensorflow_in_container : (Python 3.9) save tests stderr output to file --- 0.71s Build step 'Execute shell' marked build as failure
ldcg-ai-ml@op-lists.linaro.org