See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/487/display/redirect
Changes:
------------------------------------------ [...truncated 1.55 MB...] //tensorflow/python/saved_model:load_test_cpu PASSED in 57.4s Stats over 10 runs: max = 57.4s, min = 38.5s, avg = 42.4s, dev = 5.3s //tensorflow/compiler/tests:fft_test_cpu PASSED in 22.3s Stats over 12 runs: max = 22.3s, min = 9.8s, avg = 15.5s, dev = 4.5s //tensorflow/compiler/xla/service:triangular_solve_expander_test PASSED in 4.8s Stats over 12 runs: max = 4.8s, min = 2.2s, avg = 3.4s, dev = 0.8s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 17.8s Stats over 12 runs: max = 17.8s, min = 6.3s, avg = 10.7s, dev = 3.6s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 14.7s Stats over 12 runs: max = 14.7s, min = 9.3s, avg = 11.1s, dev = 1.4s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 11.5s Stats over 12 runs: max = 11.5s, min = 7.7s, avg = 9.8s, dev = 1.2s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 228.8s Stats over 12 runs: max = 228.8s, min = 17.3s, avg = 56.2s, dev = 58.1s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 226.0s Stats over 12 runs: max = 226.0s, min = 15.8s, avg = 51.2s, dev = 58.5s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 23.1s Stats over 15 runs: max = 23.1s, min = 16.6s, avg = 19.5s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 80.4s Stats over 15 runs: max = 80.4s, min = 69.2s, avg = 73.7s, dev = 3.2s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 54.2s Stats over 15 runs: max = 54.2s, min = 11.7s, avg = 17.4s, dev = 10.6s //tensorflow/compiler/tests:ftrl_test_cpu PASSED in 10.4s Stats over 16 runs: max = 10.4s, min = 7.1s, avg = 8.4s, dev = 0.9s //tensorflow/compiler/tests:ternary_ops_test_cpu PASSED in 13.7s Stats over 16 runs: max = 13.7s, min = 4.3s, avg = 8.4s, dev = 2.9s //tensorflow/compiler/tests:ternary_ops_test_cpu_mlir_bridge_test PASSED in 13.1s Stats over 16 runs: max = 13.1s, min = 3.9s, avg = 9.6s, dev = 2.6s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 15.6s Stats over 16 runs: max = 15.6s, min = 8.3s, avg = 12.6s, dev = 2.3s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 31.8s Stats over 16 runs: max = 31.8s, min = 8.0s, avg = 14.9s, dev = 5.6s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 44.3s Stats over 16 runs: max = 44.3s, min = 13.9s, avg = 32.2s, dev = 7.4s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 28.5s Stats over 16 runs: max = 28.5s, min = 5.5s, avg = 11.2s, dev = 5.0s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 13.4s Stats over 16 runs: max = 13.4s, min = 7.7s, avg = 9.4s, dev = 1.4s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 14.7s Stats over 16 runs: max = 14.7s, min = 8.7s, avg = 12.1s, dev = 1.9s //tensorflow/python/ops:image_ops_test_cpu PASSED in 23.7s Stats over 16 runs: max = 23.7s, min = 10.9s, avg = 15.4s, dev = 3.2s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 78.2s Stats over 17 runs: max = 78.2s, min = 4.4s, avg = 23.8s, dev = 26.4s //tensorflow/python/data/kernel_tests:map_test PASSED in 56.2s Stats over 19 runs: max = 56.2s, min = 8.8s, avg = 18.5s, dev = 10.0s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 9.3s Stats over 20 runs: max = 9.3s, min = 4.3s, avg = 6.7s, dev = 1.4s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 9.4s Stats over 20 runs: max = 9.4s, min = 3.6s, avg = 5.7s, dev = 1.8s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 15.3s Stats over 20 runs: max = 15.3s, min = 5.2s, avg = 8.1s, dev = 2.3s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 17.6s Stats over 20 runs: max = 17.6s, min = 4.2s, avg = 7.9s, dev = 3.8s //tensorflow/compiler/xla/tests:convolution_dimension_numbers_test_cpu PASSED in 9.7s Stats over 20 runs: max = 9.7s, min = 7.5s, avg = 8.1s, dev = 0.6s //tensorflow/compiler/xla/tests:dot_operation_single_threaded_runtime_test_cpu PASSED in 13.8s Stats over 20 runs: max = 13.8s, min = 10.4s, avg = 11.8s, dev = 0.9s //tensorflow/compiler/xla/tests:dot_operation_test_cpu PASSED in 11.0s Stats over 20 runs: max = 11.0s, min = 9.7s, avg = 10.3s, dev = 0.4s //tensorflow/compiler/xla/tests:prng_test_cpu PASSED in 10.1s Stats over 20 runs: max = 10.1s, min = 7.2s, avg = 8.0s, dev = 0.6s //tensorflow/compiler/xla/tests:reduce_window_test_cpu PASSED in 40.9s Stats over 20 runs: max = 40.9s, min = 7.5s, avg = 17.0s, dev = 10.7s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 28.0s Stats over 20 runs: max = 28.0s, min = 16.2s, avg = 22.7s, dev = 3.6s //tensorflow/python/kernel_tests:metrics_test PASSED in 43.8s Stats over 20 runs: max = 43.8s, min = 7.4s, avg = 19.1s, dev = 9.9s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 9.3s Stats over 20 runs: max = 9.3s, min = 3.9s, avg = 6.2s, dev = 2.1s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 18.3s Stats over 20 runs: max = 18.3s, min = 8.5s, avg = 11.3s, dev = 2.7s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 43.6s Stats over 20 runs: max = 43.6s, min = 4.9s, avg = 15.5s, dev = 12.1s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 93.5s Stats over 20 runs: max = 93.5s, min = 29.8s, avg = 56.0s, dev = 18.8s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 10.4s Stats over 20 runs: max = 10.4s, min = 4.6s, avg = 8.1s, dev = 2.0s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 16.4s Stats over 20 runs: max = 16.4s, min = 6.7s, avg = 11.7s, dev = 2.8s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 179.3s Stats over 20 runs: max = 179.3s, min = 38.8s, avg = 108.2s, dev = 43.1s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 28.0s Stats over 20 runs: max = 28.0s, min = 4.9s, avg = 13.1s, dev = 6.6s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 25.7s Stats over 20 runs: max = 25.7s, min = 8.2s, avg = 16.6s, dev = 6.1s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 21.3s Stats over 20 runs: max = 21.3s, min = 18.6s, avg = 20.4s, dev = 0.6s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 76.7s Stats over 20 runs: max = 76.7s, min = 10.3s, avg = 33.8s, dev = 22.8s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 18.2s Stats over 20 runs: max = 18.2s, min = 10.2s, avg = 13.5s, dev = 2.4s //tensorflow/python/data/experimental/kernel_tests/service:local_workers_test PASSED in 23.3s Stats over 24 runs: max = 23.3s, min = 7.3s, avg = 17.5s, dev = 3.6s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 19.8s Stats over 24 runs: max = 19.8s, min = 6.7s, avg = 11.7s, dev = 3.6s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 20.5s Stats over 24 runs: max = 20.5s, min = 4.3s, avg = 9.9s, dev = 4.9s //tensorflow/compiler/xla/tests:array_elementwise_ops_test_cpu PASSED in 11.9s Stats over 25 runs: max = 11.9s, min = 8.0s, avg = 9.6s, dev = 1.1s //tensorflow/compiler/xla/tests:select_and_scatter_test_cpu PASSED in 37.5s Stats over 25 runs: max = 37.5s, min = 7.8s, avg = 12.6s, dev = 7.4s //tensorflow/compiler/xla/tests:convolution_variants_test_cpu PASSED in 9.3s Stats over 30 runs: max = 9.3s, min = 7.7s, avg = 8.3s, dev = 0.5s //tensorflow/compiler/xla/tests:iota_test_cpu PASSED in 13.6s Stats over 30 runs: max = 13.6s, min = 12.0s, avg = 12.7s, dev = 0.4s //tensorflow/compiler/xla/tests:params_test_cpu PASSED in 10.9s Stats over 30 runs: max = 10.9s, min = 8.0s, avg = 8.7s, dev = 0.5s //tensorflow/compiler/xla/tests:reshape_test_cpu PASSED in 8.3s Stats over 30 runs: max = 8.3s, min = 7.6s, avg = 8.0s, dev = 0.2s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 21.8s Stats over 30 runs: max = 21.8s, min = 4.4s, avg = 11.4s, dev = 5.4s //tensorflow/compiler/xla/tests:reduce_test_cpu PASSED in 9.3s Stats over 31 runs: max = 9.3s, min = 7.9s, avg = 8.6s, dev = 0.4s //tensorflow/compiler/xla/tests:scalar_computations_test_cpu PASSED in 11.0s Stats over 32 runs: max = 11.0s, min = 8.0s, avg = 8.9s, dev = 0.7s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 25.8s Stats over 32 runs: max = 25.8s, min = 6.7s, avg = 12.2s, dev = 4.5s //tensorflow/compiler/xla/tests:batch_normalization_test_cpu PASSED in 11.1s Stats over 40 runs: max = 11.1s, min = 8.1s, avg = 10.2s, dev = 0.6s //tensorflow/compiler/xla/tests:bfloat16_test_cpu PASSED in 12.6s Stats over 40 runs: max = 12.6s, min = 9.5s, avg = 11.3s, dev = 0.8s //tensorflow/compiler/xla/tests:conv_depthwise_backprop_filter_test_cpu PASSED in 11.7s Stats over 40 runs: max = 11.7s, min = 8.9s, avg = 9.8s, dev = 0.6s //tensorflow/compiler/xla/tests:slice_test_cpu PASSED in 12.8s Stats over 40 runs: max = 12.8s, min = 8.6s, avg = 10.2s, dev = 0.9s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.7s Stats over 48 runs: max = 1.7s, min = 0.4s, avg = 0.6s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 62.0s Stats over 50 runs: max = 62.0s, min = 21.0s, avg = 34.3s, dev = 9.6s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 43.8s Stats over 50 runs: max = 43.8s, min = 3.5s, avg = 11.8s, dev = 8.7s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 41.9s Stats over 50 runs: max = 41.9s, min = 3.4s, avg = 11.5s, dev = 8.1s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 24.3s Stats over 50 runs: max = 24.3s, min = 3.6s, avg = 8.1s, dev = 4.0s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 24.7s Stats over 50 runs: max = 24.7s, min = 3.6s, avg = 6.6s, dev = 4.5s //tensorflow/compiler/xla/tests:conv_depthwise_test_cpu PASSED in 12.7s Stats over 50 runs: max = 12.7s, min = 8.2s, avg = 10.0s, dev = 0.9s //tensorflow/compiler/xla/tests:convolution_test_1d_no_vmodule_cpu PASSED in 14.9s Stats over 50 runs: max = 14.9s, min = 10.0s, avg = 12.5s, dev = 1.2s //tensorflow/compiler/xla/tests:convolution_test_cpu PASSED in 14.4s Stats over 50 runs: max = 14.4s, min = 9.1s, avg = 11.6s, dev = 1.1s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 11.6s Stats over 50 runs: max = 11.6s, min = 4.6s, avg = 7.7s, dev = 1.9s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 10.2s Stats over 50 runs: max = 10.2s, min = 3.6s, avg = 5.0s, dev = 1.6s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 10.0s Stats over 50 runs: max = 10.0s, min = 3.7s, avg = 5.4s, dev = 1.8s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 32.4s Stats over 50 runs: max = 32.4s, min = 8.0s, avg = 15.5s, dev = 5.9s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 10.9s Stats over 50 runs: max = 10.9s, min = 3.8s, avg = 5.8s, dev = 1.8s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 14.4s Stats over 50 runs: max = 14.4s, min = 3.7s, avg = 6.0s, dev = 2.8s
Executed 3787 out of 3788 tests: 3786 tests pass, 1 fails to build and 1 fails locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. stdout_lines: <omitted> ...ignoring
TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages] *** Monday 10 July 2023 04:35:07 +0100 (2:19:15.878) 3:25:01.766 *********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save list of installed Python packages] *** Monday 10 July 2023 04:35:09 +0100 (0:00:02.044) 3:25:03.810 *********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save tests output to file] ******** Monday 10 July 2023 04:35:09 +0100 (0:00:00.686) 3:25:04.497 *********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save tests stderr output to file] *** Monday 10 July 2023 04:35:10 +0100 (0:00:00.700) 3:25:05.198 *********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Extract number of tests run] ****** Monday 10 July 2023 04:35:11 +0100 (0:00:00.707) 3:25:05.905 *********** ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Show number of tests run] ********* Monday 10 July 2023 04:35:11 +0100 (0:00:00.080) 3:25:05.986 *********** ok: [localhost] => ut_tests_line: 'Executed 3787 out of 3788 tests: 3786 tests pass, 1 fails to build and 1 fails locally.'
TASK [tensorflow_in_container : (Python 3.9) Extract failed tests] ************* Monday 10 July 2023 04:35:11 +0100 (0:00:00.040) 3:25:06.027 *********** ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Show which tests failed] ********** Monday 10 July 2023 04:35:11 +0100 (0:00:00.071) 3:25:06.099 *********** ok: [localhost] => tests_failed: - 'FAILED: //tensorflow/compiler/xla/service/gpu:fusion_merger_test (Summary)' - 'FAILED: Build did NOT complete successfully'
TASK [tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one] *** Monday 10 July 2023 04:35:11 +0100 (0:00:00.042) 3:25:06.141 *********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Get list of wheel files] ********** Monday 10 July 2023 04:40:33 +0100 (0:05:21.401) 3:30:27.542 *********** ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Get filename of latest wheel] ***** Monday 10 July 2023 04:40:33 +0100 (0:00:00.605) 3:30:28.148 *********** ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) move manylinux2014 wheel outside] *** Monday 10 July 2023 04:40:33 +0100 (0:00:00.060) 3:30:28.209 *********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Remove manylinux2014 wheel] ******* Monday 10 July 2023 04:40:36 +0100 (0:00:02.527) 3:30:30.736 *********** changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) stop build container] ************* Monday 10 July 2023 04:40:36 +0100 (0:00:00.374) 3:30:31.111 *********** fatal: [localhost]: FAILED! => changed=false msg: 'Error removing container 831ffaeddd9ebd17732a49a09af84f7121ed20ce42cde90e5a74e9ebf8cf10e8: UnixHTTPConnectionPool(host=''localhost'', port=None): Read timed out. (read timeout=600)'
PLAY RECAP ********************************************************************* localhost : ok=47 changed=32 unreachable=0 failed=1 skipped=8 rescued=0 ignored=1
Monday 10 July 2023 04:51:12 +0100 (0:10:35.469) 3:41:06.580 *********** =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow-oneDNN-ACL git tests 8355.88s tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git 3733.35s tensorflow_in_container : (Python 3.9) stop build container ----------- 635.47s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one - 321.40s tensorflow_in_container : (Python 3.9) build Python package ----------- 129.41s Cleanup after previous build ------------------------------------------- 32.00s tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 22.59s tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 3.96s tensorflow_in_container : (Python 3.9) start build container ------------ 3.84s Gathering Facts --------------------------------------------------------- 2.66s tensorflow_in_container : (Python 3.9) move manylinux2014 wheel outside --- 2.53s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 2.04s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 1.99s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.67s Create cache dirs ------------------------------------------------------- 1.42s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.22s tensorflow : Save git commit hash to file ------------------------------- 1.13s tensorflow_in_container : (Python 3.9) cleanup Bazel install directory --- 0.89s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.86s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.83s Build step 'Execute shell' marked build as failure