See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/978/display/redirect
Changes:
------------------------------------------ [...truncated 1.20 MB...] //tensorflow/python/kernel_tests/array_ops:array_ops_test_cpu PASSED in 16.2s Stats over 10 runs: max = 16.2s, min = 11.4s, avg = 13.8s, dev = 1.5s //tensorflow/python/kernel_tests/array_ops:inplace_ops_test_cpu PASSED in 10.5s Stats over 10 runs: max = 10.5s, min = 4.2s, avg = 8.3s, dev = 2.1s //tensorflow/python/kernel_tests/data_structures:tensor_array_ops_test_cpu PASSED in 11.9s Stats over 10 runs: max = 11.9s, min = 8.0s, avg = 10.2s, dev = 1.2s //tensorflow/python/kernel_tests/linalg:linear_operator_low_rank_update_test_cpu PASSED in 91.4s Stats over 10 runs: max = 91.4s, min = 83.0s, avg = 86.2s, dev = 2.2s //tensorflow/python/kernel_tests/linalg:tridiagonal_matmul_op_test_cpu PASSED in 112.4s Stats over 10 runs: max = 112.4s, min = 6.4s, avg = 18.2s, dev = 31.4s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_ops_test_cpu PASSED in 65.3s Stats over 10 runs: max = 65.3s, min = 8.2s, avg = 40.0s, dev = 19.6s //tensorflow/python/kernel_tests/math_ops:segment_reduction_ops_test_cpu PASSED in 27.3s Stats over 10 runs: max = 27.3s, min = 7.1s, avg = 16.2s, dev = 7.7s //tensorflow/python/kernel_tests/nn_ops:pooling_ops_test_cpu PASSED in 140.5s Stats over 10 runs: max = 140.5s, min = 124.2s, avg = 131.0s, dev = 4.5s //tensorflow/python/kernel_tests/nn_ops:rnn_test_cpu PASSED in 17.7s Stats over 10 runs: max = 17.7s, min = 13.9s, avg = 15.5s, dev = 1.3s //tensorflow/python/kernel_tests/random:random_index_shuffle_test PASSED in 11.2s Stats over 10 runs: max = 11.2s, min = 8.3s, avg = 10.4s, dev = 0.8s //tensorflow/python/kernel_tests/random:stateless_random_ops_test_cpu PASSED in 100.7s Stats over 10 runs: max = 100.7s, min = 20.4s, avg = 58.8s, dev = 37.2s //tensorflow/python/ops:special_math_ops_test_cpu PASSED in 50.2s Stats over 10 runs: max = 50.2s, min = 10.6s, avg = 16.9s, dev = 11.3s //tensorflow/python/ops:weak_tensor_special_math_ops_test_cpu PASSED in 13.2s Stats over 10 runs: max = 13.2s, min = 10.7s, avg = 11.8s, dev = 0.8s //tensorflow/python/ops/numpy_ops/tests:np_indexing_test PASSED in 95.7s Stats over 10 runs: max = 95.7s, min = 88.2s, avg = 92.8s, dev = 2.3s //tensorflow/python/ops/ragged:ragged_tensor_supported_values_test PASSED in 24.2s Stats over 10 runs: max = 24.2s, min = 19.8s, avg = 22.1s, dev = 1.2s //tensorflow/python/saved_model:load_test_cpu PASSED in 60.2s Stats over 10 runs: max = 60.2s, min = 34.0s, avg = 40.4s, dev = 7.0s //tensorflow/compiler/tests:fft_test_cpu PASSED in 25.0s Stats over 12 runs: max = 25.0s, min = 13.8s, avg = 19.7s, dev = 4.3s //tensorflow/python/data/experimental/kernel_tests:group_by_reducer_test PASSED in 21.7s Stats over 12 runs: max = 21.7s, min = 5.8s, avg = 11.1s, dev = 4.6s //tensorflow/python/data/kernel_tests:choose_from_datasets_test PASSED in 25.1s Stats over 12 runs: max = 25.1s, min = 16.7s, avg = 20.7s, dev = 2.2s //tensorflow/python/data/kernel_tests:memory_cleanup_test_cpu PASSED in 12.8s Stats over 12 runs: max = 12.8s, min = 8.1s, avg = 10.2s, dev = 1.5s //tensorflow/python/distribute:moving_averages_test_2gpu PASSED in 24.2s Stats over 12 runs: max = 24.2s, min = 12.6s, avg = 17.7s, dev = 3.8s //tensorflow/python/distribute:moving_averages_test_cpu PASSED in 19.9s Stats over 12 runs: max = 19.9s, min = 17.2s, avg = 18.7s, dev = 0.8s //tensorflow/python/distribute:multi_process_runner_test_2gpu PASSED in 230.8s Stats over 12 runs: max = 230.8s, min = 17.4s, avg = 54.1s, dev = 58.9s //tensorflow/python/distribute:multi_process_runner_test_cpu PASSED in 228.8s Stats over 12 runs: max = 228.8s, min = 17.2s, avg = 54.9s, dev = 58.4s //tensorflow/python/eager/polymorphic_function:polymorphic_function_test_cpu PASSED in 21.6s Stats over 15 runs: max = 21.6s, min = 15.7s, avg = 17.7s, dev = 1.5s //tensorflow/python/kernel_tests/nn_ops:rnn_cell_test_cpu PASSED in 48.5s Stats over 15 runs: max = 48.5s, min = 7.9s, avg = 16.6s, dev = 10.8s //tensorflow/python/data/experimental/kernel_tests/service:dynamic_sharding_test PASSED in 15.0s Stats over 16 runs: max = 15.0s, min = 5.0s, avg = 10.4s, dev = 2.9s //tensorflow/python/data/kernel_tests:snapshot_test PASSED in 37.6s Stats over 16 runs: max = 37.6s, min = 14.0s, avg = 23.4s, dev = 7.0s //tensorflow/python/kernel_tests/control_flow:control_flow_ops_py_test_cpu PASSED in 44.2s Stats over 16 runs: max = 44.2s, min = 18.1s, avg = 26.1s, dev = 5.7s //tensorflow/python/kernel_tests/linalg:matrix_exponential_op_test PASSED in 10.7s Stats over 16 runs: max = 10.7s, min = 4.1s, avg = 5.4s, dev = 1.7s //tensorflow/python/kernel_tests/signal:dct_ops_test_cpu PASSED in 15.9s Stats over 16 runs: max = 15.9s, min = 9.0s, avg = 12.8s, dev = 1.8s //tensorflow/python/ops:image_ops_test_cpu PASSED in 21.0s Stats over 16 runs: max = 21.0s, min = 8.7s, avg = 14.6s, dev = 3.2s //tensorflow/python/data/experimental/kernel_tests/service:distributed_save_ft_test PASSED in 69.8s Stats over 17 runs: max = 69.8s, min = 9.5s, avg = 26.2s, dev = 18.7s //tensorflow/python/data/kernel_tests:map_test PASSED in 39.2s Stats over 19 runs: max = 39.2s, min = 10.8s, avg = 21.8s, dev = 6.2s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu PASSED in 10.3s Stats over 20 runs: max = 10.3s, min = 3.7s, avg = 6.4s, dev = 2.2s //tensorflow/compiler/tests:pooling_ops_3d_test_cpu_mlir_bridge_test PASSED in 10.3s Stats over 20 runs: max = 10.3s, min = 7.0s, avg = 9.3s, dev = 0.7s //tensorflow/compiler/tests:pooling_ops_test_cpu PASSED in 19.1s Stats over 20 runs: max = 19.1s, min = 4.1s, avg = 7.6s, dev = 3.4s //tensorflow/compiler/tests:pooling_ops_test_cpu_mlir_bridge_test PASSED in 12.7s Stats over 20 runs: max = 12.7s, min = 4.9s, avg = 8.5s, dev = 2.0s //tensorflow/compiler/tests:stochastic_cast_op_test_cpu PASSED in 15.4s Stats over 20 runs: max = 15.4s, min = 10.3s, avg = 13.4s, dev = 1.6s //tensorflow/compiler/tests:unary_ops_test_cpu PASSED in 47.2s Stats over 20 runs: max = 47.2s, min = 19.5s, avg = 25.6s, dev = 7.4s //tensorflow/compiler/tests:unary_ops_test_cpu_mlir_bridge_test PASSED in 29.9s Stats over 20 runs: max = 29.9s, min = 6.0s, avg = 12.1s, dev = 6.6s //tensorflow/python/autograph/tests:loop_control_flow_test PASSED in 28.0s Stats over 20 runs: max = 28.0s, min = 13.3s, avg = 23.1s, dev = 4.1s //tensorflow/python/kernel_tests:metrics_test PASSED in 45.1s Stats over 20 runs: max = 45.1s, min = 10.6s, avg = 23.9s, dev = 9.1s //tensorflow/python/kernel_tests/array_ops:matrix_band_part_op_test_cpu PASSED in 10.6s Stats over 20 runs: max = 10.6s, min = 9.0s, avg = 9.9s, dev = 0.5s //tensorflow/python/kernel_tests/data_structures:barrier_ops_test PASSED in 18.6s Stats over 20 runs: max = 18.6s, min = 10.0s, avg = 12.9s, dev = 2.4s //tensorflow/python/kernel_tests/linalg:eig_op_test PASSED in 42.8s Stats over 20 runs: max = 42.8s, min = 4.9s, avg = 15.6s, dev = 12.7s //tensorflow/python/kernel_tests/linalg:linalg_grad_test_cpu PASSED in 82.7s Stats over 20 runs: max = 82.7s, min = 23.8s, avg = 43.4s, dev = 15.9s //tensorflow/python/kernel_tests/linalg:norm_op_test_cpu PASSED in 13.2s Stats over 20 runs: max = 13.2s, min = 8.5s, avg = 10.8s, dev = 1.3s //tensorflow/python/kernel_tests/linalg:normalize_op_test_cpu PASSED in 16.7s Stats over 20 runs: max = 16.7s, min = 6.1s, avg = 11.1s, dev = 2.5s //tensorflow/python/kernel_tests/linalg:qr_op_test_cpu PASSED in 193.5s Stats over 20 runs: max = 193.5s, min = 67.3s, avg = 127.6s, dev = 38.0s //tensorflow/python/kernel_tests/linalg:self_adjoint_eig_op_test_cpu PASSED in 22.6s Stats over 20 runs: max = 22.6s, min = 5.6s, avg = 11.7s, dev = 5.2s //tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu PASSED in 26.7s Stats over 20 runs: max = 26.7s, min = 10.3s, avg = 18.1s, dev = 6.2s //tensorflow/python/kernel_tests/math_ops:matmul_op_test_cpu PASSED in 20.1s Stats over 20 runs: max = 20.1s, min = 13.6s, avg = 16.6s, dev = 1.9s //tensorflow/python/kernel_tests/math_ops:tensordot_op_test_cpu PASSED in 59.3s Stats over 20 runs: max = 59.3s, min = 5.9s, avg = 26.2s, dev = 17.5s //tensorflow/python/kernel_tests/nn_ops:embedding_ops_test_cpu PASSED in 21.1s Stats over 20 runs: max = 21.1s, min = 12.3s, avg = 17.4s, dev = 2.3s //tensorflow/python/data/kernel_tests:interleave_test PASSED in 23.7s Stats over 24 runs: max = 23.7s, min = 8.3s, avg = 14.8s, dev = 3.8s //tensorflow/python/data/kernel_tests:sample_from_datasets_test PASSED in 19.4s Stats over 24 runs: max = 19.4s, min = 5.7s, avg = 11.5s, dev = 4.3s //tensorflow/dtensor/python/tests:multi_device_spmd_test_cpu PASSED in 38.7s Stats over 25 runs: max = 38.7s, min = 24.9s, avg = 33.0s, dev = 3.3s //tensorflow/python/kernel_tests/nn_ops:conv_ops_3d_test_cpu PASSED in 19.0s Stats over 30 runs: max = 19.0s, min = 4.5s, avg = 10.2s, dev = 3.8s //tensorflow/python/data/experimental/kernel_tests/service:data_service_ops_test PASSED in 23.1s Stats over 32 runs: max = 23.1s, min = 4.7s, avg = 10.6s, dev = 4.4s //tensorflow/python/data/experimental/kernel_tests/service:worker_tags_test PASSED in 19.8s Stats over 32 runs: max = 19.8s, min = 4.8s, avg = 12.3s, dev = 3.9s //tensorflow/python/kernel_tests/linalg:linear_operator_circulant_test_cpu PASSED in 42.7s Stats over 32 runs: max = 42.7s, min = 29.4s, avg = 35.2s, dev = 3.1s //tensorflow/core/kernels:stochastic_cast_op_test PASSED in 1.5s Stats over 48 runs: max = 1.5s, min = 0.4s, avg = 0.6s, dev = 0.3s //tensorflow/compiler/mlir/quantization/tensorflow/python:quantize_model_test PASSED in 57.2s Stats over 50 runs: max = 57.2s, min = 28.0s, avg = 40.2s, dev = 8.6s //tensorflow/compiler/tests:sort_ops_test_cpu PASSED in 18.2s Stats over 50 runs: max = 18.2s, min = 4.4s, avg = 10.5s, dev = 3.2s //tensorflow/compiler/tests:sort_ops_test_cpu_mlir_bridge_test PASSED in 22.3s Stats over 50 runs: max = 22.3s, min = 4.0s, avg = 11.8s, dev = 4.5s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_dense_mat_mul_grad_test_cpu PASSED in 16.1s Stats over 50 runs: max = 16.1s, min = 5.7s, avg = 9.4s, dev = 3.1s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_grad_test_cpu PASSED in 10.0s Stats over 50 runs: max = 10.0s, min = 4.1s, avg = 5.2s, dev = 1.5s //tensorflow/python/kernel_tests/linalg/sparse:csr_sparse_matrix_sparse_mat_mul_grad_test_cpu PASSED in 13.1s Stats over 50 runs: max = 13.1s, min = 4.4s, avg = 5.8s, dev = 1.8s //tensorflow/python/kernel_tests/math_ops:cwise_ops_binary_test_cpu PASSED in 35.0s Stats over 50 runs: max = 35.0s, min = 8.7s, avg = 19.3s, dev = 5.8s //tensorflow/python/kernel_tests/math_ops:cwise_ops_test_cpu PASSED in 14.5s Stats over 50 runs: max = 14.5s, min = 4.2s, avg = 5.5s, dev = 1.7s //tensorflow/python/kernel_tests/math_ops:cwise_ops_unary_test_cpu PASSED in 15.4s Stats over 50 runs: max = 15.4s, min = 4.2s, avg = 6.8s, dev = 2.3s
Executed 3074 out of 3074 tests: 3073 tests pass and 1 fails locally. There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are. stdout_lines: <omitted> ...ignoring
TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages] *** Monday 06 November 2023 21:13:46 +0000 (1:57:13.917) 3:03:41.334 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save list of installed Python packages] *** Monday 06 November 2023 21:13:48 +0000 (0:00:02.002) 3:03:43.336 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save tests output to file] ******** Monday 06 November 2023 21:13:48 +0000 (0:00:00.691) 3:03:44.028 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) save tests stderr output to file] *** Monday 06 November 2023 21:13:49 +0000 (0:00:00.737) 3:03:44.766 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Extract number of tests run] ****** Monday 06 November 2023 21:13:50 +0000 (0:00:00.700) 3:03:45.467 ******* ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Show number of tests run] ********* Monday 06 November 2023 21:13:50 +0000 (0:00:00.078) 3:03:45.546 ******* ok: [localhost] => ut_tests_line: 'Executed 3074 out of 3074 tests: 3073 tests pass and 1 fails locally.'
TASK [tensorflow_in_container : (Python 3.9) Extract failed tests] ************* Monday 06 November 2023 21:13:50 +0000 (0:00:00.042) 3:03:45.588 ******* ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) Show which tests are flaky] ******* Monday 06 November 2023 21:13:50 +0000 (0:00:00.129) 3:03:45.718 ******* ok: [localhost] => tests_flaky: []
TASK [tensorflow_in_container : (Python 3.9) Show which tests failed] ********** Monday 06 November 2023 21:13:50 +0000 (0:00:00.041) 3:03:45.760 ******* ok: [localhost] => tests_failed: - 'FAILED: //tensorflow/compiler/mlir/quantization/stablehlo:convert_tf_quant_to_mhlo_int_test (Summary)'
TASK [tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one] *** Monday 06 November 2023 21:13:50 +0000 (0:00:00.041) 3:03:45.801 ******* fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- + for wheel in /tf/pkg/*.whl + echo 'Checking and renaming /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl...' + python3 -m auditwheel repair --plat manylinux2014_aarch64 /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl --wheel-dir /tf/pkg + tee check.txt
real 2m28.629s user 2m13.970s sys 0m14.404s stderr_lines: <omitted> stdout: |- Checking and renaming /tf/pkg/tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl... INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.16.0-cp39-cp39-linux_aarch64.whl Traceback (most recent call last): File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.9/dist-packages/auditwheel/__main__.py", line 6, in <module> sys.exit(main()) File "/usr/local/lib/python3.9/dist-packages/auditwheel/main.py", line 59, in main rval = args.func(args, p) File "/usr/local/lib/python3.9/dist-packages/auditwheel/main_repair.py", line 173, in execute out_wheel = repair_wheel( File "/usr/local/lib/python3.9/dist-packages/auditwheel/repair.py", line 80, in repair_wheel raise ValueError( ValueError: Cannot repair wheel, because required library "_pywrap_tensorflow_internal.so" could not be located stdout_lines: <omitted>
PLAY RECAP ********************************************************************* localhost : ok=43 changed=29 unreachable=0 failed=1 skipped=8 rescued=0 ignored=1
Monday 06 November 2023 21:16:19 +0000 (0:02:29.442) 3:06:15.244 ******* =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow git tests ------ 7033.92s tensorflow_in_container : (Python 3.9) build TensorFlow git ---------- 3786.30s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one - 149.44s tensorflow_in_container : (Python 3.9) build Python package ----------- 123.47s Cleanup after previous build ------------------------------------------- 27.40s tensorflow : Fetch TensorFlow from git --------------------------------- 21.76s tensorflow : Remove TensorFlow git sources from previous build ---------- 4.06s tensorflow_in_container : (Python 3.9) start build container ------------ 3.78s Gathering Facts --------------------------------------------------------- 2.62s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 2.00s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 1.98s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.46s Create cache dirs ------------------------------------------------------- 1.43s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.22s tensorflow : Save git commit hash to file ------------------------------- 1.14s tensorflow_in_container : (Python 3.9) cleanup Bazel install directory --- 0.88s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.83s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.81s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.80s tensorflow_in_container : (Python 3.9) save tests output to file -------- 0.74s Build step 'Execute shell' marked build as failure