See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/666/display/redirect

Changes:

------------------------------------------ [...truncated 12.58 KB...] skipping: [localhost] => (item={'key': '1.15', 'value': {'tensorflow_version': '1.15.5', 'bazel_version': '0.29.1', 'flatbuffers_version': '~=1.12.0', 'gast_version': '==0.2.2', 'grpcio_version': '~=1.32.0', 'h5py_version': '==2.10', 'numpy_version': '~=1.18.0', 'absl_version': '~=0.10', 'scipy_version': '~=1.5.4', 'keras_version': '==1.2.2', 'tensorboard_version': '==1.15.0', 'tensorflow_estimator_version': '==1.15.2', 'python_install_packages': ['keras', 'tensorboard', 'tensorflow-estimator'], 'run_unit_tests': False, 'run_serial_unit_tests': False, 'onednn_ut_exclude': '', 'unit_tests_exclude': '', 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': '', 'python_versions': ['cp36-cp36m', 'cp37-cp37m']}})

TASK [tensorflow : Set TensorFlow and other package versions] ****************** Saturday 28 January 2023 18:10:09 +0000 (0:00:00.260) 0:00:04.660 ****** ok: [localhost]

TASK [tensorflow : Set build dir for TensorFlow git] *************************** Saturday 28 January 2023 18:10:09 +0000 (0:00:00.082) 0:00:04.743 ****** ok: [localhost]

TASK [tensorflow : Remove TensorFlow git sources from previous build] ********** Saturday 28 January 2023 18:10:09 +0000 (0:00:00.044) 0:00:04.788 ****** changed: [localhost]

TASK [tensorflow : Fetch TensorFlow] ******************************************* Saturday 28 January 2023 18:10:12 +0000 (0:00:03.687) 0:00:08.475 ****** included: https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/ws/configs/ldcg-python-manylinux-tensorflow/ansible/roles/tensorflow/tasks/fetch.yml for localhost

TASK [tensorflow : Fetch TensorFlow git source] ******************************** Saturday 28 January 2023 18:10:12 +0000 (0:00:00.098) 0:00:08.574 ****** skipping: [localhost]

TASK [tensorflow : Unpack TensorFlow git source] ******************************* Saturday 28 January 2023 18:10:12 +0000 (0:00:00.037) 0:00:08.612 ****** skipping: [localhost]

TASK [tensorflow : Rename source dir for oneDNN build] ************************* Saturday 28 January 2023 18:10:12 +0000 (0:00:00.036) 0:00:08.648 ****** skipping: [localhost]

TASK [tensorflow : Fetch TensorFlow from git] ********************************** Saturday 28 January 2023 18:10:13 +0000 (0:00:00.037) 0:00:08.686 ****** changed: [localhost]

TASK [tensorflow : Set world write access to source] *************************** Saturday 28 January 2023 18:10:51 +0000 (0:00:38.337) 0:00:47.023 ****** changed: [localhost]

TASK [tensorflow : Get oneline version of git log for this checkout] *********** Saturday 28 January 2023 18:10:51 +0000 (0:00:00.398) 0:00:47.422 ****** changed: [localhost]

TASK [tensorflow : Show git log] *********************************************** Saturday 28 January 2023 18:10:52 +0000 (0:00:00.561) 0:00:47.983 ****** ok: [localhost] => git_log.stdout: '0e89f1a141c3356c6da293411344ba92751b93e4 Merge pull request #59100 from lhutton1:tosa-atan2-tfl-legalization'

TASK [tensorflow : Record git commit hash to variable] ************************* Saturday 28 January 2023 18:10:52 +0000 (0:00:00.044) 0:00:48.028 ****** ok: [localhost]

TASK [tensorflow : Save git commit hash to file] ******************************* Saturday 28 January 2023 18:10:52 +0000 (0:00:00.065) 0:00:48.093 ****** changed: [localhost]

TASK [Build Bazel] ************************************************************* Saturday 28 January 2023 18:10:53 +0000 (0:00:01.108) 0:00:49.202 ****** skipping: [localhost]

TASK [Build TensorFlow for each Python version] ******************************** Saturday 28 January 2023 18:10:53 +0000 (0:00:00.035) 0:00:49.237 ****** skipping: [localhost] => (item=3.8) skipping: [localhost] => (item=3.9) skipping: [localhost] => (item=3.10)

TASK [Build TensorFlow for each Python version (dual abi)] ********************* Saturday 28 January 2023 18:10:53 +0000 (0:00:00.050) 0:00:49.288 ******

TASK [tensorflow_in_container : Reset checkout to clean state] ***************** Saturday 28 January 2023 18:10:54 +0000 (0:00:00.466) 0:00:49.755 ****** changed: [localhost]

TASK [tensorflow_in_container : Remove TensorFlow git sources from previous build] *** Saturday 28 January 2023 18:10:54 +0000 (0:00:00.653) 0:00:50.408 ****** skipping: [localhost]

TASK [Fetch TensorFlow] ******************************************************** Saturday 28 January 2023 18:10:54 +0000 (0:00:00.036) 0:00:50.445 ****** skipping: [localhost]

TASK [(Python 3.8) patch TensorFlow tree] ************************************** Saturday 28 January 2023 18:10:54 +0000 (0:00:00.037) 0:00:50.483 ******

TASK [tensorflow : Create linklibs shell script] ******************************* Saturday 28 January 2023 18:10:54 +0000 (0:00:00.142) 0:00:50.625 ****** changed: [localhost]

TASK [tensorflow : Create readme file about unit tests] ************************ Saturday 28 January 2023 18:10:55 +0000 (0:00:00.690) 0:00:51.316 ****** changed: [localhost]

TASK [tensorflow : Add to the copied files] ************************************ Saturday 28 January 2023 18:10:56 +0000 (0:00:00.696) 0:00:52.013 ****** changed: [localhost]

TASK [tensorflow : Add to the long_description in setup.py] ******************** Saturday 28 January 2023 18:10:56 +0000 (0:00:00.587) 0:00:52.600 ****** changed: [localhost]

TASK [tensorflow : Use nightly package requirements] *************************** Saturday 28 January 2023 18:10:57 +0000 (0:00:00.553) 0:00:53.154 ****** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) start build container] ************ Saturday 28 January 2023 18:10:57 +0000 (0:00:00.389) 0:00:53.544 ****** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) create buildslave bazel cache dir] *** Saturday 28 January 2023 18:11:01 +0000 (0:00:03.561) 0:00:57.105 ****** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) create link for bazel cache] ****** Saturday 28 January 2023 18:11:02 +0000 (0:00:01.172) 0:00:58.277 ****** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) create link for bazelisk cache] *** Saturday 28 January 2023 18:11:03 +0000 (0:00:00.817) 0:00:59.095 ****** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) create link for pip cache] ******** Saturday 28 January 2023 18:11:04 +0000 (0:00:00.853) 0:00:59.949 ****** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) copy upper constraints] *********** Saturday 28 January 2023 18:11:05 +0000 (0:00:00.815) 0:01:00.764 ****** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) install Python packages] ********** Saturday 28 January 2023 18:11:05 +0000 (0:00:00.718) 0:01:01.482 ****** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) build TensorFlow git] ************* Saturday 28 January 2023 18:11:17 +0000 (0:00:11.638) 0:01:13.120 ****** fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- Extracting Bazel installation... Starting local Bazel server and connecting to it... INFO: Invocation ID: 8ee2f1d7-d9b3-4705-bfc4-407c38629753 INFO: Options provided by the client: Inherited 'common' options: --isatty=0 --terminal_columns=80 INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc: Inherited 'common' options: --experimental_repo_remote_exec INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc: 'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils INFO: Reading rc options for 'build' from /usertools/aarch64.bazelrc: 'build' options: --action_env=CACHEBUSTER=20220325 --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/tf_python --python_path=/usr/bin/python3 --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 --distinct_host_configuration=false --profile=/tf/pkg/profile.json.gz --crosstool_top=@ml2014_aarch64_config_aarch64//crosstool:toolchain --copt=-mtune=generic --copt=-march=armv8-a --copt=-O3 INFO: Found applicable config definition build:short_logs in file /tf/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING INFO: Found applicable config definition build:v2 in file /tf/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 INFO: Found applicable config definition build:sigbuild_local_cache in file /usertools/aarch64.bazelrc: --disk_cache=/tf/cache INFO: Found applicable config definition build:ambe in file /usertools/aarch64.bazelrc: --distinct_host_configuration=false --flaky_test_attempts=3 --action_env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin --host_crosstool_top=@ml2014_aarch64_config_aarch64//crosstool:toolchain --crosstool_top=@ml2014_aarch64_config_aarch64//crosstool:toolchain --extra_toolchains=@ml2014_aarch64_config_aarch64//crosstool:toolchain-linux-aarch64 --extra_execution_platforms=@ml2014_aarch64_config_aarch64_platform//:platform --host_platform=@ml2014_aarch64_config_aarch64_platform//:platform --platforms=@ml2014_aarch64_config_aarch64_platform//:platform --repo_env=TF_PYTHON_CONFIG_REPO=@ml2014_aarch64_config_python INFO: Found applicable config definition build:linux in file /tf/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes INFO: Found applicable config definition build:dynamic_kernels in file /tf/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS Loading: Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/r... failed: class java.io.FileNotFoundException GET returned 404 Not Found Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/protocolbuff... failed: class java.io.FileNotFoundException GET returned 404 Not Found Analyzing: target //tensorflow/tools/pip_package:build_pip_package (1 packages loaded, 0 targets configured) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (285 packages loaded, 4232 targets configured) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (582 packages loaded, 30334 targets configured) WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/borin... failed: class java.io.FileNotFoundException GET returned 404 Not Found INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (584 packages loaded, 33521 targets configured). INFO: Found 1 target... [0 / 72] [Prepa] BazelWorkspaceStatusAction stable-status.txt ... (15 actions, 0 running) [0 / 109] [Prepa] BazelWorkspaceStatusAction stable-status.txt ... (31 actions, 0 running) [3,946 / 16,022] Compiling src/google/protobuf/util/internal/datapiece.cc; 5s local, remote-cache ... (109 actions, 106 running) [5,264 / 16,022] Compiling src/google/protobuf/descriptor.cc; 14s local, remote-cache ... (111 actions, 110 running) [6,097 / 16,022] Compiling src/google/protobuf/descriptor.cc; 24s local, remote-cache ... (111 actions, 110 running) [6,611 / 16,022] Compiling tensorflow/compiler/xla/service/cpu/runtime_conv3d.cc; 30s local, remote-cache ... (111 actions, 110 running) [6,685 / 16,022] Compiling tensorflow/compiler/xla/service/cpu/runtime_conv3d.cc; 43s local, remote-cache ... (111 actions, 110 running) [10,220 / 19,103] Compiling tensorflow/compiler/xla/service/cpu/runtime_conv2d.cc; 53s local, remote-cache ... (105 actions, 100 running) [11,531 / 21,387] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 71s local, remote-cache ... (111 actions, 110 running) [15,224 / 22,540] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 93s local, remote-cache ... (111 actions, 110 running) [15,568 / 22,839] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 117s local, remote-cache ... (111 actions, 110 running) [16,149 / 22,839] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 145s local, remote-cache ... (111 actions, 110 running) [16,545 / 22,839] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 177s local, remote-cache ... (109 actions, 108 running) [16,786 / 22,839] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 213s local, remote-cache ... (112 actions running) [17,063 / 22,839] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 255s local, remote-cache ... (112 actions running) [17,488 / 22,839] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 303s local, remote-cache ... (112 actions running) [17,746 / 22,839] Compiling tensorflow/compiler/xla/service/memory_space_assignment.cc; 45s local, remote-cache ... (112 actions running) [18,040 / 22,839] Compiling tensorflow/compiler/xla/service/algebraic_simplifier.cc; 51s local, remote-cache ... (112 actions running) [18,565 / 22,839] Compiling tensorflow/core/util/batch_util.cc; 20s local, remote-cache ... (112 actions running) [19,142 / 22,839] Compiling tensorflow/core/util/batch_util.cc; 104s local, remote-cache ... (112 actions running) [19,785 / 22,839] Compiling tensorflow/core/kernels/training_ops.cc; 103s local, remote-cache ... (112 actions running) [20,312 / 22,839] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops_n_z.cc; 138s local, remote-cache ... (111 actions, 110 running) [20,756 / 22,839] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops_n_z.cc; 266s local, remote-cache ... (112 actions running) [20,984 / 22,839] Compiling tensorflow/core/kernels/linalg/matrix_square_root_op.cc; 177s local, remote-cache ... (112 actions running) [21,339 / 22,839] Compiling tensorflow/core/kernels/argmax_op.cc; 245s local, remote-cache ... (112 actions running) [21,713 / 22,839] Compiling tensorflow/core/kernels/matmul_op_real.cc; 434s local, remote-cache ... (112 actions running) [22,973 / 23,066] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc; 210s local, remote-cache ... (36 actions running) [22,997 / 23,066] Compiling tensorflow/core/kernels/conv_ops.cc; 418s local, remote-cache [22,999 / 23,066] Linking tensorflow/libtensorflow_cc.so.2.12.0; 235s local, remote-cache [22,999 / 23,066] Linking tensorflow/libtensorflow_cc.so.2.12.0; 576s local, remote-cache ERROR: /tf/tensorflow/tensorflow/BUILD:1611:19: Executing genrule //tensorflow:tf_python_api_gen_v2 failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped) 2023-01-28 18:55:44.903975: E tensorflow/tsl/lib/monitoring/collection_registry.cc:81] Cannot register 2 metrics with the same name: /tensorflow/core/bfc_allocator_delay Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 22, in <module> from tensorflow.python.tools.api.generator import doc_srcs File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 37, in <module> from tensorflow.python.eager import context File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 28, in <module> from tensorflow.core.framework import function_pb2 File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/core/framework/function_pb2.py", line 5, in <module> from google.protobuf.internal import builder as _builder ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/home/buildslave/.local/lib/python3.8/site-packages/google/protobuf/internal/__init__.py) Target //tensorflow/tools/pip_package:build_pip_package failed to build Use --verbose_failures to see the command lines of failed build steps. ERROR: /tf/tensorflow/tensorflow/lite/python/BUILD:69:10 Middleman _middlemen/tensorflow_Slite_Spython_Stflite_Uconvert-runfiles failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped) INFO: Elapsed time: 2667.176s, Critical Path: 1132.61s INFO: 23065 processes: 8992 disk cache hit, 5474 internal, 8599 local. FAILED: Build did NOT complete successfully FAILED: Build did NOT complete successfully stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>

PLAY RECAP ********************************************************************* localhost : ok=27 changed=20 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0

Saturday 28 January 2023 18:55:45 +0000 (0:44:28.343) 0:45:41.463 ****** =============================================================================== tensorflow_in_container : (Python 3.8) build TensorFlow git ---------- 2668.34s tensorflow : Fetch TensorFlow from git --------------------------------- 38.34s tensorflow_in_container : (Python 3.8) install Python packages --------- 11.64s tensorflow : Remove TensorFlow git sources from previous build ---------- 3.69s tensorflow_in_container : (Python 3.8) start build container ------------ 3.56s Gathering Facts --------------------------------------------------------- 1.64s Create cache dirs ------------------------------------------------------- 1.38s Cleanup after previous build -------------------------------------------- 1.29s tensorflow_in_container : (Python 3.8) create buildslave bazel cache dir --- 1.17s tensorflow : Save git commit hash to file ------------------------------- 1.11s tensorflow_in_container : (Python 3.8) create link for bazelisk cache --- 0.85s tensorflow_in_container : (Python 3.8) create link for bazel cache ------ 0.82s tensorflow_in_container : (Python 3.8) create link for pip cache -------- 0.82s tensorflow_in_container : (Python 3.8) copy upper constraints ----------- 0.72s tensorflow : Create readme file about unit tests ------------------------ 0.70s tensorflow : Create linklibs shell script ------------------------------- 0.69s tensorflow_in_container : Reset checkout to clean state ----------------- 0.65s tensorflow : Add to the copied files ------------------------------------ 0.59s tensorflow : Get oneline version of git log for this checkout ----------- 0.56s tensorflow : Add to the long_description in setup.py -------------------- 0.55s Build step 'Execute shell' marked build as failure