See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/323/display/redirect

Changes:

------------------------------------------ [...truncated 14.46 KB...] Sunday 29 January 2023 01:56:09 +0000 (0:00:00.081) 0:00:04.721 ******** ok: [localhost]

TASK [tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build] *** Sunday 29 January 2023 01:56:09 +0000 (0:00:00.044) 0:00:04.765 ******** changed: [localhost]

TASK [tensorflow : Fetch TensorFlow] ******************************************* Sunday 29 January 2023 01:56:12 +0000 (0:00:03.401) 0:00:08.167 ******** included: https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/ws/configs/ldcg-python-manylinux-tensorflow/ansible/roles/tensorflow/tasks/fetch.yml for localhost

TASK [tensorflow : Fetch TensorFlow-oneDNN-ACL git source] ********************* Sunday 29 January 2023 01:56:12 +0000 (0:00:00.098) 0:00:08.265 ******** skipping: [localhost]

TASK [tensorflow : Unpack TensorFlow-oneDNN-ACL git source] ******************** Sunday 29 January 2023 01:56:12 +0000 (0:00:00.038) 0:00:08.304 ******** skipping: [localhost]

TASK [tensorflow : Rename source dir for oneDNN build] ************************* Sunday 29 January 2023 01:56:12 +0000 (0:00:00.036) 0:00:08.340 ******** skipping: [localhost]

TASK [tensorflow : Fetch TensorFlow-oneDNN-ACL from git] *********************** Sunday 29 January 2023 01:56:12 +0000 (0:00:00.037) 0:00:08.378 ******** changed: [localhost]

TASK [tensorflow : Set world write access to source] *************************** Sunday 29 January 2023 01:56:33 +0000 (0:00:20.491) 0:00:28.869 ******** changed: [localhost]

TASK [tensorflow : Get oneline version of git log for this checkout] *********** Sunday 29 January 2023 01:56:33 +0000 (0:00:00.372) 0:00:29.242 ******** changed: [localhost]

TASK [tensorflow : Show git log] *********************************************** Sunday 29 January 2023 01:56:34 +0000 (0:00:00.584) 0:00:29.827 ******** ok: [localhost] => git_log.stdout: 7a7399797b003936cc672408120fe62f3b493b12 Update TFRT dependency to use revision http://github.com/tensorflow/runtime/commit/d1c45099b1bc7db1cf4c27772d0b4874....

TASK [tensorflow : Record git commit hash to variable] ************************* Sunday 29 January 2023 01:56:34 +0000 (0:00:00.041) 0:00:29.868 ******** ok: [localhost]

TASK [tensorflow : Save git commit hash to file] ******************************* Sunday 29 January 2023 01:56:34 +0000 (0:00:00.084) 0:00:29.953 ******** changed: [localhost]

TASK [Build Bazel] ************************************************************* Sunday 29 January 2023 01:56:35 +0000 (0:00:01.112) 0:00:31.066 ******** skipping: [localhost]

TASK [Build TensorFlow for each Python version] ******************************** Sunday 29 January 2023 01:56:35 +0000 (0:00:00.036) 0:00:31.102 ******** skipping: [localhost] => (item=3.8) skipping: [localhost] => (item=3.9) skipping: [localhost] => (item=3.10)

TASK [Build TensorFlow for each Python version (dual abi)] ********************* Sunday 29 January 2023 01:56:35 +0000 (0:00:00.049) 0:00:31.151 ********

TASK [tensorflow_in_container : Reset checkout to clean state] ***************** Sunday 29 January 2023 01:56:36 +0000 (0:00:00.455) 0:00:31.607 ******** changed: [localhost]

TASK [tensorflow_in_container : Remove TensorFlow-oneDNN-ACL git sources from previous build] *** Sunday 29 January 2023 01:56:36 +0000 (0:00:00.626) 0:00:32.233 ******** skipping: [localhost]

TASK [Fetch TensorFlow] ******************************************************** Sunday 29 January 2023 01:56:36 +0000 (0:00:00.035) 0:00:32.269 ******** skipping: [localhost]

TASK [(Python 3.8) patch TensorFlow tree] ************************************** Sunday 29 January 2023 01:56:36 +0000 (0:00:00.036) 0:00:32.305 ********

TASK [tensorflow : Create linklibs shell script] ******************************* Sunday 29 January 2023 01:56:36 +0000 (0:00:00.161) 0:00:32.467 ******** changed: [localhost]

TASK [tensorflow : Create readme file about unit tests] ************************ Sunday 29 January 2023 01:56:37 +0000 (0:00:00.692) 0:00:33.159 ******** changed: [localhost]

TASK [tensorflow : Add to the copied files] ************************************ Sunday 29 January 2023 01:56:38 +0000 (0:00:00.700) 0:00:33.860 ******** changed: [localhost]

TASK [tensorflow : Add to the long_description in setup.py] ******************** Sunday 29 January 2023 01:56:38 +0000 (0:00:00.592) 0:00:34.452 ******** changed: [localhost]

TASK [tensorflow : Use nightly package requirements] *************************** Sunday 29 January 2023 01:56:39 +0000 (0:00:00.584) 0:00:35.037 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) start build container] ************ Sunday 29 January 2023 01:56:39 +0000 (0:00:00.389) 0:00:35.427 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) create buildslave bazel cache dir] *** Sunday 29 January 2023 01:57:13 +0000 (0:00:34.028) 0:01:09.455 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) create link for bazel cache] ****** Sunday 29 January 2023 01:57:15 +0000 (0:00:01.169) 0:01:10.625 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) create link for bazelisk cache] *** Sunday 29 January 2023 01:57:15 +0000 (0:00:00.843) 0:01:11.468 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) create link for pip cache] ******** Sunday 29 January 2023 01:57:16 +0000 (0:00:00.829) 0:01:12.298 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) copy upper constraints] *********** Sunday 29 January 2023 01:57:17 +0000 (0:00:00.819) 0:01:13.118 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) install Python packages] ********** Sunday 29 January 2023 01:57:18 +0000 (0:00:00.717) 0:01:13.835 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.8) build TensorFlow-oneDNN-ACL git] *** Sunday 29 January 2023 01:57:30 +0000 (0:00:11.868) 0:01:25.704 ******** fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- Extracting Bazel installation... Starting local Bazel server and connecting to it... INFO: Invocation ID: 84d238ae-feb5-4856-8840-c7dd78ce031f INFO: Options provided by the client: Inherited 'common' options: --isatty=0 --terminal_columns=80 INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc: Inherited 'common' options: --experimental_repo_remote_exec INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc: 'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils INFO: Reading rc options for 'build' from /usertools/aarch64.bazelrc: 'build' options: --action_env=CACHEBUSTER=20220325 --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/tf_python --python_path=/usr/bin/python3 --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 --distinct_host_configuration=false --profile=/tf/pkg/profile.json.gz --crosstool_top=@ml2014_aarch64_config_aarch64//crosstool:toolchain --copt=-mtune=generic --copt=-march=armv8-a --copt=-O3 INFO: Found applicable config definition build:short_logs in file /tf/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING INFO: Found applicable config definition build:v2 in file /tf/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 INFO: Found applicable config definition build:sigbuild_local_cache in file /usertools/aarch64.bazelrc: --disk_cache=/tf/cache INFO: Found applicable config definition build:ambe in file /usertools/aarch64.bazelrc: --distinct_host_configuration=false --flaky_test_attempts=3 --action_env=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin --host_crosstool_top=@ml2014_aarch64_config_aarch64//crosstool:toolchain --crosstool_top=@ml2014_aarch64_config_aarch64//crosstool:toolchain --extra_toolchains=@ml2014_aarch64_config_aarch64//crosstool:toolchain-linux-aarch64 --extra_execution_platforms=@ml2014_aarch64_config_aarch64_platform//:platform --host_platform=@ml2014_aarch64_config_aarch64_platform//:platform --platforms=@ml2014_aarch64_config_aarch64_platform//:platform --repo_env=TF_PYTHON_CONFIG_REPO=@ml2014_aarch64_config_python INFO: Found applicable config definition build:mkl_aarch64_threadpool in file /tf/tensorflow/.bazelrc: --define=build_with_mkl_aarch64=true -c opt INFO: Found applicable config definition build:linux in file /tf/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes INFO: Found applicable config definition build:dynamic_kernels in file /tf/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS Loading: Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/r... failed: class java.io.FileNotFoundException GET returned 404 Not Found Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded Loading: 0 packages loaded WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-pr... failed: class java.io.FileNotFoundException GET returned 404 Not Found Loading: 0 packages loaded WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/protocolbuff... failed: class java.io.FileNotFoundException GET returned 404 Not Found Analyzing: target //tensorflow/tools/pip_package:build_pip_package (1 packages loaded, 0 targets configured) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (287 packages loaded, 4232 targets configured) WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/borin... failed: class java.io.FileNotFoundException GET returned 404 Not Found INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (587 packages loaded, 36763 targets configured). INFO: Found 1 target... [0 / 408] [Prepa] BazelWorkspaceStatusAction stable-status.txt ... (40 actions, 0 running) [186 / 1,486] Compiling src/google/protobuf/compiler/command_line_interface.cc; 10s local, remote-cache ... (111 actions, 110 running) [1,098 / 4,214] Compiling src/google/protobuf/descriptor.cc; 21s local, remote-cache ... (111 actions, 110 running) [2,516 / 8,161] Compiling src/google/protobuf/descriptor.cc; 20s local, remote-cache ... (111 actions, 110 running) [2,925 / 8,163] Compiling tensorflow/compiler/xla/xla_data.pb.cc; 15s local, remote-cache ... (111 actions, 110 running) [7,840 / 18,874] Compiling tensorflow/compiler/xla/debug_options_flags.cc; 15s local, remote-cache ... (109 actions, 106 running) [9,277 / 20,749] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_fft.cc; 28s local, remote-cache ... (109 actions, 108 running) [11,568 / 24,050] Compiling tensorflow/compiler/xla/service/cpu/runtime_conv2d.cc; 51s local, remote-cache ... (111 actions, 110 running) [12,098 / 24,050] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 74s local, remote-cache ... (111 actions running) [12,797 / 24,050] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 106s local, remote-cache ... (111 actions, 110 running) [13,228 / 24,050] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 142s local, remote-cache ... (111 actions, 110 running) [15,018 / 24,050] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 184s local, remote-cache ... (111 actions, 110 running) [15,522 / 24,050] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 232s local, remote-cache ... (111 actions, 110 running) [15,866 / 24,050] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 288s local, remote-cache ... (111 actions, 110 running) [16,244 / 24,050] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 351s local, remote-cache ... (111 actions, 110 running) [16,624 / 24,050] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops_a_m.cc; 148s local, remote-cache ... (111 actions, 110 running) [16,981 / 24,050] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops_a_m.cc; 233s local, remote-cache ... (111 actions, 110 running) [17,408 / 24,050] Compiling llvm/lib/Transforms/IPO/AttributorAttributes.cpp; 52s local, remote-cache ... (111 actions, 110 running) [17,911 / 24,050] Compiling llvm/lib/Passes/PassBuilder.cpp; 113s local, remote-cache ... (111 actions, 110 running) [18,712 / 24,050] Compiling llvm/lib/Target/X86/X86ISelLowering.cpp; 107s local, remote-cache ... (111 actions, 110 running) [19,511 / 24,050] Compiling tensorflow/core/kernels/training_ops.cc; 178s local, remote-cache ... (109 actions, 108 running) [20,124 / 24,050] Compiling tensorflow/compiler/mlir/lite/transforms/optimize.cc; 166s local, remote-cache ... (111 actions, 110 running) [20,458 / 24,050] Compiling tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc; 319s local, remote-cache ... (111 actions, 110 running) [21,328 / 24,050] Compiling tensorflow/core/kernels/matmul_op_real.cc; 438s local, remote-cache ... (111 actions, 110 running) [22,785 / 24,050] Compiling tensorflow/core/kernels/conv_ops.cc; 486s local, remote-cache ... (111 actions, 110 running) [23,707 / 24,050] Compiling tensorflow/compiler/mlir/xla/transforms/legalize_tf_with_tf2xla.cc; 45s local, remote-cache ... (111 actions, 110 running) [24,092 / 24,159] Linking tensorflow/libtensorflow_cc.so.2.12.0; 216s local, remote-cache ERROR: /tf/tensorflow/tensorflow/BUILD:1611:19: Executing genrule //tensorflow:tf_python_api_gen_v2 failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped) 2023-01-29 02:46:00.778469: E tensorflow/tsl/lib/monitoring/collection_registry.cc:81] Cannot register 2 metrics with the same name: /tensorflow/core/bfc_allocator_delay Traceback (most recent call last): File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/tools/api/generator/create_python_api.py", line 22, in <module> from tensorflow.python.tools.api.generator import doc_srcs File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/__init__.py", line 37, in <module> from tensorflow.python.eager import context File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/python/eager/context.py", line 28, in <module> from tensorflow.core.framework import function_pb2 File "/home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/execroot/org_tensorflow/bazel-out/aarch64-opt/bin/tensorflow/create_tensorflow.python_api_tf_python_api_gen_v2.runfiles/org_tensorflow/tensorflow/core/framework/function_pb2.py", line 5, in <module> from google.protobuf.internal import builder as _builder ImportError: cannot import name 'builder' from 'google.protobuf.internal' (/home/buildslave/.local/lib/python3.8/site-packages/google/protobuf/internal/__init__.py) Target //tensorflow/tools/pip_package:build_pip_package failed to build Use --verbose_failures to see the command lines of failed build steps. ERROR: /tf/tensorflow/tensorflow/tools/pip_package/BUILD:276:10 Middleman _middlemen/tensorflow_Stools_Spip_Upackage_Sbuild_Upip_Upackage-runfiles failed: (Exit 1): bash failed: error executing command /bin/bash -c ... (remaining 1 argument skipped) INFO: Elapsed time: 2911.454s, Critical Path: 1127.43s INFO: 24158 processes: 5817 disk cache hit, 5530 internal, 12811 local. FAILED: Build did NOT complete successfully FAILED: Build did NOT complete successfully stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>

PLAY RECAP ********************************************************************* localhost : ok=27 changed=20 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0

Sunday 29 January 2023 02:46:02 +0000 (0:48:32.660) 0:49:58.365 ******** =============================================================================== tensorflow_in_container : (Python 3.8) build TensorFlow-oneDNN-ACL git 2912.66s tensorflow_in_container : (Python 3.8) start build container ----------- 34.03s tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 20.49s tensorflow_in_container : (Python 3.8) install Python packages --------- 11.87s tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 3.40s Gathering Facts --------------------------------------------------------- 1.62s Create cache dirs ------------------------------------------------------- 1.38s Cleanup after previous build -------------------------------------------- 1.29s tensorflow_in_container : (Python 3.8) create buildslave bazel cache dir --- 1.17s tensorflow : Save git commit hash to file ------------------------------- 1.11s tensorflow_in_container : (Python 3.8) create link for bazel cache ------ 0.84s tensorflow_in_container : (Python 3.8) create link for bazelisk cache --- 0.83s tensorflow_in_container : (Python 3.8) create link for pip cache -------- 0.82s tensorflow_in_container : (Python 3.8) copy upper constraints ----------- 0.72s tensorflow : Create readme file about unit tests ------------------------ 0.70s tensorflow : Create linklibs shell script ------------------------------- 0.69s tensorflow_in_container : Reset checkout to clean state ----------------- 0.63s tensorflow : Add to the copied files ------------------------------------ 0.59s tensorflow : Add to the long_description in setup.py -------------------- 0.58s tensorflow : Get oneline version of git log for this checkout ----------- 0.58s Build step 'Execute shell' marked build as failure