See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/1122/display/redirect
Changes:
------------------------------------------ [...truncated 2.44 KiB...] ok: [localhost] => (item=bazel-cache) changed: [localhost] => (item=wheels) changed: [localhost] => (item=ml_wheels)
TASK [Go through each TensorFlow version] ************************************** Wednesday 27 March 2024 18:10:21 +0000 (0:00:01.396) 0:00:16.420 ******* skipping: [localhost] => (item={'key': '2.16', 'value': {'tensorflow_version': '2.16.1', 'bazel_version': 'bazelisk', 'old_pip_package': True, 'image_prefix': '2.16', 'run_unit_tests': True, 'run_serial_unit_tests': True, 'onednn_ut_exclude': '', 'unit_tests_exclude': ['//tensorflow/lite/...'], 'unit_tests_gcc_exclude': ['//tensorflow/compiler/mlir/lite/debug:debug_test', '//tensorflow/compiler/mlir/quantization/tensorflow/debugging:mlir_dump_test'], 'unit_tests_clang_exclude': ['//tensorflow/core/kernels/image:resize_bicubic_op_test'], 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': 'NOT_USED', 'python_versions': ['3.9', '3.10', '3.11', '3.12']}}) skipping: [localhost] => (item={'key': '2.15', 'value': {'tensorflow_version': '2.15.1', 'bazel_version': 'bazelisk', 'old_pip_package': True, 'image_prefix': '2.15', 'run_unit_tests': True, 'run_serial_unit_tests': True, 'onednn_ut_exclude': ['//tensorflow/core/grappler/optimizers:auto_mixed_precision_test', '//tensorflow/core/grappler/optimizers:remapper_test'], 'unit_tests_exclude': ['//tensorflow/lite/...', '//tensorflow/tools/api/tests:api_compatibility_test'], 'unit_tests_gcc_exclude': ['//tensorflow/compiler/mlir/lite/debug:debug_test'], 'unit_tests_clang_exclude': ['//tensorflow/compiler/mlir/lite/quantization/lite:quantize_model_test', '//tensorflow/compiler/mlir/lite/quantization/lite:quantize_weights_test', '//tensorflow/compiler/mlir/lite/sparsity:sparsify_model_test', '//tensorflow/core/kernels/image:resize_bicubic_op_test'], 'serial_unit_tests_exclude': '', 'unit_tests_filter_flags': 'NOT_USED', 'python_versions': ['3.9', '3.10', '3.11']}})
TASK [tensorflow : Set TensorFlow and other package versions] ****************** Wednesday 27 March 2024 18:10:21 +0000 (0:00:00.114) 0:00:16.535 ******* ok: [localhost]
TASK [tensorflow : Set build dir for TensorFlow git] *************************** Wednesday 27 March 2024 18:10:21 +0000 (0:00:00.069) 0:00:16.605 ******* ok: [localhost]
TASK [tensorflow : Remove TensorFlow git sources from previous build] ********** Wednesday 27 March 2024 18:10:21 +0000 (0:00:00.037) 0:00:16.642 ******* changed: [localhost]
TASK [tensorflow : Fetch TensorFlow] ******************************************* Wednesday 27 March 2024 18:10:25 +0000 (0:00:04.110) 0:00:20.753 ******* included: https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/ws/configs/ldcg-python-manylinux-tensorflow/ansible/roles/tensorflow/tasks/fetch.yml for localhost
TASK [tensorflow : Fetch TensorFlow git source] ******************************** Wednesday 27 March 2024 18:10:25 +0000 (0:00:00.078) 0:00:20.831 ******* skipping: [localhost]
TASK [tensorflow : Unpack TensorFlow git source] ******************************* Wednesday 27 March 2024 18:10:25 +0000 (0:00:00.032) 0:00:20.864 ******* skipping: [localhost]
TASK [tensorflow : Rename source dir for oneDNN build] ************************* Wednesday 27 March 2024 18:10:25 +0000 (0:00:00.029) 0:00:20.893 ******* skipping: [localhost]
TASK [tensorflow : Fetch TensorFlow from git] ********************************** Wednesday 27 March 2024 18:10:25 +0000 (0:00:00.032) 0:00:20.926 ******* changed: [localhost]
TASK [tensorflow : Fetch TensorFlow from git at commit] ************************ Wednesday 27 March 2024 18:10:47 +0000 (0:00:21.388) 0:00:42.315 ******* skipping: [localhost]
TASK [tensorflow : Set world write access to source] *************************** Wednesday 27 March 2024 18:10:47 +0000 (0:00:00.029) 0:00:42.344 ******* changed: [localhost]
TASK [tensorflow : Get oneline version of git log for this checkout] *********** Wednesday 27 March 2024 18:10:47 +0000 (0:00:00.366) 0:00:42.710 ******* changed: [localhost]
TASK [tensorflow : Show git log] *********************************************** Wednesday 27 March 2024 18:10:48 +0000 (0:00:00.577) 0:00:43.288 ******* ok: [localhost] => git_log.stdout: 'aa6da142f36c70da2ee324e98536876ed05f0965 Merged commit includes the following changes: 619575611 by A. Unique TensorFlowergardener@tensorflow.org:'
TASK [tensorflow : Record git commit hash to variable] ************************* Wednesday 27 March 2024 18:10:48 +0000 (0:00:00.035) 0:00:43.324 ******* ok: [localhost]
TASK [tensorflow : Save git commit hash to file] ******************************* Wednesday 27 March 2024 18:10:48 +0000 (0:00:00.055) 0:00:43.379 ******* changed: [localhost]
TASK [Build TensorFlow for each Python version (dual abi)] ********************* Wednesday 27 March 2024 18:10:49 +0000 (0:00:01.136) 0:00:44.516 *******
TASK [tensorflow_in_container : Reset checkout to clean state] ***************** Wednesday 27 March 2024 18:10:49 +0000 (0:00:00.436) 0:00:44.952 ******* changed: [localhost]
TASK [tensorflow_in_container : Remove TensorFlow git sources from previous build] *** Wednesday 27 March 2024 18:10:50 +0000 (0:00:00.640) 0:00:45.593 ******* skipping: [localhost]
TASK [Fetch TensorFlow] ******************************************************** Wednesday 27 March 2024 18:10:50 +0000 (0:00:00.029) 0:00:45.623 ******* skipping: [localhost]
TASK [(Python 3.9) patch TensorFlow tree] ************************************** Wednesday 27 March 2024 18:10:50 +0000 (0:00:00.030) 0:00:45.653 *******
TASK [tensorflow : Create linklibs shell script] ******************************* Wednesday 27 March 2024 18:10:50 +0000 (0:00:00.081) 0:00:45.735 ******* changed: [localhost]
TASK [tensorflow : Use nightly package requirements] *************************** Wednesday 27 March 2024 18:10:51 +0000 (0:00:00.687) 0:00:46.423 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) copy upper constraints for hermetic builds] *** Wednesday 27 March 2024 18:10:52 +0000 (0:00:00.620) 0:00:47.044 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) build Python versioned container] *** Wednesday 27 March 2024 18:10:52 +0000 (0:00:00.680) 0:00:47.724 ******* ok: [localhost]
TASK [tensorflow_in_container : (Python 3.9) start build container] ************ Wednesday 27 March 2024 18:10:54 +0000 (0:00:01.487) 0:00:49.211 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir] *** Wednesday 27 March 2024 18:10:58 +0000 (0:00:03.826) 0:00:53.038 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) create link for bazel cache] ****** Wednesday 27 March 2024 18:10:59 +0000 (0:00:01.214) 0:00:54.252 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) create link for bazelisk cache] *** Wednesday 27 March 2024 18:11:00 +0000 (0:00:00.846) 0:00:55.098 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) create link for pip cache] ******** Wednesday 27 March 2024 18:11:00 +0000 (0:00:00.817) 0:00:55.915 ******* changed: [localhost]
TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build] *** Wednesday 27 March 2024 18:11:01 +0000 (0:00:00.815) 0:00:56.730 ******* changed: [localhost]
TASK [tensorflow_in_container : Show list of installed Python packages for build] *** Wednesday 27 March 2024 18:11:03 +0000 (0:00:01.943) 0:00:58.674 ******* ok: [localhost] => pip_list_output.stdout: |- Package Version Location Installer ------------------- -------------------- -------------------------------------- --------- auditwheel 5.4.0 /usr/local/lib/python3.9/dist-packages pip certifi 2019.11.28 /usr/lib/python3/dist-packages cffi 1.16.0 /usr/local/lib/python3.9/dist-packages pip chardet 3.0.4 /usr/lib/python3/dist-packages cryptography 42.0.5 /usr/local/lib/python3.9/dist-packages pip Cython 3.0.9 /usr/local/lib/python3.9/dist-packages pip dbus-python 1.2.16 /usr/lib/python3/dist-packages docutils 0.20.1 /usr/local/lib/python3.9/dist-packages pip idna 2.8 /usr/lib/python3/dist-packages importlib-metadata 7.0.1 /usr/local/lib/python3.9/dist-packages pip jaraco.classes 3.3.1 /usr/local/lib/python3.9/dist-packages pip jeepney 0.8.0 /usr/local/lib/python3.9/dist-packages pip keyring 24.3.1 /usr/local/lib/python3.9/dist-packages pip markdown-it-py 3.0.0 /usr/local/lib/python3.9/dist-packages pip mdurl 0.1.2 /usr/local/lib/python3.9/dist-packages pip more-itertools 10.2.0 /usr/local/lib/python3.9/dist-packages pip nh3 0.2.15 /usr/local/lib/python3.9/dist-packages pip pip 24.0 /usr/local/lib/python3.9/dist-packages pip pkginfo 1.10.0 /usr/local/lib/python3.9/dist-packages pip portpicker 1.6.0 /usr/local/lib/python3.9/dist-packages pip psutil 5.9.8 /usr/local/lib/python3.9/dist-packages pip pycparser 2.21 /usr/local/lib/python3.9/dist-packages pip pyelftools 0.30 /usr/local/lib/python3.9/dist-packages pip Pygments 2.17.2 /usr/local/lib/python3.9/dist-packages pip PyGObject 3.36.0 /usr/lib/python3/dist-packages python-apt 2.0.1+ubuntu0.20.4.1 /usr/lib/python3/dist-packages readme_renderer 43.0 /usr/local/lib/python3.9/dist-packages pip requests 2.22.0 /usr/lib/python3/dist-packages requests-toolbelt 1.0.0 /usr/local/lib/python3.9/dist-packages pip requests-unixsocket 0.2.0 /usr/lib/python3/dist-packages rfc3986 2.0.0 /usr/local/lib/python3.9/dist-packages pip rich 13.7.1 /usr/local/lib/python3.9/dist-packages pip SecretStorage 3.3.3 /usr/local/lib/python3.9/dist-packages pip setuptools 69.1.1 /usr/local/lib/python3.9/dist-packages pip six 1.14.0 /usr/lib/python3/dist-packages twine 4.0.2 /usr/local/lib/python3.9/dist-packages pip urllib3 2.2.1 /usr/local/lib/python3.9/dist-packages pip wheel 0.42.0 /usr/local/lib/python3.9/dist-packages pip zipp 3.17.0 /usr/local/lib/python3.9/dist-packages pip
TASK [tensorflow_in_container : (Python 3.9) build TensorFlow git] ************* Wednesday 27 March 2024 18:11:03 +0000 (0:00:00.036) 0:00:58.710 ******* fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- Extracting Bazel installation... Starting local Bazel server and connecting to it... INFO: Invocation ID: fbecce73-94c3-4a99-a02a-6e831210076d INFO: Reading 'startup' options from /tf/tensorflow/.bazelrc: --windows_enable_symlinks INFO: Options provided by the client: Inherited 'common' options: --isatty=0 --terminal_columns=80 INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc: Inherited 'common' options: --experimental_repo_remote_exec INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc: 'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility INFO: Reading rc options for 'build' from /usertools/aarch64.bazelrc: 'build' options: --action_env=CACHEBUSTER=20220325 --action_env PYTHON_BIN_PATH=/usr/local/bin/python3 --python_path=/usr/local/bin/python3 --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 --distinct_host_configuration=false --profile=/tf/pkg/profile.json.gz --crosstool_top=@ml2014_aarch64_config_aarch64//crosstool:toolchain --copt=-mtune=generic --copt=-march=armv8-a --copt=-O3 INFO: Found applicable config definition build:short_logs in file /tf/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING INFO: Found applicable config definition build:v2 in file /tf/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 INFO: Found applicable config definition build:sigbuild_local_cache in file /usertools/aarch64.bazelrc: --disk_cache=/tf/cache INFO: Found applicable config definition build:linux in file /tf/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes INFO: Found applicable config definition build:dynamic_kernels in file /tf/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: Loading: 0 packages loaded Analyzing: target //tensorflow/tools/pip_package:wheel (1 packages loaded, 0 targets configured) Analyzing: target //tensorflow/tools/pip_package:wheel (10 packages loaded, 12 targets configured) Analyzing: target //tensorflow/tools/pip_package:wheel (157 packages loaded, 302 targets configured) ERROR: /tf/tensorflow/tensorflow/tools/pip_package/BUILD:260:19: no such target '//tensorflow/compiler/mlir/tensorflow:gen_mlir_passthrough_op_py': target 'gen_mlir_passthrough_op_py' not declared in package 'tensorflow/compiler/mlir/tensorflow' defined by /tf/tensorflow/tensorflow/compiler/mlir/tensorflow/BUILD (Tip: use `query "//tensorflow/compiler/mlir/tensorflow:*"` to see all the targets in that package) and referenced by '//tensorflow/tools/pip_package:py_deps_gather' INFO: Repository go_sdk instantiated at: /tf/tensorflow/WORKSPACE:94:14: in <toplevel> /tf/tensorflow/tensorflow/workspace0.bzl:135:20: in workspace /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/com_github_grpc_grpc/bazel/grpc_extra_deps.bzl:36:27: in grpc_extra_deps /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/io_bazel_rules_go/go/private/sdk.bzl:431:28: in go_register_toolchains /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/io_bazel_rules_go/go/private/sdk.bzl:130:21: in go_download_sdk Repository rule _go_download_sdk defined at: /home/buildslave/.cache/bazel/_bazel_buildslave/fbac33eb30dbfb6b11b15a7ff5ac830d/external/io_bazel_rules_go/go/private/sdk.bzl:117:35: in <toplevel> ERROR: Analysis of target '//tensorflow/tools/pip_package:wheel' failed; build aborted: INFO: Elapsed time: 53.979s INFO: 0 processes. FAILED: Build did NOT complete successfully (261 packages loaded, 4001 targets configured) stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
PLAY RECAP ********************************************************************* localhost : ok=25 changed=17 unreachable=0 failed=1 skipped=6 rescued=0 ignored=0
Wednesday 27 March 2024 18:11:58 +0000 (0:00:54.904) 0:01:53.615 ******* =============================================================================== tensorflow_in_container : (Python 3.9) build TensorFlow git ------------ 54.90s tensorflow : Fetch TensorFlow from git --------------------------------- 21.39s Cleanup after previous build ------------------------------------------- 12.35s tensorflow : Remove TensorFlow git sources from previous build ---------- 4.11s tensorflow_in_container : (Python 3.9) start build container ------------ 3.83s Gathering Facts --------------------------------------------------------- 2.64s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 1.94s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.49s Create cache dirs ------------------------------------------------------- 1.40s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.21s tensorflow : Save git commit hash to file ------------------------------- 1.14s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.85s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.82s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.82s tensorflow : Create linklibs shell script ------------------------------- 0.69s tensorflow_in_container : (Python 3.9) copy upper constraints for hermetic builds --- 0.68s tensorflow_in_container : Reset checkout to clean state ----------------- 0.64s tensorflow : Use nightly package requirements --------------------------- 0.62s tensorflow : Get oneline version of git log for this checkout ----------- 0.58s Build TensorFlow for each Python version (dual abi) --------------------- 0.44s Build step 'Execute shell' marked build as failure