[ldcg-ai-ml] Build failed in Jenkins: Tensorflow Python package (manylinux2014 nightly) #209

22 Dec 2021


      See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/209/display/redirect
Changes:
------------------------------------------
[...truncated 26.46 KB...]
ok: [localhost]
TASK [python : get filename of latest Keras_Preprocessing wheel] ***************
Tuesday 21 December 2021  11:38:51 +0000 (0:00:00.268)       0:14:51.607 ****** 
ok: [localhost]
TASK [python : install Keras_Preprocessing-1.1.2-py2.py3-none-any.whl Python package] ***
Tuesday 21 December 2021  11:38:51 +0000 (0:00:00.057)       0:14:51.664 ****** 
changed: [localhost]
TASK [python : build wheels of Python package scipy] ***************************
Tuesday 21 December 2021  11:38:53 +0000 (0:00:01.853)       0:14:53.517 ****** 
changed: [localhost]
TASK [python : get list of wheel files matching scipy] *************************
Tuesday 21 December 2021  11:38:59 +0000 (0:00:06.230)       0:14:59.748 ****** 
ok: [localhost]
TASK [python : get filename of latest scipy wheel] *****************************
Tuesday 21 December 2021  11:38:59 +0000 (0:00:00.270)       0:15:00.019 ****** 
ok: [localhost]
TASK [python : install scipy-1.5.4-cp37-cp37m-manylinux2014_aarch64.whl Python package] ***
Tuesday 21 December 2021  11:38:59 +0000 (0:00:00.055)       0:15:00.075 ****** 
changed: [localhost]
TASK [tensorflow : remove TensorFlow git sources from previous build] **********
Tuesday 21 December 2021  11:39:06 +0000 (0:00:06.569)       0:15:06.645 ****** 
changed: [localhost]
TASK [fetch tensorflow git source] *********************************************
Tuesday 21 December 2021  11:39:10 +0000 (0:00:04.372)       0:15:11.018 ****** 
skipping: [localhost]
TASK [unpack tensorflow git source] ********************************************
Tuesday 21 December 2021  11:39:10 +0000 (0:00:00.054)       0:15:11.072 ****** 
skipping: [localhost]
TASK [tensorflow : copy configure script] **************************************
Tuesday 21 December 2021  11:39:10 +0000 (0:00:00.043)       0:15:11.116 ****** 
skipping: [localhost]
TASK [tensorflow : drop Python 2.7 deps] ***************************************
Tuesday 21 December 2021  11:39:10 +0000 (0:00:00.042)       0:15:11.159 ****** 
skipping: [localhost] => (item=functools32) 
skipping: [localhost] => (item=enum34) 
skipping: [localhost] => (item=backports.weakref)
TASK [fetch tensorflow from git] ***********************************************
Tuesday 21 December 2021  11:39:10 +0000 (0:00:00.071)       0:15:11.230 ****** 
changed: [localhost]
TASK [tensorflow : copy configure script] **************************************
Tuesday 21 December 2021  11:39:32 +0000 (0:00:21.741)       0:15:32.972 ****** 
changed: [localhost]
TASK [tensorflow : create readme file about unit tests] ************************
Tuesday 21 December 2021  11:39:33 +0000 (0:00:00.452)       0:15:33.424 ****** 
changed: [localhost]
TASK [tensorflow : add to the copied files] ************************************
Tuesday 21 December 2021  11:39:33 +0000 (0:00:00.469)       0:15:33.893 ****** 
changed: [localhost]
TASK [tensorflow : add to the long_description in setup.py] ********************
Tuesday 21 December 2021  11:39:34 +0000 (0:00:00.469)       0:15:34.363 ****** 
changed: [localhost]
TASK [tensorflow : build TensorFlow git for cp37-cp37m] ************************
Tuesday 21 December 2021  11:39:34 +0000 (0:00:00.465)       0:15:34.829 ****** 
fatal: [localhost]: FAILED! => changed=true 
  cmd: |-
    set -xe
    source /tmp/workspace/venv-cp37-cp37m/bin/activate
    expect configure_tensorflow
    bazel clean --expunge
    export BAZEL_LINKLIBS=-l%:libstdc++.a
    bazel build --config=nonccl //tensorflow/tools/pip_package:build_pip_package --verbose_failures \
                --copt=-ffp-contract=off --cxxopt=-ffp-contract=off
    mkdir tensorflow-pkg
    bazel-bin/tensorflow/tools/pip_package/build_pip_package --cpu --project_name tensorflow_aarch64 ./tensorflow-pkg
  delta: '0:01:46.632962'
  end: '2021-12-21 11:41:21.396209'
  msg: non-zero return code
  rc: 1
  start: '2021-12-21 11:39:34.763247'
  stderr: |-
    + source /tmp/workspace/venv-cp37-cp37m/bin/activate
    ++ deactivate nondestructive
    ++ '[' -n '' ']'
    ++ '[' -n '' ']'
    ++ '[' -n /bin/bash -o -n '' ']'
    ++ hash -r
    ++ '[' -n '' ']'
    ++ unset VIRTUAL_ENV
    ++ '[' '!' nondestructive = nondestructive ']'
    ++ VIRTUAL_ENV=/tmp/workspace/venv-cp37-cp37m
    ++ export VIRTUAL_ENV
    ++ _OLD_VIRTUAL_PATH=/opt/rh/devtoolset-10/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    ++ PATH=/tmp/workspace/venv-cp37-cp37m/bin:/opt/rh/devtoolset-10/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
    ++ export PATH
    ++ '[' -n '' ']'
    ++ '[' -z '' ']'
    ++ _OLD_VIRTUAL_PS1=
    ++ '[' 'x(venv-cp37-cp37m) ' '!=' x ']'
    ++ PS1='(venv-cp37-cp37m) '
    ++ export PS1
    ++ '[' -n /bin/bash -o -n '' ']'
    ++ hash -r
    + expect configure_tensorflow
    + bazel clean --expunge
    Extracting Bazel installation...
    Starting local Bazel server and connecting to it...
    INFO: Options provided by the client:
      Inherited 'common' options: --isatty=0 --terminal_columns=80
    INFO: Reading rc options for 'clean' from /tmp/workspace/tensorflow-git/.bazelrc:
      Inherited 'common' options: --experimental_repo_remote_exec
    INFO: Reading rc options for 'clean' from /tmp/workspace/tensorflow-git/.bazelrc:
      Inherited 'build' options: --define framework_shared_object=true --java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --host_java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/fallback,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
    INFO: Found applicable config definition build:short_logs in file /tmp/workspace/tensorflow-git/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
    INFO: Found applicable config definition build:v2 in file /tmp/workspace/tensorflow-git/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
    INFO: Found applicable config definition build:linux in file /tmp/workspace/tensorflow-git/.bazelrc: --copt=-w --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
    INFO: Found applicable config definition build:dynamic_kernels in file /tmp/workspace/tensorflow-git/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
    INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
    + export BAZEL_LINKLIBS=-l%:libstdc++.a
    + BAZEL_LINKLIBS=-l%:libstdc++.a
    + bazel build --config=nonccl //tensorflow/tools/pip_package:build_pip_package --verbose_failures --copt=-ffp-contract=off --cxxopt=-ffp-contract=off
    Starting local Bazel server and connecting to it...
    INFO: Options provided by the client:
      Inherited 'common' options: --isatty=0 --terminal_columns=80
    INFO: Reading rc options for 'build' from /tmp/workspace/tensorflow-git/.bazelrc:
      Inherited 'common' options: --experimental_repo_remote_exec
    INFO: Reading rc options for 'build' from /tmp/workspace/tensorflow-git/.bazelrc:
      'build' options: --define framework_shared_object=true --java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --host_java_toolchain=@tf_toolchains//toolchains/java:tf_java_toolchain --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/fallback,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
    INFO: Found applicable config definition build:short_logs in file /tmp/workspace/tensorflow-git/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
    INFO: Found applicable config definition build:v2 in file /tmp/workspace/tensorflow-git/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
    INFO: Found applicable config definition build:nonccl in file /tmp/workspace/tensorflow-git/.bazelrc: --define=no_nccl_support=true
    INFO: Found applicable config definition build:linux in file /tmp/workspace/tensorflow-git/.bazelrc: --copt=-w --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes
    INFO: Found applicable config definition build:dynamic_kernels in file /tmp/workspace/tensorflow-git/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
    Loading:
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/tensorflow/r... failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    Loading: 0 packages loaded
    INFO: Repository llvm-raw instantiated at:
      /tmp/workspace/tensorflow-git/WORKSPACE:11:14: in <toplevel>
      /tmp/workspace/tensorflow-git/tensorflow/workspace3.bzl:52:9: in workspace
      /tmp/workspace/tensorflow-git/third_party/llvm/workspace.bzl:10:20: in repo
      /tmp/workspace/tensorflow-git/third_party/repo.bzl:128:21: in tf_http_archive
    Repository rule _tf_http_archive defined at:
      /tmp/workspace/tensorflow-git/third_party/repo.bzl:81:35: in <toplevel>
    INFO: Repository 'llvm-raw' used the following cache hits instead of downloading the corresponding file.
     * Hash 'd68b5fa9b851cc0b8a5f84f7659404380f477efc859f21951bc3554476047245' for https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-pr...
    If the definition of 'llvm-raw' was updated, verify that the hashes were also updated.
    ERROR: An error occurred during the fetch of repository 'llvm-raw':
       Traceback (most recent call last):
            File "/tmp/workspace/tensorflow-git/third_party/repo.bzl", line 75, column 30, in _tf_http_archive_impl
                    ctx.patch(patch_file, strip = 1)
    Error in patch: Error applying patch /tmp/workspace/tensorflow-git/third_party/llvm/Bazel-update-build-files-for.patch: Incorrect Chunk: the chunk content doesn't match the target
    **Original Position**: 840
**Original Content**:
        copts = llvm_copts,
        deps = [
            ":Core",
            ":Support",
            ":config",
        ],
**Revised Content**:
        copts = llvm_copts,
        deps = [
            ":Core",
            ":DebugInfoDWARF",
            ":Support",
            ":config",
        ],
    ERROR: no such package '@llvm-raw//utils/bazel': Error applying patch /tmp/workspace/tensorflow-git/third_party/llvm/Bazel-update-build-files-for.patch: Incorrect Chunk: the chunk content doesn't match the target
    **Original Position**: 840
**Original Content**:
        copts = llvm_copts,
        deps = [
            ":Core",
            ":Support",
            ":config",
        ],
**Revised Content**:
        copts = llvm_copts,
        deps = [
            ":Core",
            ":DebugInfoDWARF",
            ":Support",
            ":config",
        ],
    INFO: Elapsed time: 99.318s
    INFO: 0 processes.
    FAILED: Build did NOT complete successfully (0 packages loaded)
    FAILED: Build did NOT complete successfully (0 packages loaded)
  stderr_lines: <omitted>
  stdout: |-
    spawn ./configure
    You have bazel 3.7.2 installed.
    Please upgrade your bazel installation to version 4.2.1 or higher to build TensorFlow!
  stdout_lines: <omitted>
PLAY RECAP *********************************************************************
localhost                  : ok=43   changed=26   unreachable=0    failed=1    skipped=9    rescued=0    ignored=0
Tuesday 21 December 2021  11:41:21 +0000 (0:01:46.916)       0:17:21.746 ****** 
=============================================================================== 
python : build wheels of Python package h5py~=3.1.0 ------------------- 426.11s
python : build wheels of Python package Keras_Applications ------------ 249.82s
tensorflow : build TensorFlow git for cp37-cp37m ---------------------- 106.92s
install bazel build requirements --------------------------------------- 47.69s
tensorflow : populate cp37-cp37m virtualenv ---------------------------- 39.34s
python : install numpy build requirements ------------------------------ 31.97s
fetch tensorflow from git ---------------------------------------------- 21.74s
python : create cp37-cp37m virtualenv ---------------------------------- 17.99s
python : install h5py build requirements ------------------------------- 16.69s
bazel : fetch Bazel 3.7.2 binary from Github --------------------------- 15.04s
install tensorflow build requirements ----------------------------------- 7.68s
python : install scipy-1.5.4-cp37-cp37m-manylinux2014_aarch64.whl Python package --- 6.57s
python : build wheels of Python package Keras_Preprocessing ------------- 6.54s
python : build wheels of Python package scipy --------------------------- 6.23s
python : install h5py-3.1.0-cp37-cp37m-linux_aarch64.whl Python package --- 5.57s
python : build wheels of Python package grpcio -------------------------- 4.62s
Gathering Facts --------------------------------------------------------- 4.57s
tensorflow : remove TensorFlow git sources from previous build ---------- 4.37s
python : build wheels of Python package numpy~=1.19.5 ------------------- 4.10s
python : enable EPEL ---------------------------------------------------- 3.01s
Build step 'Execute shell' marked build as failure

2024

2023

2022

2021

[ldcg-ai-ml] Build failed in Jenkins: Tensorflow Python package (manylinux2014 nightly) #209