New subject: Jenkins build is back to normal : Tensorflow oneDNN Python package (manylinux2014 nightly) #606

1 Nov 2023


      See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/605/display/redirect
Changes:
------------------------------------------
[...truncated 15.70 KB...]
TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build] ***
Wednesday 01 November 2023  00:11:23 +0000 (0:00:00.843)       0:01:18.334 **** 
changed: [localhost]
TASK [tensorflow_in_container Wednesday 01 November 2023 ok: [localhost] => pip_list_output.stdout: |- Package                 Version ----------------------- absl-py                 1.0.0 astor                   0.8.1 astroid                 2.11.7 astunparse              1.6.3 auditwheel              5.3.0 cachetools              5.3.1 certifi                 2019.11.28 cffi                    1.16.0 chardet                 3.0.4 colorama                0.4.6 cryptography            41.0.4 Cython                  3.0.4 dbus-python             1.2.16 dill                    0.3.7 dm-tree                 0.1.8 docutils                0.20.1 flatbuffers             23.5.26 future                  0.18.3 gast                    0.4.0 google-auth             2.23.3 google-auth-oauthlib    1.1.0 google-pasta            0.2.0 grpcio                  1.49.1 h5py                    3.8.0 idna                    2.8 importlib-metadata      6.8.0 isort                   5.12.0 jaraco.classes          3.3.0 jax                     0.3.25 jeepney                 0.8.0 junitparser             2.2.0 Keras-Applications      1.0.8 keras-nightly Keras-Preprocessing     1.1.2 keyring                 24.2.0 lazy-object-proxy       1.9.0 lit                     16.0.6 lxml                    4.9.3 Markdown                3.5 markdown-it-py          3.0.0 MarkupSafe              2.1.3 mccabe                  0.7.0 mdurl                   0.1.2 more-itertools          10.1.0 namex                   0.0.7 nh3                     0.2.14 numpy                   1.23.5 oauthlib                3.2.2 opt-einsum              3.3.0 packaging               21.3 pip                     23.3 pkginfo                 1.9.6 platformdirs            3.11.0 portpicker              1.5.2 protobuf                4.23.4 psutil                  5.9.6 pyasn1                  0.5.0 pyasn1-modules          0.3.0 pycparser               2.21 pyelftools              0.30 Pygments                2.16.1 PyGObject               3.36.0 pylint                  2.13.9 pyparsing               3.1.1 python-apt PyYAML                  6.0.1 readme-renderer         42.0 requests                2.22.0 requests-oauthlib       1.3.1 requests-toolbelt       1.0.0 requests-unixsocket     0.2.0 rfc3986                 2.0.0 rich                    13.6.0 rsa                     4.9 scipy                   1.9.3 SecretStorage           3.3.3 setuptools              45.2.0 six                     1.15.0 tb-nightly tensorboard-data-server 0.7.1 termcolor               1.1.0 tf-estimator-nightly tomli                   2.0.1 tqdm                    4.66.1 twine                   3.6.0 typing-extensions       3.7.4.3 urllib3                 1.25.8 Werkzeug                3.0.0 wheel                   0.38.4 wrapt                   1.12.1 zipp                    3.17.0

: Show list of installed Python packages for build] *** 00:11:25 +0000 (0:00:02.136)       0:01:20.471 **** Location                               Installer -------------------- -------------------------------------- --------- /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/lib/python3/dist-packages /usr/local/lib/python3.9/dist-packages pip /usr/lib/python3/dist-packages /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/lib/python3/dist-packages /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/lib/python3/dist-packages /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip 3.0.0.dev2023101903  /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/lib/python3/dist-packages /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip 2.0.1+ubuntu0.20.4.1 /usr/lib/python3/dist-packages /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/lib/python3/dist-packages /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/lib/python3/dist-packages /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/lib/python3/dist-packages /usr/local/lib/python3.9/dist-packages pip 2.15.0a20231019      /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip 2.15.0.dev2023101608 /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/lib/python3/dist-packages /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip /usr/local/lib/python3.9/dist-packages pip
TASK [tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git] ***
Wednesday 01 November 2023  00:11:25 +0000 (0:00:00.043)       0:01:20.514 **** 
fatal: [localhost]: FAILED! => changed=true 
  rc: 1
  stderr: |-
    Extracting Bazel installation...
    Starting local Bazel server and connecting to it...
    INFO: Invocation ID: fc466641-e44a-47cc-98bb-6908fd0e57b0
    INFO: Reading 'startup' options from /tf/tensorflow/.bazelrc: --windows_enable_symlinks
    INFO: Options provided by the client:
      Inherited 'common' options: --isatty=0 --terminal_columns=80
    INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc:
      Inherited 'common' options: --experimental_repo_remote_exec
    INFO: Reading rc options for 'build' from /tf/tensorflow/.bazelrc:
      'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
    INFO: Reading rc options for 'build' from /usertools/aarch64_clang.bazelrc:
      'build' options: --action_env=CACHEBUSTER=20220325 --action_env PYTHON_BIN_PATH=/usr/local/bin/python3 --python_path=/usr/local/bin/python3 --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 --linkopt=-fuse-ld=lld --linkopt=-lm --linkopt=-Wl,--undefined-version --distinct_host_configuration=false --copt=-Wno-gnu-offsetof-extensions --profile=/tf/pkg/profile.json.gz --crosstool_top=@ml2014_clang_aarch64_config_aarch64//crosstool:toolchain --copt=-mtune=generic --copt=-march=armv8-a --copt=-O3
    INFO: Found applicable config definition build:short_logs in file /tf/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
    INFO: Found applicable config definition build:v2 in file /tf/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
    INFO: Found applicable config definition build:sigbuild_local_cache in file /usertools/aarch64_clang.bazelrc: --disk_cache=/tf/cache
    INFO: Found applicable config definition build:mkl_aarch64_threadpool in file /tf/tensorflow/.bazelrc: --define=build_with_mkl_aarch64=true -c opt
    INFO: Found applicable config definition build:linux in file /tf/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
    INFO: Found applicable config definition build:dynamic_kernels in file /tf/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    Loading:
    INFO: Repository llvm-raw instantiated at:
      /tf/tensorflow/WORKSPACE:80:14: in <toplevel>
      /tf/tensorflow/tensorflow/workspace3.bzl:63:9: in workspace
      /tf/tensorflow/third_party/llvm/workspace.bzl:10:20: in repo
      /tf/tensorflow/third_party/repo.bzl:136:21: in tf_http_archive
    Repository rule _tf_http_archive defined at:
      /tf/tensorflow/third_party/repo.bzl:89:35: in <toplevel>
    WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-pr... failed: class java.io.FileNotFoundException GET returned 404 Not Found
    ERROR: An error occurred during the fetch of repository 'llvm-raw':
       Traceback (most recent call last):
            File "/tf/tensorflow/third_party/repo.bzl", line 83, column 30, in _tf_http_archive_impl
                    ctx.patch(patch_file, strip = 1)
    Error in patch: Error applying patch /tf/tensorflow/third_party/llvm/fix_init_module_python.patch: Incorrect Chunk: the chunk content doesn't match the target
    **Original Position**: 83
**Original Content**:
# If _mlirRegisterEverything is built, then include it as an initializer
        # module.
        if process_initializer_module("_mlirRegisterEverything"):
            init_module = importlib.import_module(f"._mlirRegisterEverything", __name__)
**Revised Content**:
# If _mlirRegisterEverything is built, then include it as an initializer
        # module.
        init_module = None
        if process_initializer_module("_mlirRegisterEverything"):
            init_module = importlib.import_module(f"._mlirRegisterEverything", __name__)
ERROR: /tf/tensorflow/WORKSPACE:80:14: fetching _tf_http_archive rule //external:llvm-raw: Traceback (most recent call last):
            File "/tf/tensorflow/third_party/repo.bzl", line 83, column 30, in _tf_http_archive_impl
                    ctx.patch(patch_file, strip = 1)
    Error in patch: Error applying patch /tf/tensorflow/third_party/llvm/fix_init_module_python.patch: Incorrect Chunk: the chunk content doesn't match the target
    **Original Position**: 83
**Original Content**:
# If _mlirRegisterEverything is built, then include it as an initializer
        # module.
        if process_initializer_module("_mlirRegisterEverything"):
            init_module = importlib.import_module(f"._mlirRegisterEverything", __name__)
**Revised Content**:
# If _mlirRegisterEverything is built, then include it as an initializer
        # module.
        init_module = None
        if process_initializer_module("_mlirRegisterEverything"):
            init_module = importlib.import_module(f"._mlirRegisterEverything", __name__)
ERROR: Error computing the main repository mapping: no such package '@llvm-raw//utils/bazel': Error applying patch /tf/tensorflow/third_party/llvm/fix_init_module_python.patch: Incorrect Chunk: the chunk content doesn't match the target
    **Original Position**: 83
**Original Content**:
# If _mlirRegisterEverything is built, then include it as an initializer
        # module.
        if process_initializer_module("_mlirRegisterEverything"):
            init_module = importlib.import_module(f"._mlirRegisterEverything", __name__)
**Revised Content**:
# If _mlirRegisterEverything is built, then include it as an initializer
        # module.
        init_module = None
        if process_initializer_module("_mlirRegisterEverything"):
            init_module = importlib.import_module(f"._mlirRegisterEverything", __name__)
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
PLAY RECAP *********************************************************************
localhost                  : ok=29   changed=20   unreachable=0    failed=1    skipped=8    rescued=0    ignored=0
Wednesday 01 November 2023  00:12:25 +0000 (0:01:00.045)       0:02:20.560 **** 
=============================================================================== 
tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git -- 60.05s
Cleanup after previous build ------------------------------------------- 32.04s
tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 21.05s
tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 4.09s
tensorflow_in_container : (Python 3.9) start build container ------------ 3.72s
Gathering Facts --------------------------------------------------------- 2.65s
tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 2.14s
tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.61s
Create cache dirs ------------------------------------------------------- 1.42s
tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.22s
tensorflow : Save git commit hash to file ------------------------------- 1.14s
tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.89s
tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.84s
tensorflow_in_container : Reset checkout to clean state ----------------- 0.84s
tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.79s
tensorflow_in_container : (Python 3.9) copy upper constraints ----------- 0.72s
tensorflow : Create readme file about unit tests ------------------------ 0.70s
tensorflow : Create linklibs shell script ------------------------------- 0.69s
tensorflow : Add to the copied files ------------------------------------ 0.59s
tensorflow : Get oneline version of git log for this checkout ----------- 0.58s
Build step 'Execute shell' marked build as failure

Build failed in Jenkins: Tensorflow oneDNN Python package (manylinux2014 nightly) #605