See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/69/display/redirect
Changes:
------------------------------------------ [...truncated 29.62 KB...] TASK [python : build wheels of Python package h5py~=3.6.0] ********************* Tuesday 31 May 2022 09:15:28 +0000 (0:00:02.093) 0:03:22.679 *********** changed: [localhost]
TASK [python : get list of wheel files matching h5py~=3.6.0] ******************* Tuesday 31 May 2022 09:20:01 +0000 (0:04:32.572) 0:07:55.251 *********** ok: [localhost]
TASK [python : get filename of latest h5py~=3.6.0 wheel] *********************** Tuesday 31 May 2022 09:20:01 +0000 (0:00:00.270) 0:07:55.522 *********** ok: [localhost]
TASK [python : install h5py-3.6.0-cp37-cp37m-linux_aarch64.whl Python package] *** Tuesday 31 May 2022 09:20:01 +0000 (0:00:00.051) 0:07:55.574 *********** changed: [localhost]
TASK [python : build wheels of Python package Keras_Applications] ************** Tuesday 31 May 2022 09:20:04 +0000 (0:00:02.957) 0:07:58.532 *********** changed: [localhost]
TASK [python : get list of wheel files matching Keras_Applications] ************ Tuesday 31 May 2022 09:24:34 +0000 (0:04:30.022) 0:12:28.554 *********** ok: [localhost]
TASK [python : get filename of latest Keras_Applications wheel] **************** Tuesday 31 May 2022 09:24:34 +0000 (0:00:00.273) 0:12:28.828 *********** ok: [localhost]
TASK [python : install Keras_Applications-1.0.8-py3-none-any.whl Python package] *** Tuesday 31 May 2022 09:24:34 +0000 (0:00:00.053) 0:12:28.881 *********** changed: [localhost]
TASK [python : build wheels of Python package Keras_Preprocessing] ************* Tuesday 31 May 2022 09:24:37 +0000 (0:00:02.218) 0:12:31.100 *********** changed: [localhost]
TASK [python : get list of wheel files matching Keras_Preprocessing] *********** Tuesday 31 May 2022 09:24:41 +0000 (0:00:04.202) 0:12:35.302 *********** ok: [localhost]
TASK [python : get filename of latest Keras_Preprocessing wheel] *************** Tuesday 31 May 2022 09:24:41 +0000 (0:00:00.273) 0:12:35.576 *********** ok: [localhost]
TASK [python : install Keras_Preprocessing-1.1.2-py2.py3-none-any.whl Python package] *** Tuesday 31 May 2022 09:24:41 +0000 (0:00:00.054) 0:12:35.630 *********** changed: [localhost]
TASK [python : build wheels of Python package scipy] *************************** Tuesday 31 May 2022 09:24:43 +0000 (0:00:02.215) 0:12:37.845 *********** changed: [localhost]
TASK [python : get list of wheel files matching scipy] ************************* Tuesday 31 May 2022 09:24:50 +0000 (0:00:06.870) 0:12:44.716 *********** ok: [localhost]
TASK [python : get filename of latest scipy wheel] ***************************** Tuesday 31 May 2022 09:24:50 +0000 (0:00:00.269) 0:12:44.985 *********** ok: [localhost]
TASK [python : install scipy-1.7.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl Python package] *** Tuesday 31 May 2022 09:24:51 +0000 (0:00:00.053) 0:12:45.039 *********** changed: [localhost]
TASK [tensorflow : Set build dir for TensorFlow-oneDNN-ACL git] **************** Tuesday 31 May 2022 09:24:58 +0000 (0:00:07.502) 0:12:52.542 *********** ok: [localhost]
TASK [tensorflow : remove TensorFlow-oneDNN-ACL git sources from previous build] *** Tuesday 31 May 2022 09:24:58 +0000 (0:00:00.052) 0:12:52.594 *********** changed: [localhost]
TASK [fetch tensorflow-oneDNN-ACL git source] ********************************** Tuesday 31 May 2022 09:25:03 +0000 (0:00:04.746) 0:12:57.340 *********** skipping: [localhost]
TASK [unpack tensorflow-oneDNN-ACL git source] ********************************* Tuesday 31 May 2022 09:25:03 +0000 (0:00:00.054) 0:12:57.395 *********** skipping: [localhost]
TASK [tensorflow : Rename source dir for oneDNN build] ************************* Tuesday 31 May 2022 09:25:03 +0000 (0:00:00.040) 0:12:57.436 *********** skipping: [localhost]
TASK [fetch tensorflow-oneDNN-ACL from git] ************************************ Tuesday 31 May 2022 09:25:03 +0000 (0:00:00.043) 0:12:57.480 *********** changed: [localhost]
TASK [tensorflow : Get oneline version of git log for this checkout] *********** Tuesday 31 May 2022 09:25:25 +0000 (0:00:22.530) 0:13:20.010 *********** changed: [localhost]
TASK [tensorflow : Show git log] *********************************************** Tuesday 31 May 2022 09:25:26 +0000 (0:00:00.584) 0:13:20.594 *********** ok: [localhost] => git_log.stdout: 043be71ab3309af5cc174653b2f7a25d4d5c6aa5 [XLA:GPU] Summarize cwise HLO ops when producing fusion stats.
TASK [tensorflow : use nightly package requirements] *************************** Tuesday 31 May 2022 09:25:26 +0000 (0:00:00.049) 0:13:20.644 *********** changed: [localhost]
TASK [tensorflow : Use rolling bazel release instead of last_downstream_green] *** Tuesday 31 May 2022 09:25:27 +0000 (0:00:00.473) 0:13:21.117 *********** ok: [localhost]
TASK [tensorflow : Create linklibs shell script] ******************************* Tuesday 31 May 2022 09:25:27 +0000 (0:00:00.462) 0:13:21.580 *********** changed: [localhost]
TASK [tensorflow : create readme file about unit tests] ************************ Tuesday 31 May 2022 09:25:28 +0000 (0:00:00.461) 0:13:22.042 *********** changed: [localhost]
TASK [tensorflow : add to the copied files] ************************************ Tuesday 31 May 2022 09:25:28 +0000 (0:00:00.469) 0:13:22.511 *********** changed: [localhost]
TASK [tensorflow : add to the long_description in setup.py] ******************** Tuesday 31 May 2022 09:25:28 +0000 (0:00:00.264) 0:13:22.776 *********** changed: [localhost]
TASK [tensorflow : Get python site-packages location] ************************** Tuesday 31 May 2022 09:25:29 +0000 (0:00:00.468) 0:13:23.244 *********** changed: [localhost]
TASK [tensorflow : Configure default TensorFlow settings] ********************** Tuesday 31 May 2022 09:25:29 +0000 (0:00:00.624) 0:13:23.869 *********** changed: [localhost]
TASK [tensorflow : Set Python bin path] **************************************** Tuesday 31 May 2022 09:25:30 +0000 (0:00:00.568) 0:13:24.437 *********** changed: [localhost]
TASK [tensorflow : build TensorFlow-oneDNN-ACL git for cp37-cp37m] ************* Tuesday 31 May 2022 09:25:30 +0000 (0:00:00.566) 0:13:25.004 *********** changed: [localhost]
TASK [tensorflow : Clean bazel install] **************************************** Tuesday 31 May 2022 10:01:59 +0000 (0:36:28.500) 0:49:53.505 *********** changed: [localhost]
TASK [tensorflow : Run unit tests on TensorFlow-oneDNN-ACL git for cp37-cp37m] *** Tuesday 31 May 2022 10:01:59 +0000 (0:00:00.388) 0:49:53.893 *********** fatal: [localhost]: FAILED! => changed=true cmd: |- set -xe; source /tmp/workspace/venv-cp37-cp37m/bin/activate; source ./linklibs.sh; source tensorflow/tools/ci_build/build_scripts/DEFAULT_TEST_TARGETS.sh; bazel test --test_timeout=300,500,-1,-1 --flaky_test_attempts=3 --test_output=all --cache_test_results=no --config=nonccl \ --config=mkl_aarch64 --copt="-fopenmp" --linkopt="-lgomp" --copt="-mtune=generic" --copt="-march=armv8-a" --copt="-O3" --verbose_failures \ --test_env=TF_ENABLE_ONEDNN_OPTS=1 --build_tag_filters=-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-no_aarch64,-requires-gpu \ --test_tag_filters=-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-no_aarch64,-requires-gpu --build_tests_only \ -- ${DEFAULT_BAZEL_TARGETS} \ -//tensorflow/lite/... -//tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test -//tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu -//tensorflow/python/client:session_list_devices_test -//tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test -//tensorflow/python/data/kernel_tests:iterator_test_cpu -//tensorflow/python/training:server_lib_test \ -//tensorflow/compiler/mlir/tensorflow/tests/... -//tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test -//tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu -//tensorflow/core/grappler/optimizers:constant_folding_test -//tensorflow/core/grappler/optimizers:remapper_test_cpu -//tensorflow/core/kernels:conv_ops_test_cpu -//tensorflow/core/kernels:matmul_op_test_cpu -//tensorflow/python/eager:forwardprop_test_cpu -//tensorflow/python/keras/engine:training_arrays_test -//tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu -//tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu -//tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu -//tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu -//tensorflow/python/kernel_tests/linalg:linear_operator_kronecker_test_cpu -//tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu -//tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu -//tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu -//tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu -//tensorflow/python/ops/parallel_for:math_test_cpu -//tensorflow/python:dequantize_op_test -//tensorflow/python:nn_grad_test_cpu -//tensorflow/python:quantized_ops_test delta: '0:00:07.157956' end: '2022-05-31 10:02:07.245339' msg: non-zero return code rc: 1 start: '2022-05-31 10:02:00.087383' stderr: |- + source /tmp/workspace/venv-cp37-cp37m/bin/activate ++ deactivate nondestructive ++ '[' -n '' ']' ++ '[' -n '' ']' ++ '[' -n /bin/bash -o -n '' ']' ++ hash -r ++ '[' -n '' ']' ++ unset VIRTUAL_ENV ++ '[' '!' nondestructive = nondestructive ']' ++ VIRTUAL_ENV=/tmp/workspace/venv-cp37-cp37m ++ export VIRTUAL_ENV ++ _OLD_VIRTUAL_PATH=/opt/rh/devtoolset-10/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ++ PATH=/tmp/workspace/venv-cp37-cp37m/bin:/opt/rh/devtoolset-10/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ++ export PATH ++ '[' -n '' ']' ++ '[' -z '' ']' ++ _OLD_VIRTUAL_PS1= ++ '[' 'x(venv-cp37-cp37m) ' '!=' x ']' ++ PS1='(venv-cp37-cp37m) ' ++ export PS1 ++ '[' -n /bin/bash -o -n '' ']' ++ hash -r + source ./linklibs.sh + source tensorflow/tools/ci_build/build_scripts/DEFAULT_TEST_TARGETS.sh ++ set -x ++ DEFAULT_BAZEL_TARGETS='//tensorflow/... -//tensorflow/compiler/tf2tensorrt/... -//tensorflow/compiler/xrt/... -//tensorflow/core/tpu/... -//tensorflow/go/... -//tensorflow/java/... -//tensorflow/python/integration_testing/... -//tensorflow/tools/toolchains/...' + bazel test --test_timeout=300,500,-1,-1 --flaky_test_attempts=3 --test_output=all --cache_test_results=no --config=nonccl --config=mkl_aarch64 --copt=-fopenmp --linkopt=-lgomp --copt=-mtune=generic --copt=-march=armv8-a --copt=-O3 --verbose_failures --test_env=TF_ENABLE_ONEDNN_OPTS=1 --build_tag_filters=-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-no_aarch64,-requires-gpu --test_tag_filters=-no_oss,-oss_serial,-gpu,-tpu,-benchmark-test,-v1only,-no_aarch64,-requires-gpu --build_tests_only -- //tensorflow/... -//tensorflow/compiler/tf2tensorrt/... -//tensorflow/compiler/xrt/... -//tensorflow/core/tpu/... -//tensorflow/go/... -//tensorflow/java/... -//tensorflow/python/integration_testing/... -//tensorflow/tools/toolchains/... -//tensorflow/lite/... -//tensorflow/compiler/mlir/lite/tests:const-fold.mlir.test -//tensorflow/core/distributed_runtime/integration_test:c_api_session_coordination_test_cpu -//tensorflow/python/client:session_list_devices_test -//tensorflow/python/data/experimental/kernel_tests/service:cross_trainer_cache_test -//tensorflow/python/data/kernel_tests:iterator_test_cpu -//tensorflow/python/training:server_lib_test -//tensorflow/compiler/mlir/tensorflow/tests/... -//tensorflow/compiler/mlir/tfr/examples/mnist:mnist_ops_test -//tensorflow/core/grappler/optimizers:auto_mixed_precision_test_cpu -//tensorflow/core/grappler/optimizers:constant_folding_test -//tensorflow/core/grappler/optimizers:remapper_test_cpu -//tensorflow/core/kernels:conv_ops_test_cpu -//tensorflow/core/kernels:matmul_op_test_cpu -//tensorflow/python/eager:forwardprop_test_cpu -//tensorflow/python/keras/engine:training_arrays_test -//tensorflow/python/kernel_tests/linalg:linear_operator_block_diag_test_cpu -//tensorflow/python/kernel_tests/linalg:linear_operator_block_lower_triangular_test_cpu -//tensorflow/python/kernel_tests/linalg:linear_operator_householder_test_cpu -//tensorflow/python/kernel_tests/linalg:linear_operator_inversion_test_cpu -//tensorflow/python/kernel_tests/linalg:linear_operator_kronecker_test_cpu -//tensorflow/python/kernel_tests/math_ops:batch_matmul_op_test_cpu -//tensorflow/python/kernel_tests/nn_ops:conv2d_backprop_filter_grad_test_cpu -//tensorflow/python/kernel_tests/nn_ops:conv3d_backprop_filter_v2_grad_test_cpu -//tensorflow/python/kernel_tests/nn_ops:conv_ops_test_cpu -//tensorflow/python/ops/parallel_for:math_test_cpu -//tensorflow/python:dequantize_op_test -//tensorflow/python:nn_grad_test_cpu -//tensorflow/python:quantized_ops_test Extracting Bazel installation... INFO: Options provided by the client: Inherited 'common' options: --isatty=0 --terminal_columns=80 INFO: Reading rc options for 'test' from /tmp/workspace/tensorflow-oneDNN-ACL-git/.bazelrc: Inherited 'common' options: --experimental_repo_remote_exec INFO: Reading rc options for 'test' from /tmp/workspace/tensorflow-oneDNN-ACL-git/.bazelrc: Inherited 'build' options: --define framework_shared_object=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=true INFO: Reading rc options for 'test' from /tmp/workspace/tensorflow-oneDNN-ACL-git/.tf_configure.bazelrc: Inherited 'build' options: --action_env PYTHON_BIN_PATH=/tmp/workspace/venv-cp37-cp37m/bin/python3 --action_env PYTHON_LIB_PATH=/tmp/workspace/venv-cp37-cp37m/lib/python3.7/site-packages --python_path=/tmp/workspace/venv-cp37-cp37m/bin/python3 INFO: Reading rc options for 'test' from /tmp/workspace/tensorflow-oneDNN-ACL-git/.bazelrc: Inherited 'build' options: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/common,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils INFO: Reading rc options for 'test' from /tmp/workspace/tensorflow-oneDNN-ACL-git/.tf_configure.bazelrc: 'test' options: --flaky_test_attempts=3 --test_size_filters=small,medium INFO: Found applicable config definition build:short_logs in file /tmp/workspace/tensorflow-oneDNN-ACL-git/.bazelrc: --output_filter=DONT_MATCH_ANYTHING INFO: Found applicable config definition build:v2 in file /tmp/workspace/tensorflow-oneDNN-ACL-git/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1 INFO: Found applicable config definition test:v2 in file /tmp/workspace/tensorflow-oneDNN-ACL-git/.tf_configure.bazelrc: --test_tag_filters=-benchmark-test,-no_oss,-gpu,-oss_serial,-v1only --build_tag_filters=-benchmark-test,-no_oss,-gpu,-v1only INFO: Found applicable config definition build:nonccl in file /tmp/workspace/tensorflow-oneDNN-ACL-git/.bazelrc: --define=no_nccl_support=true INFO: Found applicable config definition build:mkl_aarch64 in file /tmp/workspace/tensorflow-oneDNN-ACL-git/.bazelrc: --define=build_with_mkl_aarch64=true --define=build_with_openmp=true -c opt INFO: Found applicable config definition build:linux in file /tmp/workspace/tensorflow-oneDNN-ACL-git/.bazelrc: --copt=-w --host_copt=-w --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --distinct_host_configuration=false --experimental_guard_against_concurrent_changes INFO: Found applicable config definition build:dynamic_kernels in file /tmp/workspace/tensorflow-oneDNN-ACL-git/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS Loading: Loading: 0 packages loaded Loading: 0 packages loaded ERROR: Skipping '//tensorflow/python/keras/engine:training_arrays_test': no such target '//tensorflow/python/keras/engine:training_arrays_test': target 'training_arrays_test' not declared in package 'tensorflow/python/keras/engine' defined by /tmp/workspace/tensorflow-oneDNN-ACL-git/tensorflow/python/keras/engine/BUILD ERROR: no such target '//tensorflow/python/keras/engine:training_arrays_test': target 'training_arrays_test' not declared in package 'tensorflow/python/keras/engine' defined by /tmp/workspace/tensorflow-oneDNN-ACL-git/tensorflow/python/keras/engine/BUILD INFO: Elapsed time: 6.797s INFO: 0 processes. FAILED: Build did NOT complete successfully (1 packages loaded) ERROR: Couldn't start the build. Unable to run tests FAILED: Build did NOT complete successfully (1 packages loaded) stderr_lines: <omitted> stdout: '' stdout_lines: <omitted>
PLAY RECAP ********************************************************************* localhost : ok=53 changed=33 unreachable=0 failed=1 skipped=8 rescued=0 ignored=0
Tuesday 31 May 2022 10:02:07 +0000 (0:00:07.457) 0:50:01.351 *********** =============================================================================== tensorflow : build TensorFlow-oneDNN-ACL git for cp37-cp37m ---------- 2188.50s python : build wheels of Python package h5py~=3.6.0 ------------------- 272.57s python : build wheels of Python package Keras_Applications ------------ 270.02s tensorflow : populate cp37-cp37m virtualenv ---------------------------- 54.07s install bazel build requirements --------------------------------------- 45.89s python : install numpy build requirements ------------------------------ 28.49s fetch tensorflow-oneDNN-ACL from git ----------------------------------- 22.53s python : install h5py build requirements ------------------------------- 17.01s python : create cp37-cp37m virtualenv ---------------------------------- 13.90s python : build wheels of Python package grpcio ------------------------- 12.27s install tensorflow build requirements ----------------------------------- 7.57s python : install scipy-1.7.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl Python package --- 7.50s tensorflow : Run unit tests on TensorFlow-oneDNN-ACL git for cp37-cp37m --- 7.46s python : build wheels of Python package scipy --------------------------- 6.87s python : build wheels of Python package numpy>=1.20 --------------------- 4.76s tensorflow : remove TensorFlow-oneDNN-ACL git sources from previous build --- 4.75s Gathering Facts --------------------------------------------------------- 4.61s python : build wheels of Python package Keras_Preprocessing ------------- 4.20s python : enable EPEL ---------------------------------------------------- 3.07s python : install h5py-3.6.0-cp37-cp37m-linux_aarch64.whl Python package --- 2.96s Build step 'Execute shell' marked build as failure