ldcg-ai-ml May 2024

ldcg-ai-ml@op-lists.linaro.org

1 participants
1 discussions

Build failed in Jenkins: Tensorflow oneDNN Python package (manylinux2014 nightly) #807

by ci_notify＠linaro.org

See <https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-onednn-nightly/8…> Changes: ------------------------------------------ [...truncated 7.14 KiB...] TASK [tensorflow : Record git commit hash to variable] ************************* Tuesday 21 May 2024 01:11:31 +0100 (0:00:00.035) 0:01:26.096 *********** ok: [localhost] TASK [tensorflow : Save git commit hash to file] ******************************* Tuesday 21 May 2024 01:11:31 +0100 (0:00:00.055) 0:01:26.152 *********** changed: [localhost] TASK [Build TensorFlow for each Python version (dual abi)] ********************* Tuesday 21 May 2024 01:11:33 +0100 (0:00:01.107) 0:01:27.260 *********** TASK [tensorflow_in_container : Reset checkout to clean state] ***************** Tuesday 21 May 2024 01:11:33 +0100 (0:00:00.433) 0:01:27.693 *********** changed: [localhost] TASK [tensorflow_in_container : Remove TensorFlow-oneDNN-ACL git sources from previous build] *** Tuesday 21 May 2024 01:11:34 +0100 (0:00:00.650) 0:01:28.344 *********** skipping: [localhost] TASK [Fetch TensorFlow] ******************************************************** Tuesday 21 May 2024 01:11:34 +0100 (0:00:00.029) 0:01:28.374 *********** skipping: [localhost] TASK [(Python 3.9) patch TensorFlow tree] ************************************** Tuesday 21 May 2024 01:11:34 +0100 (0:00:00.030) 0:01:28.405 *********** TASK [tensorflow : Create linklibs shell script] ******************************* Tuesday 21 May 2024 01:11:34 +0100 (0:00:00.081) 0:01:28.487 *********** changed: [localhost] TASK [tensorflow : Use nightly package requirements] *************************** Tuesday 21 May 2024 01:11:34 +0100 (0:00:00.686) 0:01:29.173 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) copy upper constraints for hermetic builds] *** Tuesday 21 May 2024 01:11:35 +0100 (0:00:00.632) 0:01:29.805 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) build Python versioned container] *** Tuesday 21 May 2024 01:11:36 +0100 (0:00:00.682) 0:01:30.488 *********** ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) start build container] ************ Tuesday 21 May 2024 01:11:37 +0100 (0:00:01.717) 0:01:32.205 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir] *** Tuesday 21 May 2024 01:11:56 +0100 (0:00:18.198) 0:01:50.404 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for bazel cache] ****** Tuesday 21 May 2024 01:11:57 +0100 (0:00:01.212) 0:01:51.617 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for bazelisk cache] *** Tuesday 21 May 2024 01:11:58 +0100 (0:00:00.842) 0:01:52.460 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create link for pip cache] ******** Tuesday 21 May 2024 01:11:59 +0100 (0:00:00.816) 0:01:53.276 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build] *** Tuesday 21 May 2024 01:11:59 +0100 (0:00:00.899) 0:01:54.176 *********** changed: [localhost] TASK [tensorflow_in_container : Show list of installed Python packages for build] *** Tuesday 21 May 2024 01:12:03 +0100 (0:00:03.435) 0:01:57.612 *********** ok: [localhost] => pip_list_output.stdout: |- Package Version Location Installer ------------------- -------------------- -------------------------------------- --------- auditwheel 5.4.0 /usr/local/lib/python3.9/dist-packages pip certifi 2019.11.28 /usr/lib/python3/dist-packages cffi 1.16.0 /usr/local/lib/python3.9/dist-packages pip chardet 3.0.4 /usr/lib/python3/dist-packages cryptography 42.0.5 /usr/local/lib/python3.9/dist-packages pip Cython 3.0.9 /usr/local/lib/python3.9/dist-packages pip dbus-python 1.2.16 /usr/lib/python3/dist-packages docutils 0.20.1 /usr/local/lib/python3.9/dist-packages pip idna 2.8 /usr/lib/python3/dist-packages importlib-metadata 7.0.1 /usr/local/lib/python3.9/dist-packages pip jaraco.classes 3.3.1 /usr/local/lib/python3.9/dist-packages pip jeepney 0.8.0 /usr/local/lib/python3.9/dist-packages pip keyring 24.3.1 /usr/local/lib/python3.9/dist-packages pip markdown-it-py 3.0.0 /usr/local/lib/python3.9/dist-packages pip mdurl 0.1.2 /usr/local/lib/python3.9/dist-packages pip more-itertools 10.2.0 /usr/local/lib/python3.9/dist-packages pip nh3 0.2.15 /usr/local/lib/python3.9/dist-packages pip pip 24.0 /usr/local/lib/python3.9/dist-packages pip pkginfo 1.10.0 /usr/local/lib/python3.9/dist-packages pip portpicker 1.6.0 /usr/local/lib/python3.9/dist-packages pip psutil 5.9.8 /usr/local/lib/python3.9/dist-packages pip pycparser 2.21 /usr/local/lib/python3.9/dist-packages pip pyelftools 0.30 /usr/local/lib/python3.9/dist-packages pip Pygments 2.17.2 /usr/local/lib/python3.9/dist-packages pip PyGObject 3.36.0 /usr/lib/python3/dist-packages python-apt 2.0.1+ubuntu0.20.4.1 /usr/lib/python3/dist-packages readme_renderer 43.0 /usr/local/lib/python3.9/dist-packages pip requests 2.22.0 /usr/lib/python3/dist-packages requests-toolbelt 1.0.0 /usr/local/lib/python3.9/dist-packages pip requests-unixsocket 0.2.0 /usr/lib/python3/dist-packages rfc3986 2.0.0 /usr/local/lib/python3.9/dist-packages pip rich 13.7.1 /usr/local/lib/python3.9/dist-packages pip SecretStorage 3.3.3 /usr/local/lib/python3.9/dist-packages pip setuptools 69.1.1 /usr/local/lib/python3.9/dist-packages pip six 1.14.0 /usr/lib/python3/dist-packages twine 4.0.2 /usr/local/lib/python3.9/dist-packages pip urllib3 2.2.1 /usr/local/lib/python3.9/dist-packages pip wheel 0.42.0 /usr/local/lib/python3.9/dist-packages pip zipp 3.17.0 /usr/local/lib/python3.9/dist-packages pip TASK [tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git] *** Tuesday 21 May 2024 01:12:03 +0100 (0:00:00.037) 0:01:57.649 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) build Python package] ************* Tuesday 21 May 2024 02:45:32 +0100 (1:33:28.823) 1:35:26.473 *********** skipping: [localhost] TASK [tensorflow_in_container : (Python 3.9) Copy built wheel] ***************** Tuesday 21 May 2024 02:45:32 +0100 (0:00:00.032) 1:35:26.505 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) cleanup Bazel install directory] *** Tuesday 21 May 2024 02:45:33 +0100 (0:00:01.010) 1:35:27.516 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) create script to run tests] ******* Tuesday 21 May 2024 02:45:34 +0100 (0:00:00.964) 1:35:28.481 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) run TensorFlow-oneDNN-ACL git tests] *** Tuesday 21 May 2024 02:45:34 +0100 (0:00:00.706) 1:35:29.187 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) collect list of installed Python packages] *** Tuesday 21 May 2024 04:54:45 +0100 (2:09:10.985) 3:44:40.173 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save list of installed Python packages] *** Tuesday 21 May 2024 04:54:48 +0100 (0:00:02.215) 3:44:42.388 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save tests output to file] ******** Tuesday 21 May 2024 04:54:48 +0100 (0:00:00.712) 3:44:43.101 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) save tests stderr output to file] *** Tuesday 21 May 2024 04:54:49 +0100 (0:00:00.712) 3:44:43.813 *********** changed: [localhost] TASK [tensorflow_in_container : (Python 3.9) Extract number of tests run] ****** Tuesday 21 May 2024 04:54:50 +0100 (0:00:00.724) 3:44:44.537 *********** ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) Show number of tests run] ********* Tuesday 21 May 2024 04:54:50 +0100 (0:00:00.079) 3:44:44.617 *********** ok: [localhost] => ut_tests_line: 'Executed 3085 out of 3085 tests: 3085 tests pass.' TASK [tensorflow_in_container : (Python 3.9) Extract failed tests] ************* Tuesday 21 May 2024 04:54:50 +0100 (0:00:00.041) 3:44:44.658 *********** ok: [localhost] TASK [tensorflow_in_container : (Python 3.9) Show which tests are flaky] ******* Tuesday 21 May 2024 04:54:50 +0100 (0:00:00.131) 3:44:44.790 *********** ok: [localhost] => tests_flaky: - //tensorflow/core/kernels:batch_kernels_test FLAKY, failed in 1 out of 2 in 48.5s - //tensorflow/core/kernels/mkl:mkl_qmatmul_op_test FLAKY, failed in 1 out of 2 in 0.5s TASK [tensorflow_in_container : (Python 3.9) Show which tests failed] ********** Tuesday 21 May 2024 04:54:50 +0100 (0:00:00.041) 3:44:44.832 *********** ok: [localhost] => tests_failed: [] TASK [tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one] *** Tuesday 21 May 2024 04:54:50 +0100 (0:00:00.041) 3:44:44.873 *********** fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- + for wheel in /tf/pkg/*.whl + echo 'Checking and renaming /tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl...' + python3 -m auditwheel repair --plat manylinux2014_aarch64 /tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl --wheel-dir /tf/pkg + tee check.txt real 1m28.486s user 1m10.791s sys 0m17.263s stderr_lines: <omitted> stdout: |- Checking and renaming /tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl... INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/auditwheel/repair.py", line 80, in repair_wheel raise ValueError( ValueError: Cannot repair wheel, because required library "_pywrap_tensorflow_internal.so" could not be located During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.9/dist-packages/auditwheel/__main__.py", line 8, in <module> sys.exit(main()) File "/usr/local/lib/python3.9/dist-packages/auditwheel/main.py", line 59, in main rval = args.func(args, p) File "/usr/local/lib/python3.9/dist-packages/auditwheel/main_repair.py", line 175, in execute out_wheel = repair_wheel( File "/usr/local/lib/python3.9/dist-packages/auditwheel/repair.py", line 118, in repair_wheel strip_symbols(itertools.chain(libs_to_strip, extensions)) File "/usr/local/lib/python3.9/dist-packages/auditwheel/wheeltools.py", line 137, in __exit__ dir2zip(self.name, self.out_wheel, date_time) File "/usr/local/lib/python3.9/dist-packages/auditwheel/tools.py", line 76, in dir2zip with zipfile.ZipFile(zip_fname, "w", compression=compression) as z: File "/usr/lib/python3.9/zipfile.py", line 1248, in __init__ self.fp = io.open(file, filemode) PermissionError: [Errno 13] Permission denied: '/tf/pkg/tensorflow_aarch64-2.17.0-cp39-cp39-linux_aarch64.whl' stdout_lines: <omitted> PLAY RECAP ********************************************************************* localhost : ok=39 changed=26 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0 Tuesday 21 May 2024 04:56:19 +0100 (0:01:29.341) 3:46:14.214 *********** =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow-oneDNN-ACL git tests 7750.99s tensorflow_in_container : (Python 3.9) build TensorFlow-oneDNN-ACL git 5608.82s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one -- 89.34s Cleanup after previous build ------------------------------------------- 47.84s tensorflow : Fetch TensorFlow-oneDNN-ACL from git ---------------------- 23.31s tensorflow_in_container : (Python 3.9) start build container ----------- 18.20s tensorflow : Remove TensorFlow-oneDNN-ACL git sources from previous build --- 9.40s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 3.44s Gathering Facts --------------------------------------------------------- 2.66s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 2.22s tensorflow_in_container : (Python 3.9) build Python versioned container --- 1.72s Create cache dirs ------------------------------------------------------- 1.44s tensorflow_in_container : (Python 3.9) create buildslave bazel cache dir --- 1.21s tensorflow : Save git commit hash to file ------------------------------- 1.11s tensorflow_in_container : (Python 3.9) Copy built wheel ----------------- 1.01s tensorflow_in_container : (Python 3.9) cleanup Bazel install directory --- 0.96s tensorflow_in_container : (Python 3.9) create link for pip cache -------- 0.90s tensorflow_in_container : (Python 3.9) create link for bazel cache ------ 0.84s tensorflow_in_container : (Python 3.9) create link for bazelisk cache --- 0.82s tensorflow_in_container : (Python 3.9) save tests stderr output to file --- 0.72s Build step 'Execute shell' marked build as failure

14 hours, 34 minutes

2024

2023

2022

2021

ldcg-ai-ml May 2024