See https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/1036/display/redirect

Changes:

------------------------------------------ [...truncated 30.01 KB...] Monday 01 January 2024 21:35:05 +0000 (0:00:00.043) 3:25:01.381 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.10) build Python package] ************ Monday 01 January 2024 21:39:15 +0000 (0:04:10.095) 3:29:11.476 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.10) cleanup Bazel install directory] *** Monday 01 January 2024 21:41:20 +0000 (0:02:04.904) 3:31:16.380 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.10) create script to run tests] ****** Monday 01 January 2024 21:41:21 +0000 (0:00:00.906) 3:31:17.287 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.10) run TensorFlow git tests] ******** Monday 01 January 2024 21:41:22 +0000 (0:00:00.753) 3:31:18.040 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.10) collect list of installed Python packages] *** Monday 01 January 2024 22:15:41 +0000 (0:34:19.378) 4:05:37.419 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.10) save list of installed Python packages] *** Monday 01 January 2024 22:15:43 +0000 (0:00:01.791) 4:05:39.210 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.10) save tests output to file] ******* Monday 01 January 2024 22:15:44 +0000 (0:00:00.730) 4:05:39.941 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.10) save tests stderr output to file] *** Monday 01 January 2024 22:15:45 +0000 (0:00:00.692) 4:05:40.634 ******** changed: [localhost]

TASK [tensorflow_in_container : (Python 3.10) Extract number of tests run] ***** Monday 01 January 2024 22:15:45 +0000 (0:00:00.690) 4:05:41.324 ******** ok: [localhost]

TASK [tensorflow_in_container : (Python 3.10) Show number of tests run] ******** Monday 01 January 2024 22:15:45 +0000 (0:00:00.074) 4:05:41.398 ******** ok: [localhost] => ut_tests_line: 'Executed 3060 out of 3060 tests: 3060 tests pass.'

TASK [tensorflow_in_container : (Python 3.10) Extract failed tests] ************ Monday 01 January 2024 22:15:45 +0000 (0:00:00.043) 4:05:41.441 ******** ok: [localhost]

TASK [tensorflow_in_container : (Python 3.10) Show which tests are flaky] ****** Monday 01 January 2024 22:15:45 +0000 (0:00:00.123) 4:05:41.565 ******** ok: [localhost] => tests_flaky: - //tensorflow/c/eager/parallel_device:parallel_device_remote_test FLAKY, failed in 1 out of 2 in 39.8s

TASK [tensorflow_in_container : (Python 3.10) Show which tests failed] ********* Monday 01 January 2024 22:15:46 +0000 (0:00:00.043) 4:05:41.608 ******** ok: [localhost] => tests_failed: []

TASK [tensorflow_in_container : (Python 3.10) Convert wheel to manylinux2014 compatible one] *** Monday 01 January 2024 22:15:46 +0000 (0:00:00.074) 4:05:41.683 ******** fatal: [localhost]: FAILED! => changed=true rc: 1 stderr: |- + for wheel in /tf/pkg/*.whl + echo 'Checking and renaming /tf/pkg/tensorflow_aarch64-2.16.0-cp310-cp310-linux_aarch64.whl...' + python3 -m auditwheel repair --plat manylinux2014_aarch64 /tf/pkg/tensorflow_aarch64-2.16.0-cp310-cp310-linux_aarch64.whl --wheel-dir /tf/pkg + tee check.txt

real 2m25.656s user 2m9.705s sys 0m15.862s ++ grep --extended-regexp --only-matching '/tf/pkg/\S+.whl' check.txt + new_wheel=/tf/pkg/tensorflow_aarch64-2.16.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + [[ /tf/pkg/tensorflow_aarch64-2.16.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl != /\t\f/\p\k\g/\t\e\n\s\o\r\f\l\o\w_\a\a\r\c\h\6\4-\2.\1\6.\0-\c\p\3\1\0-\c\p\3\1\0-\l\i\n\u\x_\a\a\r\c\h\6\4.\w\h\l ]] + rm /tf/pkg/tensorflow_aarch64-2.16.0-cp310-cp310-linux_aarch64.whl + wheel=/tf/pkg/tensorflow_aarch64-2.16.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + TF_WHEEL=/tf/pkg/tensorflow_aarch64-2.16.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl + bats /usertools/wheel_verification.bats --timing stderr_lines: <omitted> stdout: |- Checking and renaming /tf/pkg/tensorflow_aarch64-2.16.0-cp310-cp310-linux_aarch64.whl... INFO:auditwheel.main_repair:Repairing tensorflow_aarch64-2.16.0-cp310-cp310-linux_aarch64.whl INFO:auditwheel.wheeltools:Previous filename tags: linux_aarch64 INFO:auditwheel.wheeltools:New filename tags: manylinux_2_17_aarch64, manylinux2014_aarch64 INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp310-cp310-linux_aarch64 INFO:auditwheel.wheeltools:New WHEEL info tags: cp310-cp310-manylinux_2_17_aarch64, cp310-cp310-manylinux2014_aarch64 INFO:auditwheel.main_repair: Fixed-up wheel written to /tf/pkg/tensorflow_aarch64-2.16.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 1..6 ok 1 Wheel is manylinux2014 (manylinux_2_17) compliant in 63810ms # Size of /tf/pkg/tensorflow_aarch64-2.16.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl is 259 / 580 megabytes. ok 2 Wheel conforms to upstream size limitations in 14ms not ok 3 Wheel is installable in 27300ms # (in test file /usertools/wheel_verification.bats, line 61) # `python3 -m pip install "$TF_WHEEL"' failed with status 2 # Requirement already satisfied: setuptools in /tf/venv/lib/python3.10/site-packages (65.5.0) # Collecting setuptools # Using cached setuptools-69.0.3-py3-none-any.whl (819 kB) # Collecting wheel # Using cached wheel-0.42.0-py3-none-any.whl (65 kB) # Installing collected packages: wheel, setuptools # Attempting uninstall: setuptools # Found existing installation: setuptools 65.5.0 # Uninstalling setuptools-65.5.0: # Successfully uninstalled setuptools-65.5.0 # Successfully installed setuptools-69.0.3 wheel-0.42.0 # # [notice] A new release of pip is available: 23.0.1 -> 23.3.2 # [notice] To update, run: pip install --upgrade pip # Processing ./tensorflow_aarch64-2.16.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl # Collecting protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 # Using cached protobuf-4.25.1-cp37-abi3-manylinux2014_aarch64.whl (293 kB) # Collecting astunparse>=1.6.0 # Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB) # Collecting tensorflow-io-gcs-filesystem>=0.23.1 # Downloading tensorflow_io_gcs_filesystem-0.35.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.0 MB) # ━━━━━━━━━━━━━╸ 1.0/3.0 MB 12.0 MB/s eta 0:00:01 # ERROR: Exception: # Traceback (most recent call last): # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher # yield # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 561, in read # data = self._fp_read(amt) if not fp_closed else b"" # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 527, in _fp_read # return self._fp.read(amt) if amt is not None else self._fp.read() # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 90, in read # data = self.__fp.read(amt) # File "/usr/lib/python3.10/http/client.py", line 466, in read # s = self.fp.read(amt) # File "/usr/lib/python3.10/socket.py", line 705, in readinto # return self._sock.recv_into(b) # File "/usr/lib/python3.10/ssl.py", line 1307, in recv_into # return self.read(nbytes, buffer) # File "/usr/lib/python3.10/ssl.py", line 1163, in read # return self._sslobj.read(len, buffer) # TimeoutError: The read operation timed out # # During handling of the above exception, another exception occurred: # # Traceback (most recent call last): # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper # status = run_func(*args) # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper # return func(self, options, args) # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 419, in run # requirement_set = resolver.resolve( # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve # result = self._result = resolver.resolve( # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve # state = resolution.resolve(requirements, max_rounds=max_rounds) # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 373, in resolve # failure_causes = self._attempt_to_pin_criterion(name) # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion # criteria = self._get_updated_criteria(candidate) # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 204, in _get_updated_criteria # self._add_to_criteria(criteria, requirement, parent=candidate) # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria # if not criterion.candidates: # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__ # return bool(self._sequence) # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__ # return any(self) # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr> # return (c for c in iterator if id(c) not in self._incompatible_ids) # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built # candidate = func() # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link # self._link_candidate_cache[link] = LinkCandidate( # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in __init__ # super().__init__( # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 162, in __init__ # self.dist = self._prepare() # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 231, in _prepare # dist = self._prepare_distribution() # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution # return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True) # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement # return self._prepare_linked_requirement(req, parallel_builds) # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 536, in _prepare_linked_requirement # local_file = unpack_url( # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 166, in unpack_url # file = get_http_url( # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 107, in get_http_url # from_path, content_type = download(link, temp_dir.path) # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/network/download.py", line 147, in __call__ # for chunk in chunks: # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/cli/progress_bars.py", line 53, in _rich_progress_bar # for chunk in iterable: # File "/tf/venv/lib/python3.10/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks # for chunk in response.raw.stream( # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 622, in stream # data = self.read(amt=amt, decode_content=decode_content) # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 560, in read # with self._error_catcher(): # File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ # self.gen.throw(typ, value, traceback) # File "/tf/venv/lib/python3.10/site-packages/pip/_vendor/urllib3/response.py", line 443, in _error_catcher # raise ReadTimeoutError(self._pool, None, "Read timed out.") # pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. # # [notice] A new release of pip is available: 23.0.1 -> 23.3.2 # [notice] To update, run: pip install --upgrade pip not ok 4 TensorFlow is importable in 43ms # (in test file /usertools/wheel_verification.bats, line 66) # `python3 -c 'import tensorflow as tf; t1=tf.constant([1,2,3,4]); t2=tf.constant([5,6,7,8]); print(tf.add(t1,t2).shape)'' failed # Traceback (most recent call last): # File "<string>", line 1, in <module> # ModuleNotFoundError: No module named 'tensorflow' not ok 5 TensorFlow has Keras in 42ms # (in test file /usertools/wheel_verification.bats, line 72) # `python3 -c 'import sys; import tensorflow as tf; sys.exit(0 if "keras" in tf.keras.__name__ else 1)'' failed # Traceback (most recent call last): # File "<string>", line 1, in <module> # ModuleNotFoundError: No module named 'tensorflow' not ok 6 TensorFlow has Estimator in 42ms # (in test file /usertools/wheel_verification.bats, line 78) # `python3 -c 'import sys; import tensorflow as tf; sys.exit(0 if "_v2.estimator" in tf.estimator.__name__ else 1)'' failed # Traceback (most recent call last): # File "<string>", line 1, in <module> # ModuleNotFoundError: No module named 'tensorflow' stdout_lines: <omitted>

PLAY RECAP ********************************************************************* localhost : ok=78 changed=54 unreachable=0 failed=1 skipped=12 rescued=0 ignored=0

Monday 01 January 2024 22:19:44 +0000 (0:03:58.098) 4:09:39.782 ******** =============================================================================== tensorflow_in_container : (Python 3.9) run TensorFlow git tests ------ 7090.41s tensorflow_in_container : (Python 3.9) build TensorFlow git ---------- 3764.94s tensorflow_in_container : (Python 3.10) run TensorFlow git tests ----- 2059.38s tensorflow_in_container : (Python 3.9) stop build container ----------- 931.34s tensorflow_in_container : (Python 3.9) Convert wheel to manylinux2014 compatible one - 292.46s tensorflow_in_container : (Python 3.10) build TensorFlow git ---------- 250.10s tensorflow_in_container : (Python 3.10) Convert wheel to manylinux2014 compatible one - 238.10s tensorflow_in_container : (Python 3.10) build Python package ---------- 124.90s tensorflow_in_container : (Python 3.9) build Python package ----------- 123.00s Cleanup after previous build ------------------------------------------- 30.31s tensorflow : Fetch TensorFlow from git --------------------------------- 19.99s tensorflow : Remove TensorFlow git sources from previous build ---------- 4.17s tensorflow_in_container : (Python 3.9) start build container ------------ 3.64s tensorflow_in_container : (Python 3.9) move manylinux2014 wheel outside --- 2.68s Gathering Facts --------------------------------------------------------- 2.62s tensorflow_in_container : (Python 3.9) collect list of installed Python packages for build --- 1.92s tensorflow_in_container : (Python 3.10) collect list of installed Python packages for build --- 1.85s tensorflow_in_container : (Python 3.10) collect list of installed Python packages --- 1.79s tensorflow_in_container : (Python 3.9) collect list of installed Python packages --- 1.78s tensorflow_in_container : (Python 3.10) start build container ----------- 1.63s Build step 'Execute shell' marked build as failure