test_tarfile: test_extractall_symlinks() fails on Windows with: [WinError 206] The filename or extension is too long

[vstinner]

BPO	37528
Nosy	@pfmoore, @vstinner, @tjguk, @zware, @eryksun, @zooba, @pablogsal

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-07-09.11:34:17.760>
labels = ['tests', '3.9', 'OS-windows']
title = 'test_tarfile: test_extractall_symlinks() fails on Windows with: [WinError 206] The filename or extension is too long'
updated_at = <Date 2019-08-22.09:39:08.645>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-08-22.09:39:08.645>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests', 'Windows']
creation = <Date 2019-07-09.11:34:17.760>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 37528
keywords = []
message_count = 5.0
messages = ['347549', '347550', '347597', '350143', '350177']
nosy_count = 7.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'zach.ware', 'eryksun', 'steve.dower', 'pablogsal']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue37528'
versions = ['Python 3.9']

[vstinner]

AMD64 Windows8.1 Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/12/builds/2862

FAIL: test_extractall_symlinks (test.test_tarfile.Bz2WriteTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_tarfile.py", line 1312, in test_extractall_symlinks
    tar.extractall(path=tempdir)
FileNotFoundError: [WinError 206] The filename or extension is too long: 'D:\\buildarea\\3.x.ware-win81-release.nondebug\\build\\build\\test_python_3164\\test_python_worker_5512\\@test_5512_tmp-tardir\\testsymlinks\\buildarea\\3.x.ware-win81-release.nondebug\\build\\build\\test_python_3164\\test_python_worker_5512\\@test_5512_tmp-tardir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.nondebug\build\lib\test\test_tarfile.py", line 1314, in test_extractall_symlinks
    self.fail("extractall failed with symlinked files")
AssertionError: extractall failed with symlinked files

Extract of pythoninfo:

windows.RtlAreLongPathsEnabled: <function not available>

RtlAreLongPathsEnabled() is not available on Windows 8.1. I don't know if long paths support is enabled or not on this buildbot (I guess that no, it isn't).

I'm not sure why the test started to fail.

[vstinner]

On this buildbot build, test_tarfile succeeded when run again. Likely because the test is run in the main regrtest process which doesn't have "test_python_worker_xxx" in its current working directory.

[eryksun]

If you want to harden the test for Windows, you could transform TEMPDIR
into an extended path. os.path._getfinalpathname always returns an extended path. For example:

    >>> os.path._getfinalpathname('C:/Temp')
    '\\\\?\\C:\\Temp'

    >>> os.path._getfinalpathname('//localhost/C$/Temp')
    '\\\\?\\UNC\\localhost\\C$\\Temp'

windows.RtlAreLongPathsEnabled: <function not available>

RtlAreLongPathsEnabled() is not available on Windows 8.1. I don't
know if long paths support is enabled or not on this buildbot (I
guess that no, it isn't).

Normalized long paths are implemented in Windows 10 1607 and later. Previous versions do not have RtlAreLongPathsEnabled and cannot support normalized long paths. RtlAreLongPathsEnabled() will be true if, at process startup, the "LongPathsEnabled" policy is enabled in the registry and the application manifest claims to be "longPathAware".

In all supported versions of Windows, long paths are supported by most filesystem functions if we use an extended path, i.e. a fully-qualified Unicode path that starts with the \\?\ prefix. This path type is not normalized in a create or open context, so it can only use backslash as the path separator, not forward slash. Also, the process working directory does not allow extended paths, so when using extended paths we have to manage our own working directory and manually resolve relative paths, including "." and ".." components (e.g. by calling GetFullPathNameW).

[zooba]

I disabled the long path support on my own build and can reproduce this by passing a long path to --tempdir

Unfortunately, I'm not familiar enough with what tarfile is trying to do here - why is it recreating the entire directory structure within itself?

Browsing through lib/tarfile.py, I suspect it's the mix of "/" literals and os.path functions that's at fault here. It probably requires a much bigger job to go through and fix it up than simply fixing tests.

[vstinner]

Hum, tests are still failing, example on AMD64 Windows8.1 Non-Debug 3.x:
https://buildbot.python.org/all/#/builders/12/builds/2992

[vstinner]

While it would be nice to fix this real bug, I don't have the bandwidth to work on this issue, so I just close it.

[pfmoore]

If it’s a genuine bug why not leave it open for someone else to pick up?

[vstinner]

The issue is about a buildbot failure. But Windows buildbots pass for months: the test_tarfile failure doesn't occur anymore. Maybe buildbots were upgraded to Windows 10 or Windows 11. Maybe paths used on Windows got shorter. This is not activity for 3 years, so I understand that nobody cares, or the bug was just fixed in the meanwhile.

More generally, I have limited bandwidth to fix issues specific to Windows, and yesterday I closed most old issues that I opened to help me tracking my "TODO lists": https://github.com/python/cpython/issues/vstinner. I had 3 pages, now it's only 1 page. It's easier to manage :-)

I don't know how to "reassign" an issue to someone else, so it's no longer in my https://github.com/python/cpython/issues/vstinner list.

Note: recently I fixed another bugs about long temporary paths on Unix, I found a simple fix, but the scope of the issue was more narrow: #93852