Improve performance of ntpath.isdir/isfile/exists/islink

[zooba]

These functions should use GetFileAttributesW for their fast paths, and the ones that traverse links can fall back to stat if a reparse point is found.

Linked PRs

• gh-102765: Updated isdir/isfile/islink/exists to use Py_GetFileInformationByName in ntpath when available #103485

[zooba]

See #101324 and #102149 for original implementation and later discussion.

[zooba]

the ones that traverse links can fall back to stat if a reparse point is found.

They're probably actually best to fall back to the existing code. It will be slightly more efficient than a new stat call, especially since we already know we need to traverse and our stat implementation will check that again.

[eryksun]

I started with the assumption that using GetFileAttributesW() would be the better choice. However, after comparing 3 implementations of ntpath.isdir(), I decided that using CreateFileW() was better overall -- at least for now. I posted sample code for the CreateFileW() implementation in issue #101196, which became the basis for PR #101324.

I implemented one version of ntpath.isdir() that called GetFileAttributesW() and another that used CreateFileW(), GetFileInformationByHandleEx() : FileBasicInfo, and CloseHandle(). The GetFileAttributesW() implementation was faster, but not by much. It executed in about 90% of the time compared to the CreateFileW() implementation. Most of the cost is in the IRP_MJ_CREATE request sent to the device stack, which is common to both implementations. Overall, however, I don't think such a small benefit for regular files and directories is worth the extra cost incurred for reparse points.

I also implemented ntpath.isdir() using NtQueryInformationByName(): FileStatInformation in order to get a feel for the expected performance of GetFileInformationByName()¹. It executed in half the time compared to the GetFileAttributesW() and CreateFileW() implementations². Since GetFileInformationByName() should be relatively cheap, I figured that eventually we'd update the code to use it where available.

Footnotes

1. I don't have GetFileInformationByName() on my Windows 11 test system (22H2, 10.0.22621, general availability channel), but I assume it's based on the NtQueryInformationByName() system call. ↩

2. In a kernel debugger, I see that the open packet created by NtQueryInformationByName() is flagged as a filter-query-open, whereas for NtQueryAttributesFile it's just a query-only open. Maybe a filter driver in the device stack handles the filter-query-open case and completes the IRP more efficiently. Whatever they did, the end result is that it's fast. ↩

[zooba]

The rest of your analysis is fantastic, as usual. But here's where we disagree:

Overall, however, I don't think such a small benefit for regular files and directories is worth the extra cost incurred for reparse points.

The main scenario this is likely to be impactful is enumerating large directories (many files/subdirectories), which are unlikely to be predominantly reparse points on a Windows machine. So I weight it more heavily, and come out about equal on the perf impacts (as in, both approaches seem fine).

However, I think GetFileAttributesW can and should be improved in the OS, at which point everyone using it gets the benefit. If we work around it now because that improvement hasn't been done, we will miss out on it later. I'd rather stick to the most focused OS API and advocate for having its implementation be improved (otherwise we'd be using Nt* APIs everywhere already). Maintainability on our side is also an important criteria.

So on balance, I prefer GetFileAttributesW with a slow fallback. If it turns out that we can't get the OS to improve their own APIs, we can switch. I might even consider switching to Nt* APIs at that point.

[eryksun]

So on balance, I prefer GetFileAttributesW with a slow fallback. If it turns out that we can't get the OS to improve their own APIs, we can switch. I might even consider switching to Nt* APIs at that point.

About 80% of desktop PCs use Windows 10, which wouldn't have the improved GetFileAttributesW() implementation. However, they'd still get stuck with the extra cost when checking reparse points.

I expected this issue to be created, but I expected it would be to use GetFileInformationByName(): FileBasicStatByNameInfo in a tiered implementation, or maybe using an internal STATX() implementation.

Given my experiment with using NtQueryInformationByName(): FileStatInformation, I assume that GetFileInformationByName() executes in half the time of the combination of CreateFileW(), GetFileInformationByHandleEx(), and CloseHandle(). We would get this benefit on updated Windows 11 systems without any cost to Windows 10 users. Also, in theory GetFileInformationByName() could support information classes that execute a bit faster by requesting less file information, such as FileBasicByNameInfo (file attributes and time stamps) or FileAttributeTagByNameInfo (file attributes and the reparse tag).

[zooba]

That's applying just as much "in theory" as I am, and we're talking about unreleased software that can still change, so all theories could be invalidated.

GetFileAttributesW is efficient today, just as efficient in prerelease versions, and has the potential to be improved further without any cost to us.

About the only alternative I would entertain is to go back to calling stat directly, now that it's faster. That perf improvement won't take effect until users upgrade their OS, but neither would calling GetFileInformationByName.

So for now, I want to stick with the more-efficient-than-before option everywhere, especially since it reduces our code repetition. Meanwhile, I'll try and get GetFileAttributesW optimised so that it matches GetFileInformationByName if it doesn't already (I haven't tried on the latest build I have access to). If that falls through, well, we just have another speed boost in the future.

[eryksun]

GetFileAttributesW is efficient today, just as efficient in prerelease versions, and has the potential to be improved further without any cost to us.

Do you think that GetFileAttributesW() would also be improved in Windows 10? I'd expect it to only get updated in Windows 11, in which case Windows 10 users are stuck with a significant penalty when checking reparse points for only a small improvement when checking regular files and directories (relatively to just using CreateFileW). That's an overall improvement only as long as reparse points are rare. I don't know about Python's use on systems with hierarchical storage, if hierarchical storage based on regular reparse points is even used anymore (as opposed to the placeholder reparse points that OneDrive uses, which are transparent to GetFileAttributesW, unless placeholders have been explicitly exposed).

About the only alternative I would entertain is to go back to calling stat directly, now that it's faster. That perf improvement won't take effect until users upgrade their OS, but neither would calling GetFileInformationByName.

STAT() calls GetFileType() (requests FileFsDeviceInformation) and GetFileInformationByHandle() (requests FileAllInformation and FileFsVolumeInformation). Those 3 system calls are overly expensive when all we need is the one system call to query the FileBasicInformation or FileAttributeTagInformation (for islink / isjunction).

If you mean going all he way back to os.stat(), some of the performance improvement of these accelerator functions stems from skipping the creation of the stat tuple and avoiding the cost of raising and handling an exception. But I haven't measured the savings due to returning just the boolean result.

[zooba]

Do you think that GetFileAttributesW() would also be improved in Windows 10?

Doubtful. But then your argument must be to leave the functions alone (at least for now), because GetFileInformationByName will also not appear in Windows 10.

If we're just going to prepare for future OS updates, we can hold off on any change here until it's clearer what the faster path will be. However, if we're going to assume that reparse points are common enough to need a fast path, CreateFile is going to be the only option anyway and we may as well stay there. It's only when the common case is that no traversal is required that we benefit from any of these options.

[eryksun]

if we're going to assume that reparse points are common enough to need a fast path, CreateFile is going to be the only option anyway

When the by-name check is very fast, as GetFileInformationByName() is, then it not only provides a big win for regular files, it's also a smaller penalty for reparse points.

But then your argument must be to leave the functions alone (at least for now), because GetFileInformationByName will also not appear in Windows 10.

Implementing a three-tier query that starts with _Py_GetFileInformationByName() won't change anything for older systems on which the latter immediately returns false. Windows 10 users would still have that benefit of switching from os.[l]stat() to the builtin functions that query less information, which improves performance by 15-30%.

---

What I was worried about with using GetFileAttributesW() is the cost for a filesystem tree that's predominantly reparse points. Maybe that's not a justified, practical concern, but the cost is significant for what I checked.

I just built and ran another comparison, this time of two implementations of os__path_isfile_impl(). I compared a three-tiered solution that first checks GetFileAttributesW() vs a two-tiered solution that starts with CreateFileW(). For a set of 50 regular files, checked 1000 times, the three-tiered GetFileAttributesW() implementation finished in 0.94 the time of the two-tiered implementation, so it was slightly faster. OTOH, for a set of 50 symlinks to regular files, checked 1000 times, the GetFileAttributesW() implementation took 1.53 times longer.

In fairness, the symlink case can also be faster when using GetFileAttributesW(). For example, an isfile() implementation that starts with GetFileAttributesW() doesn't have to traverse a directory reparse point. Whether or not the target exists doesn't matter. The system requires that a directory reparse point resolves to a directory, else it's invalid. So isfile() can just return false without having to call CreateFileW().

[DefaultRyan]

I'm loving all this analysis. Two questions about your tests:

• Were you running this on your main system volume, or a secondary volume?
• Did you have Defender (or similar) enabled, disabled, enabled with folder exclusions?

I can't speak to the speed of the final released implementation of GetFileInformationByName(), but in a performance trace, GetFileInformationByName(), has roughly ~10% overhead compared to NtQueryInformationByName(). Beware, this was a somewhat limited sample size, and I'm pretty sure it doesn't cover reparse points or other scenarios. Just that the non-nt API is somewhat similar in speed to the nt API in my experience.

(Also, this is coming from somebody that is not an expert, let alone owner, of the file system APIs. Just another dev interested in performance playing with these new shiny toys).

[zooba]

That 10% overhead seems like a lot, but I guess nobody is doing much work in this stack.

In any case, the main question isn't really whether to use a lower-level function or not, but how to balance the cost of handling reparse points vs. optimising for non-reparse points. After that, the question is whether the code complexity is worth optimising for non-reparse points with GetFileInformationByName as well as GetFileAttributes or just sticking to the latter (or, I guess, not optimising for non-reparse points at all unless GetFileInformationByName is present, which would be a slight regression on the current state of things but an improvement for fully updated OS).

[eryksun]

• Were you running this on your main system volume, or a secondary volume?
• Did you have Defender (or similar) enabled, disabled, enabled with folder exclusions?

Today I expanded my last test of isfile() to also check NtQueryInformationByName(). I used a set of 50 regular files and 50 symlinks to regular files in a directory that's on the system volume of a Windows 11 22H2 system. Defender was enabled normally, with no exclusions. I used a debug build of Python 3.12 to compare the current implementation of nt._path_isfile(), based solely on CreateFileW(), versus nt._path_isfile_gfa(), which starts with GetFileAttributesW(), versus nt._path_isfile_gfi(), which starts with My_GetFileInformationByName(). In one test I queried the regular files, and in the other test I queried the symlinks. Each case was run 1000 times via timeit.timeit(), and the best result out of three consecutive trials was used (but there was very little deviation between each trial). Here are the results, normalized to the shortest time for each set (regular, symlinks):

	CreateFileW()	GetFileAttributesW()	My_GetFileInformationByName()
Regular	2.31	2.12	1.00
Symlinks	1.00	1.46	1.19

Using My_GetFileInformationByName() has a nice tradeoff. It's very fast for regular files, and there's a small performance hit for symlinks. GetFileAttributesW() is only marginally faster than just using CreateFileW(), with worse performance for symlinks. The performance of GetFileAttributesW() could certainly be improved in a future release of Windows, but that won't help on Windows 10, which I think is going to have a significant share of desktops until 2026.

---

In Windows 11 22H2, I have neither GetFileInformationByName() nor FileStatBasicInformation, so I'm using NtQueryInformationByName() to query the FileStatInformation. The latter info class is a bit different. It queries the effective access, which incurs the cost of querying the file's security descriptor and calling SeAccessCheck(). OTOH, it doesn't query the volume serial number or device type.

My implementation of My_GetFileInformationByName() is just a basic wrapper around NtQueryInformationByName().

#include <winternl.h>

typedef struct _RTLP_CURDIR_REF
{
    LONG   RefCount;
    HANDLE Handle;
} RTLP_CURDIR_REF, *PRTLP_CURDIR_REF;

typedef struct _RTL_RELATIVE_NAME_U
{
    UNICODE_STRING   RelativeName;
    HANDLE           ContainingDirectory;
    PRTLP_CURDIR_REF CurDirRef;
} RTL_RELATIVE_NAME_U, *PRTL_RELATIVE_NAME_U;

FILE_INFORMATION_CLASS FileStatInformation = 68;

typedef struct _FILE_STAT_INFORMATION {
    LARGE_INTEGER FileId;
    LARGE_INTEGER CreationTime;
    LARGE_INTEGER LastAccessTime;
    LARGE_INTEGER LastWriteTime;
    LARGE_INTEGER ChangeTime;
    LARGE_INTEGER AllocationSize;
    LARGE_INTEGER EndOfFile;
    ULONG FileAttributes;
    ULONG ReparseTag;
    ULONG NumberOfLinks;
    ACCESS_MASK EffectiveAccess;
} FILE_STAT_INFORMATION, *PFILE_STAT_INFORMATION;

NTSTATUS (NTAPI *RtlDosPathNameToRelativeNtPathName_U_WithStatus)(
    PCWSTR                DosFileName,
    PUNICODE_STRING       NtFileName,
    PCWSTR               *FilePart,
    PRTL_RELATIVE_NAME_U  RelativeName) = NULL;

VOID (NTAPI *RtlSetLastWin32ErrorAndNtStatusFromNtStatus)(
    NTSTATUS Status) = NULL;

NTSTATUS (NTAPI *NtQueryInformationByName)(
    POBJECT_ATTRIBUTES      ObjectAttributes,
    PIO_STATUS_BLOCK        IoStatusBlock,
    PVOID                   FileInformation,
    ULONG                   Length,
    FILE_INFORMATION_CLASS  FileInformationClass) = NULL;

static BOOL
My_GetFileInformationByName(
    PCWSTR fileName,
    FILE_INFO_BY_NAME_CLASS fileInfoClass,
    PVOID fileInfoBuffer,
    ULONG fileInfoBufferSize)
{
    if (!NtQueryInformationByName) {
        HMODULE ntdll = GetModuleHandleW(L"ntdll");
        *(FARPROC *)&NtQueryInformationByName =
            GetProcAddress(ntdll, "NtQueryInformationByName");
        *(FARPROC *)&RtlDosPathNameToRelativeNtPathName_U_WithStatus =
            GetProcAddress(ntdll,
                "RtlDosPathNameToRelativeNtPathName_U_WithStatus");
        *(FARPROC *)&RtlSetLastWin32ErrorAndNtStatusFromNtStatus =
            GetProcAddress(ntdll,
                "RtlSetLastWin32ErrorAndNtStatusFromNtStatus");
    }

    if (!NtQueryInformationByName || fileInfoClass != FileStatByNameInfo) {
        SetLastError(ERROR_NOT_SUPPORTED);
        return FALSE;
    }

    NTSTATUS status;
    UNICODE_STRING objectName;
    RTL_RELATIVE_NAME_U relativeName;
    OBJECT_ATTRIBUTES obja;
    IO_STATUS_BLOCK iosb;

    status = RtlDosPathNameToRelativeNtPathName_U_WithStatus(
                fileName, &objectName, NULL, &relativeName);

    if (!NT_SUCCESS(status)) {
        RtlSetLastWin32ErrorAndNtStatusFromNtStatus(status);
        return FALSE;
    }

    if (relativeName.RelativeName.Length) {
        InitializeObjectAttributes(
            &obja, &relativeName.RelativeName, OBJ_CASE_INSENSITIVE,
            relativeName.ContainingDirectory, NULL);
    }
    else {
        InitializeObjectAttributes(
            &obja, &objectName, OBJ_CASE_INSENSITIVE, NULL, NULL);
    }

    status = NtQueryInformationByName(
                &obja, &iosb, fileInfoBuffer, fileInfoBufferSize,
                FileStatInformation);

    HeapFree(GetProcessHeap(), 0, objectName.Buffer);

    if (!NT_SUCCESS(status)) {
        RtlSetLastWin32ErrorAndNtStatusFromNtStatus(status);
        return FALSE;
    }
    return TRUE;
}

[zooba]

The performance of GetFileAttributesW() could certainly be improved in a future release of Windows, but that won't help on Windows 10, which I think is going to have a significant share of desktops until 2026.

This second thought is the crux of the matter. We don't currently allow ourselves to call into ntdll directly, which means none of these options help on Windows 10. So we have to decide that we don't care about Win10 perf (not unreasonable), we don't care about ntpath.is* perf, or we don't care about code duplication and maintainability (quite unreasonable).

Do you still have raw numbers for your table? Normalising each row separately means we can't figure out the ratio of regular files to links that would break even between any of these options.

[eryksun]

Here's another set of 10 trials, still based on a set of 50 files and a set 50 symlinks to those files, checked 1000 times per trial. This time it includes the results for genericpath.isfile(), the fallback implementation that's based on os.stat(). This time every result is normalized with respect to the fastest trial.

Best of 10

	os.stat()	CreateFileW()	GetFileAttributesW()	My_GetFileInformationByName()
Regular	4.43	2.19	2.01	1.00
Symlinks	5.35	3.02	5.59	3.77

Average of 10

	os.stat()	CreateFileW()	GetFileAttributesW()	My_GetFileInformationByName()
Regular	4.47	2.25	2.03	1.05
Symlinks	5.38	3.19	5.68	3.83

---

So we have to decide that we don't care about Win10 perf (not unreasonable), we don't care about ntpath.is* perf, or we don't care about code duplication and maintainability (quite unreasonable).

The current implementation of the nt._path_exists() and nt._path_is*() functions improve performance relative to calling os.[l]stat() anywhere from 10% to 30% according to #101324.

It states that the new isfile() scores a ratio of 0.81 for regular files and 0.87 for symlinks when compared to the fallback implementation. According to my tests, that's significantly underestimated. I consistently see a ratio of about 0.5 for regular files and about 0.6 for symlinks.

Windows 10 will still have this performance improvement if we add an initial step that tries Py_GetFileInformationByName() on Windows 11.

We don't currently allow ourselves to call into ntdll directly

I only shared My_GetFileInformationByName() for the sake of transparency and to maybe help shed light on the overhead. I don't think translating to an NT path accounts for most of the 10% overhead that Ryan observed, but I haven't measured it. Maybe there's added sanity checks and telemetry in the development implementation.

---

I just noticed that we need to fix a bug in the builtin functions. When passed a file descriptor, we get the handle via _Py_get_osfhandle_noraise() and query the file information regardless of the file type. If the file type isn't FILE_TYPE_DISK, it's only okay to query file information if we just opened the file. The query is synchronized on a lock in the kernel file object. This lock might already be held by another synchronous I/O operation, which could block indefinitely if it isn't a disk file (e.g. a pipe or console input).

[zooba]

This time every result is normalized with respect to the fastest trial.

Thanks. So what this says to me is that GetFileAttributesW is better than CreateFileW if we have at least 32 regular files for each link.¹ I think this is likely for cases where N is big enough to matter (i.e. <100 files is "fast enough" regardless, and >10K files is likely to be almost all real files)

But even in the >10K case, it only seems like an incremental improvement compared to what's coming in the future.

Windows 10 will still have this performance improvement if we add an initial step that tries Py_GetFileInformationByName() on Windows 11.

Yes, I'm coming around to this POV now. Win11 gets GetFileInformationByName and Win10 gets CreateFileW, which is what naturally falls out when we handle the error returned because the API is missing. It's essentially the same logic as stat is now using, just with the overheads removed.

Okay, let's go with that then. Same logic as is used in win32_xstat_impl just inlined in the same way as the original isdir change.

https://github.com/python/cpython/blob/main/Modules/posixmodule.c#L2036-L2042

I just noticed that we need to fix a bug in the builtin functions.

Did you file a new issue already or shall I?

Footnotes

1. 2.19R + 3.02S = 2.01R + 5.59S and solve for R/S where R is count of regular files and S is count of links ↩

[eryksun]

For implementers, note that a reparse point that's set on a directory is invalid if it resolves to a file, and a reparse point that's set on a file is invalid if it resolves to a directory. Thus if _Py_GetFileInformationByName() succeeds, then nt._path_isdir() only has to fall back on the slow path for a directory reparse point, and nt._path_isfile() only has to fall back on the slow path for a non-directory reparse point.

[zooba]

I implemented the benchmark myself¹ and tried it in a number of scenarios. Most of the time, GetFileAttributes is fastest on NTFS, the main exception being on the same drive as the operating system. My guess is that there are filesystem filter drivers installed that are being triggered by GetFileAttributes. I've passed the info along to the OS team to investigate.

Interestingly, I also noticed that every operation that goes via a mount point rather than a drive name is ridiculously slow. That it's slower isn't a surprise, but that it's over 100% slower was. Not a case we can do anything about, but felt it was worth mentioning.

Given the range of performance characteristics we may be dealing with here, I'm inclined to stick with the current implementation plan. I didn't actually test against the new API (don't have a machine handy right now), but since we wouldn't get any GetFileAttributes improvements before then, we'll end up with a single fast path and a single correct path, which is about as simple as we can make things.

Footnotes

1. Apart from Eryk's wrapper around NtQueryFileInformation which I simply borrowed. ↩

[eryksun]

Most of the time, GetFileAttributes is fastest on NTFS, the main exception being on the same drive as the operating system.

To clarify, your results are that GetFileAttributesW() is faster than using the wrapper that I wrote for NtqueryInformationByName() when tested on a drive other than the system drive? I only tested on the system drive.

[zooba]

your results are that GetFileAttributesW() is faster than using the wrapper that I wrote for NtqueryInformationByName() when tested on a drive other than the system drive

Yep. It varied in significance, and usually was so close as to be irrelevant, but was no slower in either files/symlinks case.

[eryksun]

I can confirm Steve's result. GetFileAttributesW() is slightly faster than NtQueryInformationByName() when tested on an NTFS volume other than the system volume. But performance on the system volume matters the most. It's the default location for program files, program data, user profiles, and temporary files.

Here's the result I get for testing isfile() on a set of files and symlinks on volume "C:" (system volume) vs a similar set of files and symlinks on volume "E:", and also compared to a mountpoint for volume "E:" on the system volume. The values are the best of 5, normalized to the shortest overall time.

	CreateFileW()	GetFileAttributesW()	My_GetFileInformationByName()
Regular (C:)	2.98	2.81	🟡 1.42
Regular (E:)	2.56	🟡 1.00	1.13
Regular (C:\Mount\volume_e)	4.09	🟡 2.50	3.02
Symlinks (C:)	🟡 4.71	7.73	5.27
Symlinks (E:)	🟡 3.47	4.04	4.26
Symlinks (C:\Mount\volume_e)	🟡 5.93	7.72	8.36

I expected traversing the mountpoint "C:\Mount\volume_e" to take more time since the system has to parse a path on volume "C:" (slower than other volumes, for whatever reason), and then restart parsing on volume "E:".

[barneygale]

Can this be resolved now that #103485 has landed?

[hugovk]

Can this be resolved now that #103485 has landed?

Let's assume "yes" and close this, we can always re-open it if necessary.