Inconsistent virtual environment hashing between 1.1 and 1.2 on Windows

[davidgilbertson]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows 11
• Poetry version: 1.2.0

Issue

This might not be a bug, but after updating to a new version yesterday (1.1.15 -> 1.2.0), a new directory was created for my project (called "learning").

I use IntelliJ (IDEA, but same as PyCharm) to manage my environments, and after running some Poetry operation (I don't recall what) I started getting errors like this:

Which was confusing, because it was definitely installed (I could jump to source, see it in the side bar), but the message didn't go away after clicking install.

I think this is because the IDE was using virtualenvs\learning-8-kP5BBp-py3.10 but Poetry was using the new virtualenvs\learning-vjcf-xYS-py3.10 (confirmed by running poetry env info --path)

If I went to a terminal outside the IDE and ran poetry show --latest I got the (undocumented!) little (!) next to this library.

The fix

I went into the IntelliJ project settings and created a new "SDK" pointing at the new environment. Not sure if this is just for me or if all PyCharm users will have to do it. Is creating a new virtualenv the expected behaviour when upgrading Poetry?

[neersighted]

The hashing function should be stable -- is there any chance you might have moved your project on disk?

[davidgilbertson]

Nope, the only major change is a new GPU (from AMD to NVIDIA, that I just happened to swap right before I upgraded Poetry). I don't suppose the GPU is used for generating hashes?

[neersighted]

Nope, it's a simple alphanumeric encoding of sha256 if my memory serves... That's odd, but I'm not sure it's reproducible. Let me double check if someone changed that code.

[neersighted]

The relevant code is at

poetry/src/poetry/utils/env.py

Lines 1147 to 1154 in 4078082

	def generate_env_name(cls, name: str, cwd: str) -> str:
	name = name.lower()
	sanitized_name = re.sub(r'[ $`!*@"\\\r\n\t]', "_", name)[:42]
	normalized_cwd = os.path.normcase(os.path.realpath(cwd))
	h_bytes = hashlib.sha256(encode(normalized_cwd)).digest()
	h_str = base64.urlsafe_b64encode(h_bytes).decode()[:8]
	return f"{sanitized_name}-{h_str}"

It looks like what was in theory a correctness fix may result in different slashes on Windows, causing the inconsistent hash (that managed to skip the changelog): 601dc78 / #6110

[davidgilbertson]

Windows slashes strike again!

This is maybe also the cause of another issue I saw along the way. I have a local package called printi and saw this

Updating printi (0.1.0 C:\Users\david\py\printi\src -> 0.1.0 C:/Users/david/py/printi)

and thought "sure, fine, I guess they changed the slashes for this version, as long as it only happens once..."

[neersighted]

Poetry doesn't determine if a project is the same or not by path -- I think that's just inconsistent normalization in the printing code. A MR making it uniform would be welcome, but is not a priority I don't think. I would instead suspect the lack of a direct_url.json causing that update...

That being said, I'm not sure what to do about this hashing inconsistency -- if we fix it in a patch release, we'll just confuse people again. Certainly we can add a unit test to make sure this doesn't move in the future -- but as far as actually changing it back (with tests), I'm not sure.

[radoering]

It looks like what was in theory a correctness fix may result in different slashes on Windows, causing the inconsistent hash (that managed to skip the changelog): 601dc78 / #6110

More likely #4813. It's is a correctness fix and IMO we won't change that back (see the issues it resolved for more information). For the sake of avoiding virtualenv confusion in poetry 1.2, it's accepted that poetry 1.1 and 1.2 will use different virtualenvs on Windows.

[neersighted]

Fair point -- I'm fine with letting this lie then, as long as you feel the current test coverage is sufficient?

[radoering]

I think so. Both mentioned PRs have tests. It's clear that changes to generate_env_name() may change virtualenv names. It's always a tradeoff between correctness and consistency. I wouldn't change it in a patch release, but I think it's fine in a minor. (That said, I don't expect further changes in the near future.)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.