tmp is on a different filesystem: 'OSError: [Errno 18] Invalid cross-device link'

[vbabiy]

diff --git a/pip.py b/pip.py
--- a/pip.py
+++ b/pip.py
@@ -4714,7 +4714,7 @@
new_dir = os.path.dirname(new_path)
logger.info('Removing file or directory %s' % full_path)
self._moved_paths.append((prefix, path))

•                os.renames(full_path, new_path)
  

•                shutil.move(full_path, new_path)
           for pth in self.pth.values():
               pth.remove()
           logger.notify('Successfully uninstalled %s' % self.dist.project_name)
  

---

• Bitbucket: https://bitbucket.org/ianb/pip/issue/27
• Originally Reported By: Anonymous
• Originally Created At: 2009-11-13 14:18:19

[vbabiy]

diff --git a/pip.py b/pip.py

--- a/pip.py

+++ b/pip.py

@@ -4714,7 +4714,7 @@

                     new_dir = os.path.dirname(new_path)

                     logger.info('Removing file or directory %s' %

full_path)

                     self._moved_paths.append((prefix, path))

-                    os.renames(full_path, new_path)

+                    shutil.move(full_path, new_path)

                 for pth in self.pth.values():

                     pth.remove()

                 logger.notify('Successfully uninstalled %s' %

self.dist.project_name)

---

Original Comment By: Anonymous

[vbabiy]

Hmm, one problem is that we depend on the auto-creation of directories that
os.renames does for us, and shutil.move doesn't do that.

I'm also concerned about the loss of file metadata (permissions etc) that it
seems would happen if shutil.move had to resort to a copy (based on the
warning in shutil docs: http://docs.python.org/library/shutil.html).

We could try to create our own temp directory somewhere close to the
currently-installed location rather than in /tmp, but that seems like it might
have its own issues.

Not quite sure what to do here, input welcome.

---

Original Comment By: Carl Meyer

[vbabiy]

Further discussion: http://groups.google.com/group/python-
virtualenv/browse_thread/thread/18f5c08bd599288b

virtualenv/browse_thread/thread/18f5c08bd599288b

---

Original Comment By: Carl Meyer

[vbabiy]

There has been some discussion on list of this issue:

How portable does this need to be in terms of python versions?

shutil.copytree from python 2.5 handles copy then copystat which should
preserve permissions, etc

[1][1]

[2][2]

[3][3]

[4][4]

[5][5]

[6][6]



try:

   os.rename(src, dst)

except os.error, e:

   if e.errno == errno.EXDEV:

       shutil.copytree(src, dst, symlinks=True)

       shutil.rmtree(src)

---

Original Comment By: Paul Nasrat

[vbabiy]

I've worked around this by taking advantage of the fact that the tempfile
module will look at the environment, so I can

export TMPDIR=pwd/tmp

to use a local tmp directory, and things go fine.

---

Original Comment By: Anonymous

[vbabiy]

Oops, sorry that anonymous, was me.

---

Original Comment By: cdent

[vbabiy]

Issue #35 was marked as a duplicate of this issue.

device

---

Original Comment By: Carl Meyer

[vbabiy]

Ouch. #35 brings up the case of something called AFS where apparently
"every directory is presented to the system as a different device." And here I
was about ready to fix this by putting the save_dir in site-packages if
necessary; even that wouldn't help with AFS.

Patch on #35 is at http://lass.lfod.us/~will/pip-renames.diff --
uses shutil.move, so it still has the problem of potentially losing file
metadata. (Also, Paul Nasrat, shutil.copytree doesn't fully solve that problem
even if it uses copystat; file ownership information is still lost).

I may end up doing both: try putting the save_dir in site-packages to allow
use of os.renames on sane filesystems, and if even site-packages is still on a
different filesystem, resort to shutil.copytree.

device

---

Original Comment By: Carl Meyer

[vbabiy]

use shutil.move for rollback-save (fixes #27)

If TMPDIR is on a different filesystem, this has the potential to lose file
ownership info. In general that shouldn't be an issue; seems to be the best we
can do.

→ [Author: Carl Meyer

Age: 16 months ago

use shutil.move for rollback-save (fixes 27) If TMPDIR is on a different
filesystem, this has the potential to lose file ownership info. In general
that shouldn't be an issue; seems to be the best we can
do.">r398:4ae24807d08b]2

oserror

---

Original Comment By: Carl Meyer

[vbabiy]

Issue #52 was marked as a duplicate of this issue.

---

Original Comment By: Carl Meyer

[ghost]

This issue still occurs with pip 8.1.2 and python 3.5.2

See the following output from an attempt to install pgcli:

(pgcli) rdahlgren@athas:~/venvs $ pip --version
pip 8.1.2 from /home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages (python 3.5)
(pgcli) rdahlgren@athas:~/venvs $ python --version
Python 3.5.2
(pgcli) rdahlgren@athas:~/venvs $ pip install pgcli
Collecting pgcli
Collecting configobj>=5.0.6 (from pgcli)
Collecting setproctitle>=1.1.9 (from pgcli)
Collecting prompt-toolkit<1.1.0,>=1.0.0 (from pgcli)
  Using cached prompt_toolkit-1.0.5-py3-none-any.whl
Collecting psycopg2>=2.5.4 (from pgcli)
Collecting sqlparse==0.1.19 (from pgcli)
Requirement already satisfied (use --upgrade to upgrade): wcwidth>=0.1.6 in ./pgcli/lib/python3.5/site-packages (from pgcli)
Collecting pgspecial>=1.5.0 (from pgcli)
Requirement already satisfied (use --upgrade to upgrade): click>=4.1 in ./pgcli/lib/python3.5/site-packages (from pgcli)
Requirement already satisfied (use --upgrade to upgrade): Pygments>=2.0 in ./pgcli/lib/python3.5/site-packages (from pgcli)
Collecting humanize>=0.5.1 (from pgcli)
Requirement already satisfied (use --upgrade to upgrade): six in ./pgcli/lib/python3.5/site-packages (from configobj>=5.0.6->pgcli)
Installing collected packages: configobj, setproctitle, prompt-toolkit, psycopg2, sqlparse, pgspecial, humanize, pgcli
  Found existing installation: prompt-toolkit 0.46
    Uninstalling prompt-toolkit-0.46:
Exception:
Traceback (most recent call last):
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/shutil.py", line 538, in move
    os.rename(src, real_dst)
OSError: [Errno 18] Invalid cross-device link: '/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/prompt_toolkit-0.46.dist-info/DESCRIPTION.rst' -> '/tmp/pip-7zlbeq9d-uninstall/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/prompt_toolkit-0.46.dist-info/DESCRIPTION.rst'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/req/req_set.py", line 736, in install
    requirement.uninstall(auto_confirm=True)
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/req/req_install.py", line 742, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/home/rdahlgren/venvs/pgcli/lib/python3.5/shutil.py", line 553, in move
    os.unlink(src)
PermissionError: [Errno 13] Permission denied: '/home/rdahlgren/venvs/pgcli/lib/python3.5/site-packages/prompt_toolkit-0.46.dist-info/DESCRIPTION.rst'

My /tmp is a separate filesystem and flagged noexec, as is the long standing best practice for /tmp:

(pgcli) rdahlgren@athas:~/venvs $ cat /etc/fstab | grep '/tmp'
UUID=REDACTED /tmp          btrfs       rw,relatime,space_cache,noexec  0 0

[WhyNotHugo]

This is still an issue on pip 9.0.1. Why is this closed?

[Visdoom]

I also get this error with pip 9.0.1 when trying to install ipywidgets

[alexandrnikitin]

I also get this error. Can it be addressed please. Especially when it has a fix in comments already.

[paigeweber13]

I am having this problem when trying to upgrade pip from 9.0.3 to 10.0.1

[neofutur]

Same problem here,is this unfixed since 2011 ???

My /tmp is a separate filesystem and flagged noexec, as is the long standing best security practice for /tmp !

why not automatically use a local ( $pwd/tmp ) TMPDIR when /tmp is noexec and / or on a different filesystem ?

[paigeweber13]

I did eventually solve this problem on my end. For me, this was happening because I was running sudo pip install --upgrade pip, which I later learned is not the intended way for one to upgrade pip. I was told by one of the pip maintainers that using sudo with pip will not work by design, even though pip itself recommends using sudo when it fails to install a package due to insufficient permissions. (I have raised an issue to correct this in the pip repository) Because I was forcing pip to do something it was not designed to do, I ended up messing up my entire installation, causing both this error and leagues of other strange errors. I fixed the problem by completely removing python3 and pip3 from my computer, deleting any files left behind, and reinstalling both with my package manger. From then on, I've just used the --user flag so that you don't need root permission. Alternatively, you may let the operating system keep pip updated along with python.

@neofutur, what exactly where you doing to cause this error? It is possible that you did something similar to me.

[jeffsilverm]

I think I am having a similar error, although I am not sure if I still have the original error or if I am encountering a new error in the process of fixing the last error:

alias pip3='~/.local/bin/pip3'
        ~/.local/bin/pip3
(python3) [jq9877@lab-stacktools ~]$ pip3 --version
Traceback (most recent call last):
  File "/home/jq9877/.local/bin/pip3", line 7, in <module>
    from pip._internal import main
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/cli/cmdoptions.py", line 22, in <module>
    from pip._internal.utils.hashes import STRONG_HASHES
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/utils/hashes.py", line 10, in <module>
    from pip._internal.utils.misc import read_chunks
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/utils/misc.py", line 20, in <module>
    from pip._vendor import pkg_resources
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 574, in _build_master
    ws = cls()
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 567, in __init__
    self.add_entry(entry)
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 623, in add_entry
    for dist in find_distributions(entry, True):
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2056, in find_on_path
    for dist in factory(fullpath):
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2118, in distributions_from_metadata
    if len(os.listdir(path)) == 0:
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.4/site-packages/urllib3-1.22.dist-info'
(python3) [jq9877@lab-stacktools ~]$
(python3) [jq9877@lab-stacktools ~]$ pip3 --version -build-dir ~/python3
Traceback (most recent call last):
  File "/home/jq9877/.local/bin/pip3", line 7, in <module>
    from pip._internal import main
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/__init__.py", line 40, in <module>
    from pip._internal.cli.autocompletion import autocomplete
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
    from pip._internal.cli.main_parser import create_main_parser
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/cli/main_parser.py", line 8, in <module>
    from pip._internal.cli import cmdoptions
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/cli/cmdoptions.py", line 22, in <module>
    from pip._internal.utils.hashes import STRONG_HASHES
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/utils/hashes.py", line 10, in <module>
    from pip._internal.utils.misc import read_chunks
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_internal/utils/misc.py", line 20, in <module>
    from pip._vendor import pkg_resources
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3241, in <module>
    @_call_aside
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3225, in _call_aside
    f(*args, **kwargs)
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3254, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 574, in _build_master
    ws = cls()
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 567, in __init__
    self.add_entry(entry)
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 623, in add_entry
    for dist in find_distributions(entry, True):
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2056, in find_on_path
    for dist in factory(fullpath):
  File "/home/jq9877/.local/lib/python3.4/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2118, in distributions_from_metadata
    if len(os.listdir(path)) == 0:
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3.4/site-packages/urllib3-1.22.dist-info'
(python3) [jq9877@lab-stacktools ~]$

Rather frustrating - I don't have root privs on this machine.

Jeff

[pradyunsg]

Please look at #5599. It's an issue with how Debian repackages pip.