AttributeError: 'thread._local' object has no attribute 'indentation'

[drnextgis]

Traceback points here https://github.com/pypa/pip/blob/develop/pip/utils/logging.py#L39

[dstufft]

Can you give the entire error?

[dstufft]

Also what you were doing to trigger it.

[drnextgis]

I cannot reproduce it here, the following stacktrace sent me my colleague:

Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 846, in emit
    msg = self.format(record)
  File "/home/zadmin/ngw/env/local/lib/python2.7/site-packages/pip/utils/logging.py", line 103, in format
    msg = logging.StreamHandler.format(self, record)
  File "/usr/lib/python2.7/logging/__init__.py", line 723, in format
    return fmt.format(record)
  File "/home/zadmin/ngw/env/local/lib/python2.7/site-packages/pip/utils/logging.py", line 52, in format
    for line in formatted.splitlines(True)
  File "/home/zadmin/ngw/env/local/lib/python2.7/site-packages/pip/utils/logging.py", line 39, in get_indentation
    return _log_state.indentation
AttributeError: 'thread._local' object has no attribute 'indentation'
Logged from file channel.py, line 340
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/handlers.py", line 77, in emit
    if self.shouldRollover(record):
  File "/usr/lib/python2.7/logging/handlers.py", line 156, in shouldRollover
    msg = "%s\n" % self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 723, in format
    return fmt.format(record)
  File "/home/zadmin/ngw/env/local/lib/python2.7/site-packages/pip/utils/logging.py", line 52, in format
    for line in formatted.splitlines(True)
  File "/home/zadmin/ngw/env/local/lib/python2.7/site-packages/pip/utils/logging.py", line 39, in get_indentation
    return _log_state.indentation
AttributeError: 'thread._local' object has no attribute 'indentation'
Logged from file channel.py, line 340

This error occurs in Pyramid application installed within virtual environmen with the following command:

env/bin/pip install -e appname

When we downgraded pip to 1.5.4 this error was gone.

[airhorns]

Also seeing this issue after upgrading to pip 6. I think it has to do with logging from something other than the main thread if pip has been imported in a python process.

[balloob]

Would it be an idea to have get_indentation return 0 if indentation does not exist in current thread local?

This would do the job:

def get_indentation():
    return getattr(_log_state, 'indentation', 0)

[matthewhampton]

@balloob @dstufft I think this just masks the problem... The issue is that _log_state is a thread.local() object - I'm not sure why - and the indentation attribute is set in the global scope - i.e. it is only visible to the thread that first imports pip. Any other thread has a problem...

@dstufft Is there a reason that it is a thread local object?

[zjffdu]

I hit this issue in pip '9.0.1'.

Here's the code snippet I used

        def _run_pip(packages, iterator):
            import pip
            pip.main(["install"] + packages)

        if install_driver:
            import threading
            driver_thread = threading.Thread(target=_run_pip, args=(packages, None),
                                             name='driver_install_thread')
            driver_thread.start()

[jmdacruz]

Same here (pip 9.0.1). The issue seems to be that this line is not using get_indentation(...) and it's accessing the attribute directly: https://github.com/pypa/pip/blob/0da18e9/pip/utils/logging.py#L34

[monty6400]

So how can I fix this on the pip module?

[pradyunsg]

@zjffdu pip does not support in-process usage. If you want to invoke pip from a script, the only supported manner is to run it as a subprocess.

So how can I fix this on the pip module?

@monty6400 You just have to modify 2 lines in pip.utils.logging. But IIUC one won't hit this error unless they're using pip incorrectly.

[monty6400]

@pradyunsg I understood that, but the lines I need to modify are zipped, so I can't modify them, is there any way I'm not aware of to modify these lines?
(and to be sure I should replace _log_state.indentation to getattr(_log_state, indentation, 0)?)

Thanks anyway!

[pradyunsg]

@monty6400 As I said, it's likely that your use of pip is incorrect if you're running into this error.

is there any way I'm not aware of to modify these lines?

Nope. I don't think so.

(and to be sure I should replace _log_state.indentation to getattr(_log_state, indentation, 0)?)

Yes, that's the modification.

[monty6400]

@pradyunsg
this is the code I try to run:

can you see what is the problem?

[pradyunsg]

You're importing pip and calling pip.main. This is in-process invocation of pip and not supported. You should use the subprocess module (see subprocess.run) and invoke pip using that...

That said, I'm in the middle of something so, I'll let myself out of this discussion now. :)

[monty6400]

I saw no subprocess.run, only call (I'm using python 2.7 maybe that's the reason), anyway I saw multiprocessing and made the code run the pip installer in another window (the problem was not with the call of main, but with the call to the pip installer), and I see no problem for now.

thanks for the help

[mitar]

I made #4654 to address this.

[pfmoore]

Summary (for clarity) pip does not support use in-process via its internal API, and in particular the pip code makes no guarantees that it can be used safely in the presence of threads.

[dmvieira]

Do you know when this fix will be released?

[RonnyPfannschmidt]

@dmvieira never, as the pr got closed

[pradyunsg]

@dmvieira This won't be fixed since fixing this is essentially moving a bit towards supporting people's use of an unsupported API.

FWIW, we know that there'll be some sort of other bug that'll bite you even if this fix gets merged simply because [what @pfmoore stated above].

That said, if you're feeling fancy, you can patch pip's functions but then you're on you own. :)

[dmvieira]

Ok! Better answer 😊 thx

[pradyunsg]

@pypa/pip-committers Do any of you think now reverting #2977 is a good idea?

[pfmoore]

I think we probably shouldn't have committed it, and we should have been stricter over our "no supported API" position, but hindsight is easy, and I don't think it's worth reverting now. But it shouldn't be taken as a precedent that we'll accept further changes in support of using the internal API.

[dstufft]

I agree with @pfmoore

[pradyunsg]

Cool. :)