bpo-26175: Fix SpooledTemporaryFile IOBase abstract #3249

GFernie · 2017-08-30T21:57:47Z

One would assume that this class implements the IOBase abstract. As the
underlying file-like object is either io.BytesIO, io.StringIO, or a true
file object, this is a reasonable abstract expect and to implement.

Regardless, the behaviour of this class does not change much in the case
of the attribute being missing from the underlying file-like; an
AttributeError is still raised, albeit from one additional frame on the
stack trace.

https://bugs.python.org/issue26175

One would assume that this class implements the IOBase abstract. As the underlying file-like object is either io.BytesIO, io.StringIO, or a true file object, this is a reasonable abstract expect and to implement. Regardless, the behaviour of this class does not change much in the case of the attribute being missing from the underlying file-like; an AttributeError is still raised, albeit from one additional frame on the stack trace.

the-knights-who-say-ni · 2017-08-30T21:57:49Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Unfortunately we couldn't find an account corresponding to your GitHub username on bugs.python.org (b.p.o) to verify you have signed the CLA (this might be simply due to a missing "GitHub Name" entry in your b.p.o account settings). This is necessary for legal reasons before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

Thanks again to your contribution and we look forward to looking at it!

merwok

Patch looks good! Note that the bug contains a stronger motivation than the first mssage here: «This was discovered when seeking a SpooledTemporaryFile-backed lzma file.»

Lib/test/test_tempfile.py

bedevere-bot · 2017-09-20T18:07:59Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I didn't expect the Spanish Inquisition!. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

GFernie · 2017-09-30T13:23:11Z

Thanks for the feedback @merwok

I've made some changes to more accurately test how SpooledTemporaryFile implements IOBase. So, there are now two separate tests:

Assert that an instance of SpooledTemporaryFile is an instance of IOBase. Because IOBase already implements the interface (nearly) this makes it redundant to check for the individual attributes that SpooledTemporaryFile should implement.
Assert that the attributes which not declared in IOBase but should be implemented by subclasses are implemented by SpooledTemporaryFile.

From the IOBase docs:

Even though IOBase does not declare read(), readinto(), or write() because their signatures will vary, implementations and clients should consider those methods part of the interface.

I didn't expect the Spanish Inquisition!

bedevere-bot · 2017-09-30T13:23:13Z

Nobody expects the Spanish Inquisition!

@merwok: please review the changes made to this pull request.

embe · 2018-02-19T17:49:32Z

There is at least one more incompatibility with IOBase: seek should return the current position instead of None.

pppery · 2018-06-06T00:39:00Z

Lib/tempfile.py

@@ -685,6 +685,9 @@ def __exit__(self, exc, value, tb):
    def __iter__(self):
        return self._file.__iter__()

+    def __del__(self):


This shouldn't be added: deleting the SpooledTemporaryFile will null out the reference to the underline file, and therefor call its __del__

I think if you inherit the default IOBase.__del__ implementation, it will call close and defeat any ResourceWarning that might otherwise be emitted. Perhaps it is better to make __del__ do nothing, ~~or set it to object.__del__~~. [Seems that last option doesn’t exist.]

I agree: the underlying file should not be explicitly deleted as this is not expected behaviour. I can imagine a situation where someone is deliberately holding a reference to the underlying file and they would not expect/want it to be closed until their own reference has fallen out of scope. I've changed the method to do nothing and added a docstring to reflect this.

Thanks for your feedback

I agree: the underlying file should not be explicitly deleted as this is not expected behaviour

The doc says: """This function operates exactly as TemporaryFile() does, except [irrelevant differences]."""

And then, about TemporaryFile: """On completion of the context or destruction of the file object the temporary file will be removed from the filesystem."""

So it seems the underlying file should be deleted when the file object disappears.

The point is that IOBase implements a __del__ which has some side-effects. Those side effects are not desirable here. Any underlying buffers being wrapped, e.g. the TemporaryFile or the BytesIO, will be gc'd and handled as they should.

I think this implementation is correct.

Part of file interface: https://docs.python.org/library/io.html#io.IOBase.seek

We don't want to delete the underlying file explicitly as the expected behaviour is for the file to be deleted *after* it falls out of scope.

GFernie · 2018-09-09T21:58:02Z

You are right, @embe; it's part of the interface:
https://docs.python.org/library/io.html#io.IOBase.seek

The method now returns the value returned from self._file.seek(), delegating the responsibility of returning the current new absolute file position to the underlying file.

Thanks for pointing this out

Part of file interface: https://docs.python.org/library/io.html#io.IOBase.truncate

GFernie · 2018-09-09T22:33:47Z

Re seek: I've also done the same with truncate, as per Martin's recommendations on the Python bug tracker: https://bugs.python.org/issue26175#msg319145

This is also part of the file interface:
https://docs.python.org/library/io.html#io.IOBase.truncate

GFernie · 2019-11-23T16:59:18Z

@pitrou I've considered your comments on what we should expect when calling __del__. There's three options which have been discussed:

Do nothing. The underlying buffer or temp file will be deleted when the object is finalised anyway. If the object is deleted with del, this would happen immediately. However, if __del__ is called then this would delay the deletion of the underlying buffer or temp file until the object is deleted with del or is garbage collected, so possibly never. Here, we are in a situation where we called __del__, expected the resources to be freed, but not even a warning was raised.
Inherit the io.IOBase.__del__ implementation. The underlying buffer or temp file will be deleted when __del__ is called, the object is deleted with del, or the object is garbage collected. This is the expected behaviour, but base implementation will explicitly call close, suppressing the ResourceWarning which should be raised when the object is deleted before closing the file.
Delegate the __del__ call to the underlying file. If it is a buffer then then the buffer will be freed and if it's rolled over to disk then the temp file will be deleted. If the temp file was not closed before the object was deleted then a ResourceWarning will be raised, just like TemporaryFile. This ensures that resources are freed exactly when we expect them to be and also that we are raising a warning when resources are not being managed properly by the programmer.

I've picked the third option, as you suggested. Thanks for your review.

I have made the requested changes; please review again.

bedevere-bot · 2019-11-23T16:59:22Z

Thanks for making the requested changes!

@merwok, @pitrou: please review the changes made to this pull request.

pitrou

Thanks for posting this. Generally speaking, calling __del__ directly is incorrect. Only the interpreter should do that. From a user's point of view, what's important is whether an object is still reachable or not.

pitrou · 2019-11-23T17:57:02Z

Doc/library/tempfile.rst

@@ -116,6 +116,7 @@ The module defines the following user-callable items:

   .. versionchanged:: 3.8
      Added *errors* parameter.
+      Implemented :class:`io.IOBase` inteface.


Since 3.8.0 has been released, this should be moved to a new versionchanged:: 3.9 block.

pitrou · 2019-11-23T17:59:22Z

Lib/tempfile.py

@@ -679,6 +679,9 @@ def __exit__(self, exc, value, tb):
    def __iter__(self):
        return self._file.__iter__()

+    def __del__(self):
+        return self._file.__del__()


Hmm, calling the __del__ method directly like that is not correct. __del__ is only to be called when all external references are gone. Instead I would suggest something like the following:

def __del__(self): if not self.closed: import warnings warnings.warn('unclosed file %r' % (self,), ResourceWarning, stacklevel=2, source=self) self.close()

(this is adapted from FileIO.__del__ in Lib/pyio.py)

pitrou · 2019-11-23T18:09:25Z

Lib/test/test_tempfile.py

+        # ResourceWarning since the file was not explicitly closed.
+        f = self.do_create(max_size=2)
+        f.write(b'foo')
+        filename = f.name


Does this actually work? Here I get something like:

>>> f = tempfile.SpooledTemporaryFile() >>> f.name >>> f.rollover() >>> f.name 3

pitrou · 2019-11-23T18:10:21Z

Lib/test/test_tempfile.py

+        f.write(b'foo')
+        filename = f.name
+        self.assertTrue(os.path.exists(filename))
+        with self.assertWarns(ResourceWarning):


Do we also expect a ResourceWarning to be raised when the file is not rolled over? If yes, there should be a test for it. If not, then the __del__ implementation can be removed.

pitrou · 2019-11-23T18:10:45Z

Lib/test/test_tempfile.py

+        filename = f.name
+        self.assertTrue(os.path.exists(filename))
+        with self.assertWarns(ResourceWarning):
+            f.__del__()


It is not correct to call __del__ directly. Instead, use del f or f = None.

bedevere-bot · 2019-11-23T18:12:37Z

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

csabella · 2020-01-12T13:17:31Z

@GFernie, thank you for your last comments and for addressing the code review. It looks like @pitrou had some additional notes after that, so it would be great if you could look at this again. Thank you!

thehesiod · 2020-02-12T00:37:02Z

Lib/tempfile.py

@@ -743,18 +755,21 @@ def tell(self):

    def truncate(self, size=None):
        if size is None:
-            self._file.truncate()
+            return self._file.truncate()
        else:


this else is not needed, PEP warning, might as well fix

It's not necessary to remove the else. It does no harm.

I never said it was necessary, just bad style as it unnecessarily increased the code depth, take my comment as an FYI

Just wanted to confirm @merwok - nothing needs to be changed here for approval?

Right, I advise to not change the else line.

danieldjewell · 2020-05-13T20:29:14Z

Not sure if @GFernie is still around -- are the outstanding changes/questions still those raised by @pitrou? @merwok?

merwok · 2020-05-13T21:18:09Z

This PR needs to merge master and address Antoine’s latest review.

csabella · 2020-05-23T15:36:59Z

@danieldjewell, you're right. Would you like to create a new PR based on this one? If so, please follow this section of the devguide to give GFernie the credit as the original author. Thanks!

carlos-jenkins · 2020-10-31T04:09:04Z

Hello,

For anyone that got here looking for the error 'SpooledTemporaryFile' object has no attribute 'seekable'

In my case it was trying to pass an SpooledTemporaryFile to an ZipFile to decompress it (Starlette's UploadFile class if anyone cares). But there are other cases. Until the issue referenced in this PR is fixed, an option is to monkey patch the class. Sorry, ugly hack follows.

In one of your modules:

from tempfile import SpooledTemporaryFile

def _seekable(self):
    """
    Monkey patched seekable() method for the SpooledTemporaryFile class.

    Sadly, we cannot send an instance of the SpooledTemporaryFile to the
    ZipFile object.

    This is because the SpooledTemporaryFile doesn't inherit / implement the
    IOBase class.

    This bug is reported in CPython:

        https://bugs.python.org/issue26175

    And an unmerged PR is located here:

        https://github.com/python/cpython/pull/3249/files

    In Python 3.8, attempting to pass the tmpfd fails with:

        AttributeError:
            'SpooledTemporaryFile' object has no attribute 'seekable'

    So, the workaround to avoid copying the file is to monkey patch the class 
    and add the missing method. Sorry.
    """
    return self._file.seekable()


SpooledTemporaryFile.seekable = _seekable

Molaire · 2021-01-18T02:38:56Z

To Google people: this also breaks pandas.read_csv() and this monkey patch can be used for property readable writable and seekable to fix it.

Hello,

For anyone that got here looking for the error 'SpooledTemporaryFile' object has no attribute 'seekable'

In my case it was trying to pass an SpooledTemporaryFile to an ZipFile to decompress it (Starlette's UploadFile class if anyone cares). But there are other cases. Until the issue referenced in this PR is fixed, an option is to monkey patch the class. Sorry, ugly hack follows.

In one of your modules:

from tempfile import SpooledTemporaryFile

def _seekable(self):
    """
    Monkey patched seekable() method for the SpooledTemporaryFile class.

    Sadly, we cannot send an instance of the SpooledTemporaryFile to the
    ZipFile object.

    This is because the SpooledTemporaryFile doesn't inherit / implement the
    IOBase class.

    This bug is reported in CPython:

        https://bugs.python.org/issue26175

    And an unmerged PR is located here:

        https://github.com/python/cpython/pull/3249/files

    In Python 3.8, attempting to pass the tmpfd fails with:

        AttributeError:
            'SpooledTemporaryFile' object has no attribute 'seekable'

    So, the workaround to avoid copying the file is to monkey patch the class 
    and add the missing method. Sorry.
    """
    return self._file.seekable()


SpooledTemporaryFile.seekable = _seekable

MikeEcho · 2021-12-26T12:18:21Z

Python dev here from the future (2021 - save yourselves while you can). Still getting SpooledTemporaryFile errors while reading CSVs (via tempfiles) in. Kindly asking if requested changes will be addressed @GFernie ?

pR0Ps · 2021-12-31T05:52:44Z

@MikeEcho since it seemed like @GFernie had moved on I submitted an updated and rebased PR for this fix in #29560 with all of the code review comments addressed. Just waiting on a code review.

github-actions · 2022-02-21T00:08:36Z

This PR is stale because it has been open for 30 days with no activity.

methane · 2022-05-03T12:20:17Z

Close this because I merged #29560.

the-knights-who-say-ni added the CLA not signed label Aug 30, 2017

bedevere-bot added the awaiting review label Aug 30, 2017

Add news entry for bpo-26175 patch

3b69baa

the-knights-who-say-ni added CLA signed and removed CLA not signed labels Aug 30, 2017

merwok requested changes Sep 20, 2017

View reviewed changes

Lib/test/test_tempfile.py Outdated Show resolved Hide resolved

bedevere-bot added awaiting changes and removed awaiting review labels Sep 20, 2017

merwok and others added 3 commits September 20, 2017 14:10

Update 2017-08-30-23-14-17.bpo-26175.GOaj9y.rst

753b4fb

Make SpooledTemporaryFile subclass IOBase

a70113e

Refactor tests to more reliably test abstract

fb28362

bedevere-bot added awaiting change review and removed awaiting changes labels Sep 30, 2017

merwok approved these changes Sep 30, 2017

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting change review labels Sep 30, 2017

GFernie mentioned this pull request Oct 9, 2017

Use io.IOBase as base class for StreamingBody boto/botocore#879

Closed

pppery reviewed Jun 6, 2018

View reviewed changes

terryjreedy requested a review from vadmium June 8, 2018 17:51

GFernie added 2 commits September 9, 2018 23:49

Return new absolute position from underlying file on seek

bc9d0a9

Part of file interface: https://docs.python.org/library/io.html#io.IOBase.seek

Do nothing on __del__

b588dec

We don't want to delete the underlying file explicitly as the expected behaviour is for the file to be deleted *after* it falls out of scope.

Return new file size from underlying file on truncate

ab49dd1

Part of file interface: https://docs.python.org/library/io.html#io.IOBase.truncate

GFernie requested a review from pitrou November 23, 2019 16:22

bedevere-bot added awaiting change review and removed awaiting changes labels Nov 23, 2019

bedevere-bot requested a review from merwok November 23, 2019 16:59

pitrou requested changes Nov 23, 2019

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting change review labels Nov 23, 2019

thehesiod reviewed Feb 12, 2020

View reviewed changes

vorpal-buildbot mentioned this pull request Jun 13, 2020

Initial Update PennyDreadfulMTG/Penny-Dreadful-Tools#7430

Merged

btel mentioned this pull request Dec 11, 2020

add more channel_actions mamba-org/quetz#230

Merged

jonathan-s mentioned this pull request Dec 18, 2020

AttributeError: 'SpooledTemporaryFile' object has no attribute 'readinto' jazzband/django-dbbackup#378

Open

tony-sappe mentioned this pull request Feb 12, 2021

[FIX] Augment tempfile.SpooledTemporaryFile() for expected behavior fedspendingtransparency/usaspending-api#2963

Merged

10 tasks

fberanizo mentioned this pull request Aug 4, 2021

Correcting file upload. platiagro/datasets#64

Merged

pR0Ps mentioned this pull request Nov 15, 2021

bpo-26175: Implement io.IOBase interface for SpooledTemporaryFile #29560

Merged

github-actions bot added the stale Stale PR or inactive for long period of time. label Feb 21, 2022

methane closed this May 3, 2022

GFernie mannequin mentioned this pull request May 3, 2022

Fully implement IOBase abstract on SpooledTemporaryFile #70363

Closed

bpo-26175: Fix SpooledTemporaryFile IOBase abstract #3249

bpo-26175: Fix SpooledTemporaryFile IOBase abstract #3249

Conversation

GFernie commented Aug 30, 2017 • edited by bedevere-bot Loading

the-knights-who-say-ni commented Aug 30, 2017

merwok left a comment

Choose a reason for hiding this comment

bedevere-bot commented Sep 20, 2017

GFernie commented Sep 30, 2017

bedevere-bot commented Sep 30, 2017

embe commented Feb 19, 2018

pppery Jun 6, 2018 • edited Loading

Choose a reason for hiding this comment

vadmium Jun 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GFernie commented Sep 9, 2018 • edited Loading

GFernie commented Sep 9, 2018

GFernie commented Nov 23, 2019

bedevere-bot commented Nov 23, 2019

pitrou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bedevere-bot commented Nov 23, 2019

csabella commented Jan 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danieldjewell commented May 13, 2020

merwok commented May 13, 2020

csabella commented May 23, 2020

carlos-jenkins commented Oct 31, 2020

Molaire commented Jan 18, 2021

MikeEcho commented Dec 26, 2021

pR0Ps commented Dec 31, 2021

github-actions bot commented Feb 21, 2022

methane commented May 3, 2022

GFernie commented Aug 30, 2017 •

edited by bedevere-bot

Loading

pppery Jun 6, 2018 •

edited

Loading

vadmium Jun 9, 2018 •

edited

Loading

GFernie commented Sep 9, 2018 •

edited

Loading