Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: pyarrow duration arrays constructed from data containing NaT can overflow #52843

Merged
merged 4 commits into from
Apr 23, 2023

Conversation

lukemanley
Copy link
Member

The underlying bug is upstream: apache/arrow#35088

Fixes the following:

import pandas as pd
import pyarrow as pa

data_ts = pd.to_datetime([1, None])
data_td = pd.to_timedelta([1, None])

ser_ts = pd.Series(data_ts, dtype=pd.ArrowDtype(pa.timestamp("ns")))
ser_td = pd.Series(data_td, dtype=pd.ArrowDtype(pa.duration("ns")))

result = ser_ts + ser_td  # ArrowInvalid: overflow

@lukemanley lukemanley added Bug Upstream issue Issue related to pandas dependency Arrow pyarrow functionality labels Apr 22, 2023
@lukemanley lukemanley added this to the 2.0.1 milestone Apr 22, 2023
@phofl phofl merged commit 4539f3e into pandas-dev:main Apr 23, 2023
@phofl
Copy link
Member

phofl commented Apr 23, 2023

thx @lukemanley

@lumberbot-app
Copy link

lumberbot-app bot commented Apr 23, 2023

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

  1. Checkout backport branch and update it.
git checkout 2.0.x
git pull
  1. Cherry pick the first parent branch of the this PR on top of the older branch:
git cherry-pick -x -m1 4539f3eff40f0889b1e089c68cfe2a0b034d7ec9
  1. You will likely have some merge/cherry-pick conflict here, fix them and commit:
git commit -am 'Backport PR #52843: BUG: pyarrow duration arrays constructed from data containing NaT can overflow'
  1. Push to a named branch:
git push YOURFORK 2.0.x:auto-backport-of-pr-52843-on-2.0.x
  1. Create a PR against branch 2.0.x, I would have named this PR:

"Backport PR #52843 on branch 2.0.x (BUG: pyarrow duration arrays constructed from data containing NaT can overflow)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.

phofl pushed a commit to phofl/pandas that referenced this pull request Apr 23, 2023
phofl added a commit that referenced this pull request Apr 23, 2023
…tructed from data containing NaT can overflow) (#52869)

BUG: pyarrow duration arrays constructed from data containing NaT can overflow (#52843)

(cherry picked from commit 4539f3e)

Co-authored-by: Luke Manley <lukemanley@gmail.com>
@lukemanley lukemanley deleted the arrow-duration-nat branch April 23, 2023 16:26
Rylie-W pushed a commit to Rylie-W/pandas that referenced this pull request May 19, 2023
Daquisu pushed a commit to Daquisu/pandas that referenced this pull request Jul 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Arrow pyarrow functionality Bug Upstream issue Issue related to pandas dependency
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants