BUG: dt64 <-> dt64tz roundup

[jbrockmendel]

xref #33401 which discusses astype, #24559 which discusses constructors. I think these need to be considered jointly.

Two big-picture questions.

1. when we pass constructor(dt64values, tz=tz), do we interpret dt64values as UTC or wall-times?
2. when we pass constructor(dt64values).astype("M8[ns, tz]")`, do we interpret dt64values as UTC or wall-times?

Let's check the current behavior on master for Timestamp, DatetimeIndex, and Series (see code block below if you want to check my work)

cls	Timestamp	DatetimeIndex	Series
cls(dt64, dtype=dt64tz)	UTC	Wall	UTC
cls(dt64).astype(dt64tz)	N/A	Wall	UTC
cls(dt64tz).astype("M8[ns]")	N/A	UTC	UTC

If we were starting fresh, I would probably lean towards making these all interpret dt64 as wall-times. But because these are already mostly-UTC, it would be an easier change to bring the DatetimeIndex behavior in line with the others.

Thoughts? cc @jorisvandenbossche @mroeschke @jreback

values = (3600 * 10**9 * np.arange(4)).astype("datetime64[ns]")
dt64 = values[-1]

tz = "US/Pacific"
dtype = pd.DatetimeTZDtype(tz=tz)

ts = pd.Timestamp(dt64, tz=tz)
dti = pd.DatetimeIndex(values, dtype=dtype)
ser = pd.Series(values, dtype=dtype)

if_wall = pd.DatetimeIndex(values).tz_localize(tz)
if_utc = pd.DatetimeIndex(values).tz_localize("UTC").tz_convert(tz)

assert ts == if_utc[-1]
assert (dti == if_wall).all()
assert (ser == if_utc).all()

dti_naive = pd.DatetimeIndex(values)
ser_naive = pd.Series(values)

res_dti = dti_naive.astype(dtype)
res_ser = ser.astype(dtype)

assert (res_dti == dti).all()
assert (res_ser == ser).all()

# astype from tzaware -> tznaive
assert (dti.astype(dti_naive.dtype) == dti.tz_convert("UTC").tz_localize(None)).all()
assert (ser_naive.astype(ser.dtype) == ser).all()

[mroeschke]

If we were starting fresh, I would probably lean towards making these all interpret dt64 as wall-times.

Agreed

Is there any possibility of just deprecating any naive -> tz aware (and vice versa) converstion with astype #33401 (comment)?

[jbrockmendel]

Is there any possibility of just deprecating any naive -> tz aware (and vice versa) converstion with astype #33401 (comment)?

I agree that would be nice. The question is if we are OK with breaking frame.astype({col: tz_dtype}) without having a "do X instead" available.

[mroeschke]

I'd be okay with that with just specifying "any tz_dtype conversion should use tz_localize/convert", but should probably hear from others

[jbrockmendel]

I'd be okay with that with just specifying "any tz_dtype conversion should use tz_localize/convert", but should probably hear from others

agreed. if @jorisvandenbossche and @jreback are on board ill make a deprecation PR

[jbrockmendel]

gentle ping @jorisvandenbossche @jreback

[jbrockmendel]

@jreback gentle ping

[jreback]

I'd be okay with that with just specifying "any tz_dtype conversion should use tz_localize/convert", but should probably hear from others

agreed. if @jorisvandenbossche and @jreback are on board ill make a deprecation PR

I am +1 on this, even though generally pandas makes life easy by just doing the right thing. when there is a choice where its not completely obvious as is the case for localize/convert we should raise.

I think this ok for a 1.3 release.