-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Spark] Block unsupported operations on identity columns #3457
[Spark] Block unsupported operations on identity columns #3457
Conversation
spark/src/main/scala/org/apache/spark/sql/delta/commands/WriteIntoDelta.scala
Show resolved
Hide resolved
if (Cast.canUpCast(LongType, targetType)) { | ||
val ex = intercept[DeltaAnalysisException] { | ||
sql(s"ALTER TABLE $tblName $keyword COLUMN id TYPE ${targetType.sql}") | ||
} | ||
assert(ex.getErrorClass === "DELTA_IDENTITY_COLUMNS_ALTER_COLUMN_NOT_SUPPORTED") | ||
} else { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we do a case match on targetType
instead of this?
targetType match {
case IntegerType => // foo
case DoubleType => // bar
case _ => fail("unexpected targetType")
}
spark/src/test/scala/org/apache/spark/sql/delta/IdentityColumnAdmissionSuite.scala
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
60feb9d
to
d519881
Compare
spark/src/main/scala/org/apache/spark/sql/delta/catalog/DeltaCatalog.scala
Outdated
Show resolved
Hide resolved
spark/src/main/scala/org/apache/spark/sql/delta/catalog/DeltaCatalog.scala
Outdated
Show resolved
Hide resolved
spark/src/main/scala/org/apache/spark/sql/delta/catalog/DeltaCatalog.scala
Outdated
Show resolved
Hide resolved
spark/src/main/scala/org/apache/spark/sql/delta/commands/alterDeltaTableCommands.scala
Outdated
Show resolved
Hide resolved
d519881
to
26c363d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! :)
spark/src/main/scala/org/apache/spark/sql/delta/catalog/DeltaCatalog.scala
Show resolved
Hide resolved
26c363d
to
3431990
Compare
Which Delta project/connector is this regarding?
Description
This PR is part of #1959
In this PR, we block unsupported operations on identity columns including:
How was this patch tested?
A new test suite
IdentityColumnAdmissionScalaSuite
is added.Does this PR introduce any user-facing changes?
Yes. The aforementioned operations on identity columns are blocked.