Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ct 1873/support insert overwrite #700

Merged
merged 14 commits into from
Apr 28, 2023
Merged

Conversation

VersusFacit
Copy link
Contributor

@VersusFacit VersusFacit commented Mar 29, 2023

resolves #600
Closes #430

Description

The insert overwrite incremental strategy can be used with Delta tables so we no longer need to raise an exception saying that this is not a possibility. And the test case that says that it's a bad strategy obviously needs to be changed to something that is a positive example of the functionality at work

This PR looks like it's ready to go. I've moved the community PR to this branch so that we don't run into a CI issue that was stumbling their process and made this fall onto the backlog.

Checklist

@VersusFacit VersusFacit requested a review from a team as a code owner March 29, 2023 10:13
@VersusFacit VersusFacit self-assigned this Mar 29, 2023
@cla-bot cla-bot bot added the cla:yes label Mar 29, 2023
@VersusFacit VersusFacit force-pushed the CT-1873/support_insert_overwrite branch 3 times, most recently from d161c6c to d47c92d Compare March 30, 2023 00:20
@VersusFacit VersusFacit force-pushed the CT-1873/support_insert_overwrite branch from d47c92d to 74942f8 Compare April 7, 2023 09:13
@VersusFacit
Copy link
Contributor Author

VersusFacit commented Apr 27, 2023

it appears that you cannot do these Delta overwrite strategies for whatever reason at the same time the database is configured with access control list.

spark.databricks.acl.dfAclsEnabled true

When I flip this to false, desta-overwrites work!! But, grants STOP working.

org.apache.spark.SparkException: Trying to perform permission action on Hive Metastore

Likewise, if I turn it back on, I run into the really weird cryptic errors that I’ve been dealing with for a while on the database cluster for delta dynamic overwrites.

I only found this out by finding an obscure unanswered question on the databricks forum plus a user of dbt five months ago struggling to solve this in the community Slack with no real resolution.

This may be a bug on databricks' side or that something to do with the implementation of a databricks environment prevents the simultaneous presence of these options. Either way it's not documented as to what is happening as far as I can tell. If there ends up being a solution I'd love to push that in and get this merged with future


Note, Setting the partition overwrite option to "dynamic" in the cluster set up does not resolve this issue

@nssalian
Copy link
Contributor

The solution looks alright from glancing through. I wonder if we should document those parameters for users somewhere so they are able to troubleshoot if needed.

@VersusFacit
Copy link
Contributor Author

I'll clean up the commit history before I merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[CT-1873] [PR Review] CT-1013 PR Review tracking for 430
4 participants