SNOW-1058583: Add support for iceberg tables to create_table and copy_into_table statements. #2162
Conversation
…iceberg_save' into jrose_snow_1058583_iceberg_save
|
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
sfc-gh-jrose
changed the title
|
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
1 similar comment
|
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
| [ | ||
| StructField("a", StringType()), | ||
| StructField("b", IntegerType()), | ||
| ] |
There was a problem hiding this comment.
Does this work for ArrayType and MapType?
There was a problem hiding this comment.
After some testing it looks like there's several edge cases that break for structured types. This change is large already so I will address those edge cases in a second PR.
| "external_volume": "PYTHON_CONNECTOR_ICEBERG_EXVOL", | ||
| "catalog": "SNOWFLAKE", | ||
| "base_location": "snowpark_python_tests", |
There was a problem hiding this comment.
can we also support them in a case insensitive way. i.e.
iceberg_config={
"external_volume": "PYTHON_CONNECTOR_ICEBERG_EXVOL",
"CATALOG": "SNOWFLAKE",
"BASE_LOCATION": "snowpark_python_tests"
}
should also work
| CHANGE_TRACKING: change_tracking, | ||
| } | ||
|
|
||
| if iceberg_config is not None: |
There was a problem hiding this comment.
Suggest also checking empty dict. if iceberg_config and "external_volume" in iceberg_config.
|
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
|
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
|
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
|
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
|
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-1058583
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
This PR adds support for iceberg table parameters to save_as_table and saveAsTable. I also looked into copy_into_table, but that appears to work already without the
ICEBERGkeyword. I've passed the parameters through in this function as well in case it needs to create a table.The Python test environment has weak iceberg support so these tests may need to be disabled if they prove to be flaky.