-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SNOW-1570734: Reduce describe query when there is no schema change #2126
base: main
Are you sure you want to change the base?
Conversation
108c337
to
9da3614
Compare
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
@@ -4,6 +4,10 @@ | |||
|
|||
### Snowpark Python API Updates | |||
|
|||
#### Improvements | |||
|
|||
- Reduced the number of additional [describe queries](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example#retrieving-column-metadata) sent to the server to fetch the metadata of a DataFrame. It is still an experimental feature not enabled by default, and can be enabled by setting `session.reduce_describe_query_enabled` to `True`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Reduced the number of additional [describe queries](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example#retrieving-column-metadata) sent to the server to fetch the metadata of a DataFrame. It is still an experimental feature not enabled by default, and can be enabled by setting `session.reduce_describe_query_enabled` to `True`. | |
- Reduced the number of additional [describe queries](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example#retrieving-column-metadata) sent to the server to fetch the metadata of a DataFrame. It is still an experimental improvement not enabled by default, and can be enabled by setting `session.reduce_describe_query_enabled` to `True`. |
lambda df: df.filter(col("a") > 2).sample(0.5), | ||
], | ||
) | ||
def test_schema_no_change(session, action): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a way to count the number of describe queries so you can assert they are 0 (or whatever the reduced number is) as part of the test?
@@ -4,6 +4,10 @@ | |||
|
|||
### Snowpark Python API Updates | |||
|
|||
#### Improvements | |||
|
|||
- Reduced the number of additional [describe queries](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example#retrieving-column-metadata) sent to the server to fetch the metadata of a DataFrame. It is still an experimental feature not enabled by default, and can be enabled by setting `session.reduce_describe_query_enabled` to `True`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we have any experimental results on the impact of this improvement?
6f99186
to
465c937
Compare
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-1570734
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
For
filter
,sort
,limit
andsample
APIs, we don't need to fetch schema/metadata from server because they are not changed.