[SNOW-1625268] Refresh with recent snowpark python changes 1.22-1.23 #2416
+50,195
−27,844
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eekday/time and DatetimeIndex.time (#2128)
…2113) Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com> Co-authored-by: Naren Krishna <naren.krishna@snowflake.com>
…rations. (#2130) Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com>
<!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1418500 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. This PR refactors `Session._resolve_packages` to not have any side-effects. In the old implementation, we relied on this function to make updates to `Session._packages` variable. Now we return the final resulting state after resolving packages and update `Session._packages` only in `Session.add_packages` making `Session._resolve_packages` have no side-effect.
<!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1458127 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Implemented `Index.min` and `Index.max` using the existing Series implementation. ```py >>> idx = pd.Index([3, 2, 1]) >>> idx.max() 3 >>> idx = pd.Index(['c', 'b', 'a']) >>> idx.max() 'c' >>> idx = pd.Index([3, 2, 1]) >>> idx.min() 1 >>> idx = pd.Index(['c', 'b', 'a']) >>> idx.min() 'a' ```
…#2110) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1010216, #2019 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Adding support to specify the following: - REFERSH_MODE - INITIALIZE - CLUSTER BY - TRANSIENT - DATA_RETENTION_TIME_IN_DAYS - MAX_DATA_EXTENSION_TIME_IN_DAYS --------- Co-authored-by: Jamison Rose <Jamison.Rose@snowflake.com>
<!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1635810 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue.
<!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1625536 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue.
#2141) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> SNOW-1625468 Support indexing with Timedelta data columns 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. Support indexing with Timedelta data columns: - Added sufficient tests cases for indexing including get and set values with the same timedelta type or different types. - To achieve this, I updated several util methods related to transpose, join too. - Also, I need to refactor `SnowparkPandasType` since there are some confusing on `type` object and class there. This pull request includes several changes to enhance support for `Timedelta` and improve type safety in the Snowflake Snowpark Modin plugin. The most important changes are summarized below: ### Enhancements to Timedelta Support: * Added support for indexing with `Timedelta` data columns. (`CHANGELOG.md`: [CHANGELOG.mdR30-L37](diffhunk://#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4edR30-L37)) ### Type Safety Improvements: * Added assertions to check that `data_column_types` and `index_column_types` are instances of `SnowparkPandasType` in `_create_snowflake_quoted_identifier_to_snowpark_pandas_type`. (`src/snowflake/snowpark/modin/plugin/_internal/frame.py`: [src/snowflake/snowpark/modin/plugin/_internal/frame.pyR92-R106](diffhunk://#diff-dc59d6fb5be73824e72c1e84ca671739e68c28f651a164e96af7f19a3f732edeR92-R106)) * Updated `project_columns` to include an optional `column_types` parameter and ensure its length matches `pandas_labels`. (`src/snowflake/snowpark/modin/plugin/_internal/frame.py`: [[1]](diffhunk://#diff-dc59d6fb5be73824e72c1e84ca671739e68c28f651a164e96af7f19a3f732edeR999) [[2]](diffhunk://#diff-dc59d6fb5be73824e72c1e84ca671739e68c28f651a164e96af7f19a3f732edeR1013-R1021) [[3]](diffhunk://#diff-dc59d6fb5be73824e72c1e84ca671739e68c28f651a164e96af7f19a3f732edeL1023-R1037) * Modified `set_frame_2d_labels` and `set_frame_2d_positional` to handle `SnowparkPandasColumn` and `SnowparkPandasType` correctly. (`src/snowflake/snowpark/modin/plugin/_internal/indexing_utils.py`: [[1]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eL2439-R2499) [[2]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eL2502-R2555) [[3]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eL2677-R2709) [[4]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eR2775) [[5]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eR2818-R2837) [[6]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eL2800-R2848) * Updated `get_item_series_as_single_row_frame` to include `SnowparkPandasType`. (`src/snowflake/snowpark/modin/plugin/_internal/indexing_utils.py`: [src/snowflake/snowpark/modin/plugin/_internal/indexing_utils.pyR3065](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eR3065)) * Enhanced `_create_internal_frame_with_join_or_align_result` to handle `data_column_types` and `index_column_types` during joins. (`src/snowflake/snowpark/modin/plugin/_internal/join_utils.py`: [[1]](diffhunk://#diff-67e1df8ec1e45b14cf51e35c6f67ac04982e41f85580cdfff391e35e025546d0R239-R246) [[2]](diffhunk://#diff-67e1df8ec1e45b14cf51e35c6f67ac04982e41f85580cdfff391e35e025546d0R265-R291) [[3]](diffhunk://#diff-67e1df8ec1e45b14cf51e35c6f67ac04982e41f85580cdfff391e35e025546d0R316-R327)
Fixes SNOW-1618349 This PR adds initial support for `pd.merge_asof` -- parameters `by`, `left_by`, `right_by`, `left_index`, `right_index`, `suffixes`, and `tolerance` are not yet supported. Additionally `direction=nearest` is not yet supported but this can be done in a follow-up PR. --------- Signed-off-by: Naren Krishna <naren.krishna@snowflake.com>
…atetimeIndex.normalize (#2143) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1632900, SNOW-1625232 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Add support for Series.dt.normalize and DatetimeIndex.normalize.
…, `is_floating`, and `is_object` (#2146) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1458123 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Implemented Index `is_numeric`, `is_integer`, `is_boolean`, `is_floating`, and `is_object`.
SNOW-1620412 This PR adds support for `astype` by allowing conversion to Timedelta. --------- Signed-off-by: Naren Krishna <naren.krishna@snowflake.com>
<!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1620446 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Adding notebooks with timedelta usecases and negative integration tests --------- Signed-off-by: Labanya Mukhopadhyay <labanya.mukhopadhyay@snowflake.com>
<!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1625378 Test write timedelta in I/O 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. Warn user the timedelta type may be lost during writing back to Snowflake.
<!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-NNNNNNN 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. Signed-off-by: Naren Krishna <naren.krishna@snowflake.com>
…hods (#2135) Fixes SNOW-1558919 Added support for DatetimeIndex ceil, floor and round methods. Raise not implemented error if ambiguous or nonexistent parameter is set.
Fixes SNOW-1636767 Fixes SNOW-1635405 --------- Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com> Co-authored-by: Andong Zhan <andong.zhan@snowflake.com>
SNOW-1638433 Add AssertionError message into Telemetry Also make sure all assert errors from plugin code have error message
<!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1637932 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. The main purpose for this refactoring is to avoid calling schema when cached type is available, e.g., avoid calling schema when we know the type is Timedelta in binary ops. This pull request includes several changes to the Snowflake Snowpark Modin plugin, focusing on refactoring the way Snowflake types are retrieved and used. The main changes involve replacing the `quoted_identifier_to_snowflake_type` method with a new `get_snowflake_type` method, which simplifies type retrieval by allowing it to accept a single identifier or a list of identifiers. ### Refactoring Type Retrieval * [`src/snowflake/snowpark/modin/plugin/_internal/aggregation_utils.py`](diffhunk://#diff-036a8cce05771914c03d260ad7fb1ab74a1578b353ff5156a65fbe546788872cL1044-L1047): Updated `generate_column_agg_info` to use `get_snowflake_type` instead of `quoted_identifier_to_snowflake_type` for retrieving Snowflake types. [[1]](diffhunk://#diff-036a8cce05771914c03d260ad7fb1ab74a1578b353ff5156a65fbe546788872cL1044-L1047) [[2]](diffhunk://#diff-036a8cce05771914c03d260ad7fb1ab74a1578b353ff5156a65fbe546788872cR1067-R1070) [[3]](diffhunk://#diff-036a8cce05771914c03d260ad7fb1ab74a1578b353ff5156a65fbe546788872cL1109-R1109) * [`src/snowflake/snowpark/modin/plugin/_internal/binary_op_utils.py`](diffhunk://#diff-dd6dcb779b1e636fa0bc9541f9c0f8f0e18227367ef40a779146c5a6108676ebL631-R631): Modified `prepare_binop_pairs_between_dataframe_and_dataframe` to use `get_snowflake_type` for type mapping. [[1]](diffhunk://#diff-dd6dcb779b1e636fa0bc9541f9c0f8f0e18227367ef40a779146c5a6108676ebL631-R631) [[2]](diffhunk://#diff-dd6dcb779b1e636fa0bc9541f9c0f8f0e18227367ef40a779146c5a6108676ebL649-R649) [[3]](diffhunk://#diff-dd6dcb779b1e636fa0bc9541f9c0f8f0e18227367ef40a779146c5a6108676ebL671-R671) * [`src/snowflake/snowpark/modin/plugin/_internal/frame.py`](diffhunk://#diff-dc59d6fb5be73824e72c1e84ca671739e68c28f651a164e96af7f19a3f732edeL360-R417): Added `get_snowflake_type` method and updated existing methods to use it. [[1]](diffhunk://#diff-dc59d6fb5be73824e72c1e84ca671739e68c28f651a164e96af7f19a3f732edeL360-R417) [[2]](diffhunk://#diff-dc59d6fb5be73824e72c1e84ca671739e68c28f651a164e96af7f19a3f732edeL518-R564) ### Updating Indexing Utilities * [`src/snowflake/snowpark/modin/plugin/_internal/indexing_utils.py`](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eL386-R388): Replaced `quoted_identifier_to_snowflake_type` with `get_snowflake_type` in multiple functions for checking data types. [[1]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eL386-R388) [[2]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eL1241-R1243) [[3]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eL1655-R1655) [[4]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eL1742-R1743) [[5]](diffhunk://#diff-524607b71f519819352dae1467474661e35164db39180ad106e30e7bf2e3265eL2682-R2682) ### Enhancing Join and Where Utilities * [`src/snowflake/snowpark/modin/plugin/_internal/join_utils.py`](diffhunk://#diff-67e1df8ec1e45b14cf51e35c6f67ac04982e41f85580cdfff391e35e025546d0L1070-R1071): Updated `convert_incompatible_types_to_variant` to use `get_snowflake_type` for type mapping. * [`src/snowflake/snowpark/modin/plugin/_internal/where_utils.py`](diffhunk://#diff-ddf55ef822ec0c6f0e5406f498175dce3ae3a2177c036884d7356961d5b15015L19-R26): Refactored `validate_expected_boolean_data_columns` to use `get_snowflake_type` for type validation. ### Compiler Adjustments * [`src/snowflake/snowpark/modin/plugin/compiler/snowflake_query_compiler.py`](diffhunk://#diff-834ee069919510e7e410c503a8afa455154c40e65389769c08d35b0ec3f8ec03L473-R477): Refactored several methods to use `get_snowflake_type` for type retrieval and mapping. [[1]](diffhunk://#diff-834ee069919510e7e410c503a8afa455154c40e65389769c08d35b0ec3f8ec03L473-R477) [[2]](diffhunk://#diff-834ee069919510e7e410c503a8afa455154c40e65389769c08d35b0ec3f8ec03L497-R502) [[3]](diffhunk://#diff-834ee069919510e7e410c503a8afa455154c40e65389769c08d35b0ec3f8ec03L1500) [[4]](diffhunk://#diff-834ee069919510e7e410c503a8afa455154c40e65389769c08d35b0ec3f8ec03L1545-R1546) [[5]](diffhunk://#diff-834ee069919510e7e410c503a8afa455154c40e65389769c08d35b0ec3f8ec03L1710-R1711) [[6]](diffhunk://#diff-834ee069919510e7e410c503a8afa455154c40e65389769c08d35b0ec3f8ec03L1870-R1877) [[7]](diffhunk://#diff-834ee069919510e7e410c503a8afa455154c40e65389769c08d35b0ec3f8ec03L1924)
…aysinmonth (#2151) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1636789, SNOW-1636790 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Add support for Series.dt.days_in_month/daysinmonth.
…ithout initializing Snowpark pandas (#2097) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1625830 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [x] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Calling `to_snowpark_pandas` on a Snowpark Python DataFrame without first performing `import snowflake.snowpark.modin.plugin` currently raises an error (see attached JIRA for exact reproduction). This was not the case in previous releases, and this PR fixes internal imports such that performing this operation implicitly initializes Snowpark pandas. After this PR, when `to_snowpark_pandas` is called without explicitly initializing Snowpark pandas, one of two things happens: 1. If modin is not installed, Snowpark pandas will surface the following error: ``` ModuleNotFoundError: Modin is not installed. Run `pip install "snowflake-snowpark-python[modin]"` to resolve. ``` This is the same error as if the user had tried to `import snowflake.snowpark.modin.plugin` without installing the modin dependency. 2. If modin is installed, Snowpark pandas will implicitly initialize Snowpark pandas by internally running `import snowflake.snowpark.modin.plugin`. If the user performs `import modin.pandas as pd` afterwards, the modin namespace will be set up with Snowpark pandas behavior.
…pected (#2138) <!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-1458137 2. Fill out the following pre-review checklist: - [x] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Verifying that `df.index = new_index` is implemented correctly.
…2428) Fixes SNOW-1730923 Signed-off-by: sfc-gh-mvashishtha <mahesh.vashishtha@snowflake.com>
|
There have been a few API changes. This PR does not add new IR modeling etc. but simply refreshes with Snowpark changes. A ticket to bring APIs up to speed is here https://snowflakecomputing.atlassian.net/browse/SNOW-1731697. This should help ease the review burden by splitting the work up into two PRs:
|
sfc-gh-lspiegelberg
changed the title
sfc-gh-lspiegelberg
requested review from
sfc-gh-nkrishna,
sfc-gh-jkew,
sfc-gh-jdu,
sfc-gh-yixie and
sfc-gh-jrose
and removed request for
a team,
sfc-gh-nkrishna,
sfc-gh-jkew,
sfc-gh-jdu,
sfc-gh-yixie and
sfc-gh-jrose
<!--- Please answer these questions before creating your pull request. Thanks! ---> 1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR. <!--- In this section, please add a Snowflake Jira issue number. Note that if a corresponding GitHub issue exists, you should still include the Snowflake Jira issue number. For example, for GitHub issue #1400, you should add "SNOW-1335071" here. ---> Fixes SNOW-NNNNNNN 2. Fill out the following pre-review checklist: - [ ] I am adding a new automated test(s) to verify correctness of my new code - [ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing - [ ] I am adding new logging messages - [ ] I am adding a new telemetry message - [ ] I am adding new credentials - [ ] I am adding a new dependency - [ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes. 3. Please describe how your code solves the related issue. Please write a short description of how your code change solves the related issue. --------- Co-authored-by: Adam Ling <adam.ling@snowflake.com>
sfc-gh-vbudati
approved these changes
Oct 10, 2024
sfc-gh-lspiegelberg
merged commit 5462de5
into
ls-SNOW-1491199-merge-phase0-server-side
35 of 39 checks passed
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refreshed with changes since Snowpark 1.22 release. Enables CTE merge gate to pass. This holds all changes till 1.23 release.