Releases: snowflakedb/snowpark-python
Releases · snowflakedb/snowpark-python
Release
1.22.1 (2024-09-11)
This is a re-release of 1.22.0. Please refer to the 1.22.0 release notes for detailed release content.
1.22.0 (2024-09-10)
Snowpark Python API Updates
New Features
- Added the following new functions in
snowflake.snowpark.functions
:array_remove
ln
Improvements
- Improved documentation for
Session.write_pandas
by makinguse_logical_type
option more explicit. - Added support for specifying the following to
DataFrameWriter.save_as_table
:enable_schema_evolution
data_retention_time
max_data_extension_time
change_tracking
copy_grants
iceberg_config
A dicitionary that can hold the following iceberg configuration options:external_volume
catalog
base_location
catalog_sync
storage_serialization_policy
- Added support for specifying the following to
DataFrameWriter.copy_into_table
:iceberg_config
A dicitionary that can hold the following iceberg configuration options:external_volume
catalog
base_location
catalog_sync
storage_serialization_policy
- Added support for specifying the following parameters to
DataFrame.create_or_replace_dynamic_table
:mode
refresh_mode
initialize
clustering_keys
is_transient
data_retention_time
max_data_extension_time
Bug Fixes
- Fixed a bug in
session.read.csv
that caused an error when settingPARSE_HEADER = True
in an externally defined file format. - Fixed a bug in query generation from set operations that allowed generation of duplicate queries when children have common subqueries.
- Fixed a bug in
session.get_session_stage
that referenced a non-existing stage after switching database or schema. - Fixed a bug where calling
DataFrame.to_snowpark_pandas
without explicitly initializing the Snowpark pandas plugin caused an error. - Fixed a bug where using the
explode
function in dynamic table creation caused a SQL compilation error due to improper boolean type casting on theouter
parameter.
Snowpark Local Testing Updates
New Features
- Added support for type coercion when passing columns as input to UDF calls.
- Added support for
Index.identical
.
Bug Fixes
- Fixed a bug where the truncate mode in
DataFrameWriter.save_as_table
incorrectly handled DataFrames containing only a subset of columns from the existing table. - Fixed a bug where function
to_timestamp
does not set the default timezone of the column datatype.
Snowpark pandas API Updates
New Features
- Added limited support for the
Timedelta
type, including the following features. Snowpark pandas will raiseNotImplementedError
for unsupportedTimedelta
use cases.- supporting tracking the Timedelta type through
copy
,cache_result
,shift
,sort_index
,assign
,bfill
,ffill
,fillna
,compare
,diff
,drop
,dropna
,duplicated
,empty
,equals
,insert
,isin
,isna
,items
,iterrows
,join
,len
,mask
,melt
,merge
,nlargest
,nsmallest
,to_pandas
. - converting non-timedelta to timedelta via
astype
. NotImplementedError
will be raised for the rest of methods that do not supportTimedelta
.- support for subtracting two timestamps to get a Timedelta.
- support indexing with Timedelta data columns.
- support for adding or subtracting timestamps and
Timedelta
. - support for binary arithmetic between two
Timedelta
values. - support for binary arithmetic and comparisons between
Timedelta
values and numeric values. - support for lazy
TimedeltaIndex
. - support for
pd.to_timedelta
. - support for
GroupBy
aggregationsmin
,max
,mean
,idxmax
,idxmin
,std
,sum
,median
,count
,any
,all
,size
,nunique
,head
,tail
,aggregate
. - support for
GroupBy
filtrationsfirst
andlast
. - support for
TimedeltaIndex
attributes:days
,seconds
,microseconds
andnanoseconds
. - support for
diff
with timestamp columns onaxis=0
andaxis=1
- support for
TimedeltaIndex
methods:ceil
,floor
andround
. - support for
TimedeltaIndex.total_seconds
method.
- supporting tracking the Timedelta type through
- Added support for index's arithmetic and comparison operators.
- Added support for
Series.dt.round
. - Added documentation pages for
DatetimeIndex
. - Added support for
Index.name
,Index.names
,Index.rename
, andIndex.set_names
. - Added support for
Index.__repr__
. - Added support for
DatetimeIndex.month_name
andDatetimeIndex.day_name
. - Added support for
Series.dt.weekday
,Series.dt.time
, andDatetimeIndex.time
. - Added support for
Index.min
andIndex.max
. - Added support for
pd.merge_asof
. - Added support for
Series.dt.normalize
andDatetimeIndex.normalize
. - Added support for
Index.is_boolean
,Index.is_integer
,Index.is_floating
,Index.is_numeric
, andIndex.is_object
. - Added support for
DatetimeIndex.round
,DatetimeIndex.floor
andDatetimeIndex.ceil
. - Added support for
Series.dt.days_in_month
andSeries.dt.daysinmonth
. - Added support for
DataFrameGroupBy.value_counts
andSeriesGroupBy.value_counts
. - Added support for
Series.is_monotonic_increasing
andSeries.is_monotonic_decreasing
. - Added support for
Index.is_monotonic_increasing
andIndex.is_monotonic_decreasing
. - Added support for
pd.crosstab
. - Added support for
pd.bdate_range
and included business frequency support (B, BME, BMS, BQE, BQS, BYE, BYS) for bothpd.date_range
andpd.bdate_range
. - Added support for lazy
Index
objects aslabels
inDataFrame.reindex
andSeries.reindex
. - Added support for
Series.dt.days
,Series.dt.seconds
,Series.dt.microseconds
, andSeries.dt.nanoseconds
. - Added support for creating a
DatetimeIndex
from anIndex
of numeric or string type. - Added support for string indexing with
Timedelta
objects. - Added support for
Series.dt.total_seconds
method.
Improvements
- Improve concat, join performance when operations are performed on series coming from the same dataframe by avoiding unnecessary joins.
- Refactored
quoted_identifier_to_snowflake_type
to avoid making metadata queries if the types have been cached locally. - Improved
pd.to_datetime
to handle all local input cases. - Create a lazy index from another lazy index without pulling data to client.
- Raised
NotImplementedError
for Index bitwise operators. - Display a more clear error message when
Index.names
is set to a non-like-like object. - Raise a warning whenever MultiIndex values are pulled in locally.
- Improve warning message for
pd.read_snowflake
include the creation reason when temp table creation is triggered. - Improve performance for
DataFrame.set_index
, or settingDataFrame.index
orSeries.index
by avoiding checks require eager evaluation. As a consequence, when the new index that does not match the currentSeries
/DataFrame
object length, aValueError
is no longer raised. Instead, when theSeries
/DataFrame
object is longer than the provided index, theSeries
/DataFrame
's new index is filled withNaN
values for the "extra" elements. Otherwise, the extra values in the provided index are ignored.
Bug Fixes
- Stopped ignoring nanoseconds in
pd.Timedelta
scalars. - Fixed AssertionError in tree of binary operations.
- Fixed bug in
Series.dt.isocalendar
using a named Series - Fixed
inplace
argument for Series objects derived from DataFrame columns. - Fixed a bug where
Series.reindex
andDataFrame.reindex
did not update the result index's name correctly. - Fixed a bug where
Series.take
did not error whenaxis=1
was specified.
Release
1.21.1 (2024-09-05)
Snowpark Python API Updates
Bug Fixes
- Fixed a bug where using
to_pandas_batches
with async jobs caused an error due to improper handling of waiting for asynchronous query completion.
Release
1.21.0 (2024-08-19)
Snowpark Python API Updates
New Features
- Added support for
snowflake.snowpark.testing.assert_dataframe_equal
that is a utility function to check the equality of two Snowpark DataFrames.
Improvements
- Added support server side string size limitations.
- Added support to create and invoke stored procedures, UDFs and UDTFs with optional arguments.
- Added support for column lineage in the DataFrame.lineage.trace API.
- Added support for passing
INFER_SCHEMA
options toDataFrameReader
viaINFER_SCHEMA_OPTIONS
. - Added support for passing
parameters
parameter toColumn.rlike
andColumn.regexp
. - Added support for automatically cleaning up temporary tables created by
df.cache_result()
in the current session, when the DataFrame is no longer referenced (i.e., gets garbage collected). It is still an experimental feature not enabled by default, and can be enabled by settingsession.auto_clean_up_temp_table_enabled
toTrue
. - Added support for string literals to the
fmt
parameter ofsnowflake.snowpark.functions.to_date
.
Bug Fixes
- Fixed a bug where SQL generated for selecting
*
column has an incorrect subquery. - Fixed a bug in
DataFrame.to_pandas_batches
where the iterator could throw an error if certain transformation is made to the pandas dataframe due to wrong isolation level. - Fixed a bug in
DataFrame.lineage.trace
to split the quoted feature view's name and version correctly. - Fixed a bug in
Column.isin
that caused invalid sql generation when passed an empty list. - Fixed a bug that fails to raise NotImplementedError while setting cell with list like item.
Snowpark Local Testing Updates
New Features
- Added support for the following APIs:
- snowflake.snowpark.functions
rank
dense_rank
percent_rank
cume_dist
ntile
datediff
array_agg
- snowflake.snowpark.column.Column.within_group
- snowflake.snowpark.functions
- Added support for parsing flags in regex statements for mocked plans. This maintains parity with the
rlike
andregexp
changes above.
Bug Fixes
- Fixed a bug where Window Functions LEAD and LAG do not handle option
ignore_nulls
properly. - Fixed a bug where values were not populated into the result DataFrame during the insertion of table merge operation.
Improvements
- Fix pandas FutureWarning about integer indexing.
Snowpark pandas API Updates
New Features
- Added support for
DataFrame.backfill
,DataFrame.bfill
,Series.backfill
, andSeries.bfill
. - Added support for
DataFrame.compare
andSeries.compare
with default parameters. - Added support for
Series.dt.microsecond
andSeries.dt.nanosecond
. - Added support for
Index.is_unique
andIndex.has_duplicates
. - Added support for
Index.equals
. - Added support for
Index.value_counts
. - Added support for
Series.dt.day_name
andSeries.dt.month_name
. - Added support for indexing on Index, e.g.,
df.index[:10]
. - Added support for
DataFrame.unstack
andSeries.unstack
. - Added support for
DataFrame.asfreq
andSeries.asfreq
. - Added support for
Series.dt.is_month_start
andSeries.dt.is_month_end
. - Added support for
Index.all
andIndex.any
. - Added support for
Series.dt.is_year_start
andSeries.dt.is_year_end
. - Added support for
Series.dt.is_quarter_start
andSeries.dt.is_quarter_end
. - Added support for lazy
DatetimeIndex
. - Added support for
Series.argmax
andSeries.argmin
. - Added support for
Series.dt.is_leap_year
. - Added support for
DataFrame.items
. - Added support for
Series.dt.floor
andSeries.dt.ceil
. - Added support for
Index.reindex
. - Added support for
DatetimeIndex
properties:year
,month
,day
,hour
,minute
,second
,microsecond
,
nanosecond
,date
,dayofyear
,day_of_year
,dayofweek
,day_of_week
,weekday
,quarter
,
is_month_start
,is_month_end
,is_quarter_start
,is_quarter_end
,is_year_start
,is_year_end
andis_leap_year
. - Added support for
Resampler.fillna
andResampler.bfill
. - Added limited support for the
Timedelta
type, including creatingTimedelta
columns andto_pandas
. - Added support for
Index.argmax
andIndex.argmin
.
Improvements
- Removed the public preview warning message when importing Snowpark pandas.
- Removed unnecessary count query from
SnowflakeQueryCompiler.is_series_like
method. Dataframe.columns
now returns native pandas Index object instead of Snowpark Index object.- Refactor and introduce
query_compiler
argument inIndex
constructor to createIndex
from query compiler. pd.to_datetime
now returns a DatetimeIndex object instead of a Series object.pd.date_range
now returns a DatetimeIndex object instead of a Series object.
Bug Fixes
- Made passing an unsupported aggregation function to
pivot_table
raiseNotImplementedError
instead ofKeyError
. - Removed axis labels and callable names from error messages and telemetry about unsupported aggregations.
- Fixed AssertionError in
Series.drop_duplicates
andDataFrame.drop_duplicates
when called aftersort_values
. - Fixed a bug in
Index.to_frame
where the result frame's column name may be wrong where name is unspecified. - Fixed a bug where some Index docstrings are ignored.
- Fixed a bug in
Series.reset_index(drop=True)
where the result name may be wrong. - Fixed a bug in
Groupby.first/last
ordering by the correct columns in the underlying window expression.
Release
1.20.0 (2024-07-17)
Snowpark Python API Updates
Improvements
- Added distributed tracing using open telemetry APIs for table stored procedure function in
DataFrame
:_execute_and_get_query_id
- Added support for the
arrays_zip
function. - Improves performance for binary column expression and
df._in
by avoiding unnecessary cast for numeric values. You can enable this optimization by settingsession.eliminate_numeric_sql_value_cast_enabled = True
. - Improved error message for
write_pandas
when the target table does not exist andauto_create_table=False
. - Added open telemetry tracing on UDxF functions in Snowpark.
- Added open telemetry tracing on stored procedure registration in Snowpark.
- Added a new optional parameter called
format_json
to theSession.SessionBuilder.app_name
function that sets the app name in theSession.query_tag
in JSON format. By default, this parameter is set toFalse
.
Bug Fixes
- Fixed a bug where SQL generated for
lag(x, 0)
was incorrect and failed with error messageargument 1 to function LAG needs to be constant, found 'SYSTEM$NULL_TO_FIXED(null)'
.
Snowpark Local Testing Updates
New Features
- Added support for the following APIs:
- snowflake.snowpark.functions
- random
- snowflake.snowpark.functions
- Added new parameters to
patch
function when registering a mocked function:distinct
allows an alternate function to be specified for when a sql function should be distinct.pass_column_index
passes a named parametercolumn_index
to the mocked function that contains the pandas.Index for the input data.pass_row_index
passes a named parameterrow_index
to the mocked function that is the 0 indexed row number the function is currently operating on.pass_input_data
passes a named parameterinput_data
to the mocked function that contains the entire input dataframe for the current expression.- Added support for the
column_order
parameter to methodDataFrameWriter.save_as_table
.
Bug Fixes
- Fixed a bug that caused DecimalType columns to be incorrectly truncated to integer precision when used in BinaryExpressions.
Snowpark pandas API Updates
New Features
- Added support for
DataFrameGroupBy.all
,SeriesGroupBy.all
,DataFrameGroupBy.any
, andSeriesGroupBy.any
. - Added support for
DataFrame.nlargest
,DataFrame.nsmallest
,Series.nlargest
andSeries.nsmallest
. - Added support for
replace
andfrac > 1
inDataFrame.sample
andSeries.sample
. - Added support for
read_excel
(Uses local pandas for processing) - Added support for
Series.at
,Series.iat
,DataFrame.at
, andDataFrame.iat
. - Added support for
Series.dt.isocalendar
. - Added support for
Series.case_when
except when condition or replacement is callable. - Added documentation pages for
Index
and its APIs. - Added support for
DataFrame.assign
. - Added support for
DataFrame.stack
. - Added support for
DataFrame.pivot
andpd.pivot
. - Added support for
DataFrame.to_csv
andSeries.to_csv
. - Added partial support for
Series.str.translate
where the values in thetable
are single-codepoint strings. - Added support for
DataFrame.corr
. - Allow
df.plot()
andseries.plot()
to be called, materializing the data into the local client - Added support for
DataFrameGroupBy
andSeriesGroupBy
aggregationsfirst
andlast
- Added support for
DataFrameGroupBy.get_group
. - Added support for
limit
parameter whenmethod
parameter is used infillna
. - Added partial support for
Series.str.translate
where the values in thetable
are single-codepoint strings. - Added support for
DataFrame.corr
. - Added support for
DataFrame.equals
andSeries.equals
. - Added support for
DataFrame.reindex
andSeries.reindex
. - Added support for
Index.astype
. - Added support for
Index.unique
andIndex.nunique
.
Bug Fixes
- Fixed an issue when using np.where and df.where when the scalar 'other' is the literal 0.
- Fixed a bug regarding precision loss when converting to Snowpark pandas
DataFrame
orSeries
withdtype=np.uint64
. - Fixed bug where
values
is set toindex
whenindex
andcolumns
contain all columns in DataFrame duringpivot_table
.
Improvements
- Added support for
Index.copy()
- Added support for Index APIs:
dtype
,values
,item()
,tolist()
,to_series()
andto_frame()
- Expand support for DataFrames with no rows in
pd.pivot_table
andDataFrame.pivot_table
. - Added support for
inplace
parameter inDataFrame.sort_index
andSeries.sort_index
.
Release
1.19.0 (2024-06-25)
Snowpark Python API Updates
Improvements
New Features
- Added support for
to_boolean
function. - Added documentation pages for
Index
and its APIs.
Bug Fixes
- Fixed a bug where python stored procedure with table return type fails when run in a task.
- Fixed a bug where df.dropna fails due to
RecursionError: maximum recursion depth exceeded
when the DataFrame has more than 500 columns. - Fixed a bug where
AsyncJob.result("no_result")
doesn't wait for the query to finish execution.
Snowpark Local Testing Updates
New Features
- Added support for the
strict
parameter when registering UDFs and Stored Procedures.
Bug Fixes
- Fixed a bug in convert_timezone that made the setting the source_timezone parameter return an error.
- Fixed a bug where creating DataFrame with empty data of type
DateType
raisesAttributeError
. - Fixed a bug that table merge fails when update clause exists but no update takes place.
- Fixed a bug in mock implementation of
to_char
that raisesIndexError
when incoming column has nonconsecutive row index. - Fixed a bug in handling of
CaseExpr
expressions that raisesIndexError
when incoming column has nonconsecutive row index. - Fixed a bug in implementation of
Column.like
that raisesIndexError
when incoming column has nonconsecutive row index.
Improvements
- Added support for type coercion in the implementation of DataFrame.replace, DataFrame.dropna and the mock function
iff
.
Snowpark pandas API Updates
New Features
- Added partial support for
DataFrame.pct_change
andSeries.pct_change
without thefreq
andlimit
parameters. - Added support for
Series.str.get
. - Added support for
Series.dt.dayofweek
,Series.dt.day_of_week
,Series.dt.dayofyear
, andSeries.dt.day_of_year
. - Added support for
Series.str.__getitem__
(Series.str[...]
). - Added support for
Series.str.lstrip
andSeries.str.rstrip
. - Added support for
DataFrameGroupby.size
andSeriesGroupby.size
. - Added support for
DataFrame.expanding
andSeries.expanding
for aggregationscount
,sum
,min
,max
,mean
,std
, andvar
withaxis=0
. - Added support for
DataFrame.rolling
andSeries.rolling
for aggregationcount
withaxis=0
. - Added support for
Series.str.match
. - Added support for
DataFrame.resample
andSeries.resample
for aggregationsize
.
Bug Fixes
- Fixed a bug that causes output of GroupBy.aggregate's columns to be ordered incorrectly.
- Fixed a bug where
DataFrame.describe
on a frame with duplicate columns of differing dtypes could cause an error or incorrect results. - Fixed a bug in
DataFrame.rolling
andSeries.rolling
sowindow=0
now throwsNotImplementedError
instead ofValueError
Improvements
- Added support for named aggregations in
DataFrame.aggregate
andSeries.aggregate
withaxis=0
. pd.read_csv
reads using the native pandas CSV parser, then uploads data to snowflake using parquet. This enables most of the parameters supported byread_csv
including date parsing and numeric conversions. Uploading via parquet is roughly twice as fast as uploading via CSV.- Initial work to support an
pd.Index
directly in Snowpark pandas. Support forpd.Index
as a first-class component of Snowpark pandas is coming soon. - Added a lazy index constructor and support for
len
,shape
,size
,empty
,to_pandas()
andnames
. Fordf.index
, Snowpark pandas creates a lazy index object. - For
df.columns
, Snowpark pandas supports a non-lazy version of anIndex
since the data is already stored locally.
Release
1.18.0 (2024-05-28)
Snowpark pandas API Updates
New Features
- Added
DataFrame.cache_result
andSeries.cache_result
methods for users to persist DataFrames and Series to a temporary table lasting the duration of the session to improve latency of subsequent operations.
Improvements
- Added partial support for
DataFrame.pivot_table
with noindex
parameter, as well as formargins
parameter. - Updated the signature of
DataFrame.shift
/Series.shift
/DataFrameGroupBy.shift
/SeriesGroupBy.shift
to match pandas 2.2.1. Snowpark pandas does not yet support the newly-addedsuffix
argument, or sequence values ofperiods
. - Re-added support for
Series.str.split
.
Bug Fixes
- Fixed how we support mixed columns for string methods (
Series.str.*
).
Snowpark Local Testing Updates
New Features
- Added support for the following DataFrameReader read options to file formats
csv
andjson
:- PURGE
- PATTERN
- INFER_SCHEMA with value being
False
- ENCODING with value being
UTF8
- Added support for
DataFrame.analytics.moving_agg
andDataFrame.analytics.cumulative_agg_agg
. - Added support for
if_not_exists
parameter during UDF and stored procedure registration.
Bug Fixes
- Fixed a bug that when processing time format, fractional second part is not handled properly.
- Fixed a bug that caused function calls on
*
to fail. - Fixed a bug that prevented creation of map and struct type objects.
- Fixed a bug that function
date_add
was unable to handle some numeric types. - Fixed a bug that
TimestampType
casting resulted in incorrect data. - Fixed a bug that caused
DecimalType
data to have incorrect precision in some cases. - Fixed a bug where referencing missing table or view raises confusing
IndexError
. - Fixed a bug that mocked function
to_timestamp_ntz
can not handle None data. - Fixed a bug that mocked UDFs handles output data of None improperly.
- Fixed a bug where
DataFrame.with_column_renamed
ignores attributes from parent DataFrames after join operations. - Fixed a bug that integer precision of large value gets lost when converted to pandas DataFrame.
- Fixed a bug that the schema of datetime object is wrong when create DataFrame from a pandas DataFrame.
- Fixed a bug in the implementation of
Column.equal_nan
where null data is handled incorrectly. - Fixed a bug where
DataFrame.drop
ignore attributes from parent DataFrames after join operations. - Fixed a bug in mocked function
date_part
where Column type is set wrong. - Fixed a bug where
DataFrameWriter.save_as_table
does not raise exceptions when inserting null data into non-nullable columns. - Fixed a bug in the implementation of
DataFrameWriter.save_as_table
where- Append or Truncate fails when incoming data has different schema than existing table.
- Truncate fails when incoming data does not specify columns that are nullable.
Improvements
- Removed dependency check for
pyarrow
as it is not used. - Improved target type coverage of
Column.cast
, adding support for casting to boolean and all integral types. - Aligned error experience when calling UDFs and stored procedures.
- Added appropriate error messages for
is_permanent
andanonymous
options in UDFs and stored procedures registration to make it more clear that those features are not yet supported. - File read operation with unsupported options and values now raises
NotImplementedError
instead of warnings and unclear error information.
Release
1.17.0 (2024-05-21)
Snowpark Python API Updates
New Features
- Added support to add a comment on tables and views using the functions listed below:
DataFrameWriter.save_as_table
DataFrame.create_or_replace_view
DataFrame.create_or_replace_temp_view
DataFrame.create_or_replace_dynamic_table
Improvements
- Improved error message to remind users to set
{"infer_schema": True}
when reading CSV file without specifying its schema.
Snowpark pandas API Updates
New Features
- Start of Public Preview of Snowpark pandas API. Refer to the Snowpark pandas API Docs for more details.
Snowpark Local Testing Updates
New Features
- Added support for NumericType and VariantType data conversion in the mocked function
to_timestamp_ltz
,to_timestamp_ntz
,to_timestamp_tz
andto_timestamp
. - Added support for DecimalType, BinaryType, ArrayType, MapType, TimestampType, DateType and TimeType data conversion in the mocked function
to_char
. - Added support for the following APIs:
- snowflake.snowpark.functions:
- to_varchar
- snowflake.snowpark.DataFrame:
- pivot
- snowflake.snowpark.Session:
- cancel_all
- snowflake.snowpark.functions:
- Introduced a new exception class
snowflake.snowpark.mock.exceptions.SnowparkLocalTestingException
. - Added support for casting to FloatType
Bug Fixes
- Fixed a bug that stored procedure and UDF should not remove imports already in the
sys.path
during the clean-up step. - Fixed a bug that when processing datetime format, the fractional second part is not handled properly.
- Fixed a bug that on Windows platform that file operations was unable to properly handle file separator in directory name.
- Fixed a bug that on Windows platform that when reading a pandas dataframe, IntervalType column with integer data can not be processed.
- Fixed a bug that prevented users from being able to select multiple columns with the same alias.
- Fixed a bug that
Session.get_current_[schema|database|role|user|account|warehouse]
returns upper-cased identifiers when identifiers are quoted. - Fixed a bug that function
substr
andsubstring
can not handle 0-basedstart_expr
.
Improvements
- Standardized the error experience by raising
SnowparkLocalTestingException
in error cases which is on par withSnowparkSQLException
raised in non-local execution. - Improved error experience of
Session.write_pandas
method thatNotImplementError
will be raised when called. - Aligned error experience with reusing a closed session in non-local execution.
Release
1.16.0 (2024-05-07)
New Features
- Added snowflake.snowpark.Session.lineage.trace to explore data lineage of Snowflake objects.
- Support stored procedure registration with packages given as Python modules.
- Added support for structured type schema parsing.
Bug Fixes
- Fixed a bug that when inferring a schema, single quotes are added to stage files that already have single quotes.
Local Testing Updates
New Features
- Added support for StringType, TimestampType and VariantType data conversion in the mocked function
to_date
. - Added support for the following APIs:
- snowflake.snowpark.functions
- get
- concat
- concat_ws
- snowflake.snowpark.functions
Bug Fixes
- Fixed a bug that caused NaT and NaN values to not be recognized.
- Fixed a bug when inferring schema, single quotes are added to stage files already have single quotes.
- Fixed a bug where DataFrameReader.csv was unable to handle quoted values containing a delimiter.
- Fixed a bug that when there is
None
value in an arithmetic calculation, the output should remainNone
instead ofmath.nan
. - Fixed a bug in function
sum
andcovar_pop
that when there ismath.nan
in the data, the output should also bemath.nan
. - Fixed a bug that stage operation can not handle directories.
- Fixed a bug that
DataFrame.to_pandas
should take Snowflake numeric types with precision 38 asint64
.
Release
1.15.0 (2024-04-24)
New Features
- Added
truncate
save mode inDataFrameWrite
to overwrite existing tables by truncating the underlying table instead of dropping it. - Added telemetry to calculate query plan height and number of duplicate nodes during collect operations.
- Added the functions below to unload data from a
DataFrame
into one or more files in a stage:DataFrame.write.json
DataFrame.write.csv
DataFrame.write.parquet
- Added distributed tracing using open telemetry APIs for action functions in
DataFrame
andDataFrameWriter
:- snowflake.snowpark.DataFrame:
- collect
- collect_nowait
- to_pandas
- count
- show
- snowflake.snowpark.DataFrameWriter:
- save_as_table
- snowflake.snowpark.DataFrame:
- Added support for snow:// URLs to
snowflake.snowpark.Session.file.get
andsnowflake.snowpark.Session.file.get_stream
- Added support to register stored procedures and UDxFs with a
comment
. - UDAF client support is ready for public preview. Please stay tuned for the Snowflake announcement of UDAF public preview.
- Added support for dynamic pivot. This feature is currently in private preview.
Improvements
- Improved the generated query performance for both compilation and execution by converting duplicate subqueries to Common Table Expressions (CTEs). It is still an experimental feature not enabled by default, and can be enabled by setting
session.cte_optimization_enabled
toTrue
.
Bug Fixes
- Fixed a bug where
statement_params
was not passed to query executions that register stored procedures and user defined functions. - Fixed a bug causing
snowflake.snowpark.Session.file.get_stream
to fail for quoted stage locations. - Fixed a bug that an internal type hint in
utils.py
might raise AttributeError in case the underlying module can not be found.
Local Testing Updates
New Features
- Added support for registering UDFs and stored procedures.
- Added support for the following APIs:
- snowflake.snowpark.Session:
- file.put
- file.put_stream
- file.get
- file.get_stream
- read.json
- add_import
- remove_import
- get_imports
- clear_imports
- add_packages
- add_requirements
- clear_packages
- remove_package
- udf.register
- udf.register_from_file
- sproc.register
- sproc.register_from_file
- snowflake.snowpark.functions
- current_database
- current_session
- date_trunc
- object_construct
- object_construct_keep_null
- pow
- sqrt
- udf
- sproc
- snowflake.snowpark.Session:
- Added support for StringType, TimestampType and VariantType data conversion in the mocked function
to_time
.
Bug Fixes
- Fixed a bug that null filled columns for constant functions.
- Fixed a bug that implementation of to_object, to_array and to_binary to better handle null inputs.
- Fixed a bug that timestamp data comparison can not handle year beyond 2262.
- Fixed a bug that
Session.builder.getOrCreate
should return the created mock session.
Release
1.14.0 (2024-03-20)
New Features
- Added support for creating vectorized UDTFs with
process
method. - Added support for dataframe functions:
- to_timestamp_ltz
- to_timestamp_ntz
- to_timestamp_tz
- locate
- Added support for ASOF JOIN type.
- Added support for the following local testing APIs:
- snowflake.snowpark.functions:
- to_double
- to_timestamp
- to_timestamp_ltz
- to_timestamp_ntz
- to_timestamp_tz
- greatest
- least
- convert_timezone
- dateadd
- date_part
- snowflake.snowpark.Session:
- get_current_account
- get_current_warehouse
- get_current_role
- use_schema
- use_warehouse
- use_database
- use_role
- snowflake.snowpark.functions:
Bug Fixes
- Fixed a bug in
SnowflakePlanBuilder
thatsave_as_table
does not filter column that name start with '$' and follow by number correctly. - Fixed a bug that statement parameters may have no effect when resolving imports and packages.
- Fixed bugs in local testing:
- LEFT ANTI and LEFT SEMI joins drop rows with null values.
- DataFrameReader.csv incorrectly parses data when the optional parameter
field_optionally_enclosed_by
is specified. - Column.regexp only considers the first entry when
pattern
is aColumn
. - Table.update raises
KeyError
when updating null values in the rows. - VARIANT columns raise errors at
DataFrame.collect
. count_distinct
does not work correctly when counting.- Null values in integer columns raise
TypeError
.
Improvements
- Added telemetry to local testing.
- Improved the error message of
DataFrameReader
to raiseFileNotFound
error when reading a path that does not exist or when there are no files under the path.