Releases: dyvenia/viadot
Releases · dyvenia/viadot
Viadot 0.4.26
What's Changed
Added
- Added option for
SAP RFC
connector to get credentials from Azure KeyVault or directly passing dictionary inside flow.
Fixed
- Fixed the
if_exists
parameter definition in theCreateTableFromBlob
task. - Changed
requirements.txt
to level up version ofdbt-sqlserver
in order to fix the bug withMAXRECURSION
error in dbt_run.
Changed
- Changed
dbt-sqlserver
version togit+https://github.com/djagoda881/dbt-sqlserver.git@v1.3.latest_option_clause
.
Removed
- Removed
dbt-core==1.3.2
fromrequirements.txt
. - Removed copying files to conformed/ and operational/ directories when running
ADLSTOAzureSQL
flow.
Shortcut
- 🎨 Delete promote_to task from the flow by @malgorzatagwinner in #849
- changed upstream order for df_validation by @dominikjedlinski in #868
- 🐛 Changed dbt packages in requirements.txt by @adrian-wojcik in #855
- Fixed the
if_exists
parameter inCreateTableFromBlob
task by @Diego-H-S in #874 - ♻️ Changed credentials logic in sap_rfc to get credentials from KeyVault by @adrian-wojcik in #866
- Modify
TransformAndCatalogToLuma
process and commands by @Rafalz13 in #860 - 🐛 Fix bug related to logger.warning and tests in sap_rfc source by @adrian-wojcik in #890
- ♻️ Change
sap_credentials
variable tocredentials
by @adrian-wojcik in #891 - Release 0.4.26 PR by @Rafalz13 in #892
Full Changelog: v0.4.25...v0.4.26
Viadot 0.4.25
What's Changed
Added
- Added logic for if_empty param:
check_if_df_empty
task toADLSToAzureSQL
flow. - Added
geopy
library torequirements
. - Added new parameter
validate_df_dict
toADLSToAzureSQL
class. - Added new ViewType
agent_timeline_summary_view
to Genesys.
Shortcut
- GitHub action bug by @burzekj in #832
- Modified
if_empty
logic inADLSToAzureSQL
by @burzekj in #833 - Fixed json normalize by @KaterynaIurieva in #829
validate_df
implementation toADLSToAzureSQL
by @burzekj in #834- Cleaned small flaws in tests by @KaterynaIurieva in #844
- New Viewtype agent timeline by @Diego-H-S in #846
- Added
geopy
library to requirements by @Rafalz13 in #841 - Release 0.4.25 PR by @Rafalz13 in #847
Full Changelog: v0.4.24...v0.4.25
Viadot 0.4.24
What's Changed
Fixed
task_utils/get_nested_value
fixed issue with non dict parameter passed without level(1st workflow)
Shortcut
- Hotfix/check_nested_value non dict value handling by @marcinpurtak in #830
- Release 0.4.24 PR by @Rafalz13 in #831
Full Changelog: v0.4.23...v0.4.24
Viadot 0.4.23
What's Changed
Added
- Added tests for new functionalities in SAPRFC and SAPRFCV2 regarding passing credentials.
- Added new params for mapping and reordering DataFrame for
Genesys
task and flow. - Added
get_task_logs
task to search for logs in the flow - Added
get_flow_run_id
task to find flow ID. - Added
search_for_msg_in_logs
task used to control flows in multiflows by searching for a given log message from a given task. - Added closing session to
SAPBW
. - Added
CSV
as a new output extension toSharepointListToADLS
flow.
Fixed
- Fixed creation of URL in
VidClub
source class. When theregion=None
the region parameter will not be included in the URL.
Changed
if_no_data_returned
added for sharepoint list flow which can fail, warn in case of no data returend or skip (continue) execution in the old way.- Changed
__init__
inSAPRFC
andSAPRFCV2
class in source in order to raise warning in prefect when credentials will be taken from DEV.
Shortcut
- Web msg logic fix by @burzekj in #820
- Sharepoint list to csv by @cgildenia in #823
- added conn.close after each session to sappw by @gwieloch in #825
- 🐛 Added warning logger for credential by @adrian-wojcik in #824
- Fix url in vidclub connector by @KaterynaIurieva in #822
- Feature/sharepoint multiflows control by @marcinpurtak in #826
- Improve
utils.py
test coverage by @Rafalz13 in #817 - Fix tests before release by @Rafalz13 in #827
- Release 0.4.23 PR by @Rafalz13 in #828
New Contributors
- @KaterynaIurieva made their first contribution in #822
Full Changelog: v0.4.22...v0.4.23
Viadot 0.4.22
What's Changed
Added
- Added
TM1
source class. - Added
TM1ToDF
task class. - Added
set_prefect_kv
parameter toBigQueryToADLS
withFalse
as a default. If there is a need to create new pair in KV Store the parameter can be changed toTrue
. - Added
_rename_duplicated_fields
method toSharepointListToDF
task class for finding and rename duplicated columns. - Added new view type
agent_interaction_view_type
inGenesys
source. - Added new logic for endpoint
users
inGenesys
task. - Added libraries
nltk
andsklearn
torequirements
.
Fixed
- Fixed bug for endpoint
conversations
in GET method inGenesys
Task.
Changed
- Splitted test for
Eurostat
on source tests and task tests. - Modified
SharepointList
source class:
-> Docstrings update. - Modified
SharepointToADLS
flow class:
-> Docstrings update.
-> Changed set_prefect_kv: bool = False to prevent forced KV store append. - Modified
SharepointListToADLS
flow class:
-> Changed set_prefect_kv: bool = False to prevent forced KV store append. - Modified
SharepointList
source class:
-> Docstrings update.
-> Changed_unpack_fields
method to handle Sharepoint MultiChoiceField type + small improvements.
-> Changedget_fields
method to handle special characters - different approach to call get() and execute_query().
-> Renamed method fromselect_expandable_user_fields
toselect_fields
+ update for MultiChoiceField type.
-> Changedcheck_filters
method errors messages and more checks added.
-> Changedoperators_mapping
method errors messages.
-> Changedmake_filter_for_df
method errors messages. - Modified
SharepointListToDF
task class:
-> Docstrings update - Splitted test for Eurostat on source tests and task tests.
- Modified
CustomerGauge
source class with simplified logic to return json structure. - Expanded
CustomerGaugeToDF
task class with separate cleaning functions and handling nested json structure flattening with two new methods_field_reference_unpacker
and_nested_dict_transformer
. - Changed
CustomerGaugeToADLS
to containing new arguments.
Shortcut
- Genesys agent interaction view type by @Diego-H-S in #803
- Added TM1 connector by @angelika233 in #801
- Eurostat tests improvments by @adrian-wojcik in #806
- ⚡️ Added
set_prefect_kv
parameter toBigQueryToADLS
flow by @Rafalz13 in #809 - Improve test coverage for viadot source package by @Rafalz13 in #807
- Dev customer gauge unpacker by @hhnks in #800
- ✨ Add new requirements by @malgorzatagwinner in #813
- Feature/sharepoint list extension kpi5 by @marcinpurtak in #810
- web_msg logic fix by @burzekj in #812
- Fixed typo in tests by @marcinpurtak in #814
- ✨ new logic to extracting users from genesys by @burzekj in #811
- Revert of changes for desired columns by @marcinpurtak in #815
- Release 0.4.22 PR by @Rafalz13 in #816
New Contributors
- @malgorzatagwinner made their first contribution in #813
Full Changelog: v0.4.21...v0.4.22
Viadot 0.4.21
[0.4.21] - 2023-10-26
Added
- Added
validate_df
task to task_utils. - Added
SharepointList
source class. - Added
SharepointListToDF
task class. - Added
SharepointListToADLS
flow class. - Added tests for
SharepointList
. - Added
get_nested_dict
to untils.py.
Fixed
Changed
- Changed
GenesysToCSV
logic for end_point == "conversations". Added new fields to extraction.
Viadot 0.4.20
What's Changed
Added
- Added
Office365-REST-Python-Client
library torequirements
. - Added the
GetSalesQuotationData
view in theBusinessCore
source. - Added new ViewType
queue_interaction_detail_view
to Genesys. - Added new column
_viadot_source
to BigQuery extraction.
Changed
- Changed the flow name from
TransformAndCatalog
toTransformAndCatalogToLuma
. - Modified
add_viadot_metadata_columns
to apply a parameter source_name to the decorator for theto_df
function or function where the DataFrame is generated. - Changed the
SharepointToDF
task in order to implementadd_viadot_metadata_columns
with valuesource_name="Sharepoint"
after changes. - Changed
Mindful
credentials passed by theauth
parameter, instead of theheader
.
Shortcut
- Add
GetSalesQuotationData
to Bussines core by @angelika233 in #752 - Mindful auth by @Diego-H-S in #744
- ♻️ Changed _and_add_viadot_metadata_columns_decorator and aplied it t… by @adrian-wojcik in #758
- 🐛 Fixed bug in test after changing decorator by @adrian-wojcik in #760
- Genesys interaction view type by @Diego-H-S in #761
- Added Office365-REST-Python-Client to the requirements.txt by @burzekj in #753
- 📝 updated documentation by @angelika233 in #754
- ✨ Modify
TransformAndCatalog
flow by @Rafalz13 in #750 - Add
_viadot_source
column toBigQuery
by @Rafalz13 in #763 - Correct tests before release by @Rafalz13 in #764
- Release 0.4.20 PR by @Rafalz13 in #765
Full Changelog: v0.4.19...v0.4.20
Viadot 0.4.19
What's Changed
Added
- Added
add_viadot_metadata_columns
function that will be used as a decorator forto_df
class methods. - Added
TransformAndCatalog
flow. - Added
CloneRepo
task. - Added
LumaIngest
task.
Fixed
- Updated Dockerfile - Changed Linux (RPM/DEB/APK) installer packages.
Shortcut
- Added to .gitignore new assumptions by @burzekj in #739
- Add
TransformAndCatalog
flow by @Rafalz13 in #720 - ✨ Added
add_viadot_metadata_columns
function by @Rafalz13 in #735 - ✨ Updated Dockerfile by @Rafalz13 in #741
- Release 0.4.19 PR by @Rafalz13 in #742
Full Changelog: v0.4.18...v0.4.19
Viadot 0.4.18
What's Changed
Added
- Added
SQLServerToParquet
flow. - Added
SAPBW
source class. - Added
SAPBWToDF
task class. - Added
SAPBWToADLS
flow class. - Added a new
end_point
parameter ingenesys_api_connection
to make it more generic. - Added
VidClubToADLS
flow class.
Fixed
- Fixed a bug in
subject
(extra separator) and inreceivers
(long strings) parameters inOutlook
connector. - Fixed issue with credentials handling in
VidClub
source class. - Fixed issue with missing arguments in
VidClubToDF
task class.
Changed
- Genesys API call method and the name changed from
genesys_generate_exports
togenesys_api_connection
. - Added
GET
connection inside the methodgenesys_api_connection
. - Added new parameters in the
GenesysToCSV
task to be able to extractweb message
files. - Changed looping structure for API calls in
VidClub
source class to use time intervals. - Changed
VidClubToDF
task class to use total_load function from source.
Removed
- Removed methods never used in production:
get_analitics_url_report
,get_all_schedules_job
,schedule_report
,
to_df
,delete_scheduled_report_job
andgenerate_reporting_export
.
Shortcut
- Outlook bug by @Diego-H-S in #710
- Genesys New Queue Performance Method by @Diego-H-S in #715
- added new connector SAPBW by @gwieloch in #707
- Added
SQLServerToParquet
flow. by @angelika233 in #725 - ✨ Added new function for the GET method by @burzekj in #723
- Update logic to
VidClub
source and task by @hhnks in #728 - Improve tests coverage for
BigQuery
andSharepoint
source classes by @Rafalz13 in #716 - Add
VidClubToADLS
flow by @hhnks in #730 - Release 0.4.18 PR by @Rafalz13 in #731
New Contributors
Full Changelog: v0.4.17...v0.4.18
Viadot 0.4.17
Fixed
- Fixed issue with
tzlocal
version for O365 package