Skip to content

Releases: dyvenia/viadot

Viadot 0.4.26

11 Apr 09:18
1cb1a4b
Compare
Choose a tag to compare

What's Changed

Added

  • Added option for SAP RFC connector to get credentials from Azure KeyVault or directly passing dictionary inside flow.

Fixed

  • Fixed the if_exists parameter definition in the CreateTableFromBlob task.
  • Changed requirements.txt to level up version of dbt-sqlserver in order to fix the bug with MAXRECURSION error in dbt_run.

Changed

  • Changed dbt-sqlserver version to git+https://github.com/djagoda881/dbt-sqlserver.git@v1.3.latest_option_clause.

Removed

  • Removed dbt-core==1.3.2 from requirements.txt.
  • Removed copying files to conformed/ and operational/ directories when running ADLSTOAzureSQL flow.

Shortcut

Full Changelog: v0.4.25...v0.4.26

Viadot 0.4.25

30 Jan 09:47
8d12bc1
Compare
Choose a tag to compare

What's Changed

Added

  • Added logic for if_empty param: check_if_df_empty task to ADLSToAzureSQL flow.
  • Added geopy library to requirements.
  • Added new parameter validate_df_dict to ADLSToAzureSQL class.
  • Added new ViewType agent_timeline_summary_view to Genesys.

Shortcut

Full Changelog: v0.4.24...v0.4.25

Viadot 0.4.24

08 Dec 13:03
48c14c0
Compare
Choose a tag to compare

What's Changed

Fixed

  • task_utils/get_nested_value fixed issue with non dict parameter passed without level(1st workflow)

Shortcut

Full Changelog: v0.4.23...v0.4.24

Viadot 0.4.23

07 Dec 12:03
b4e3156
Compare
Choose a tag to compare

What's Changed

Added

  • Added tests for new functionalities in SAPRFC and SAPRFCV2 regarding passing credentials.
  • Added new params for mapping and reordering DataFrame for Genesys task and flow.
  • Added get_task_logs task to search for logs in the flow
  • Added get_flow_run_id task to find flow ID.
  • Added search_for_msg_in_logs task used to control flows in multiflows by searching for a given log message from a given task.
  • Added closing session to SAPBW.
  • Added CSV as a new output extension to SharepointListToADLS flow.

Fixed

  • Fixed creation of URL in VidClub source class. When the region=None the region parameter will not be included in the URL.

Changed

  • if_no_data_returned added for sharepoint list flow which can fail, warn in case of no data returend or skip (continue) execution in the old way.
  • Changed __init__ in SAPRFC and SAPRFCV2 class in source in order to raise warning in prefect when credentials will be taken from DEV.

Shortcut

New Contributors

Full Changelog: v0.4.22...v0.4.23

Viadot 0.4.22

15 Nov 12:49
ee8c34d
Compare
Choose a tag to compare

What's Changed

Added

  • Added TM1 source class.
  • Added TM1ToDF task class.
  • Added set_prefect_kv parameter to BigQueryToADLS with False as a default. If there is a need to create new pair in KV Store the parameter can be changed to True.
  • Added _rename_duplicated_fields method to SharepointListToDF task class for finding and rename duplicated columns.
  • Added new view type agent_interaction_view_type in Genesyssource.
  • Added new logic for endpoint users in Genesystask.
  • Added libraries nltk and sklearn to requirements.

Fixed

  • Fixed bug for endpoint conversations in GET method in Genesys Task.

Changed

  • Splitted test for Eurostat on source tests and task tests.
  • Modified SharepointList source class:
    -> Docstrings update.
  • Modified SharepointToADLS flow class:
    -> Docstrings update.
    -> Changed set_prefect_kv: bool = False to prevent forced KV store append.
  • Modified SharepointListToADLS flow class:
    -> Changed set_prefect_kv: bool = False to prevent forced KV store append.
  • Modified SharepointList source class:
    -> Docstrings update.
    -> Changed _unpack_fields method to handle Sharepoint MultiChoiceField type + small improvements.
    -> Changed get_fields method to handle special characters - different approach to call get() and execute_query().
    -> Renamed method from select_expandable_user_fields to select_fields + update for MultiChoiceField type.
    -> Changed check_filters method errors messages and more checks added.
    -> Changed operators_mapping method errors messages.
    -> Changed make_filter_for_df method errors messages.
  • Modified SharepointListToDF task class:
    -> Docstrings update
  • Splitted test for Eurostat on source tests and task tests.
  • Modified CustomerGauge source class with simplified logic to return json structure.
  • Expanded CustomerGaugeToDF task class with separate cleaning functions and handling nested json structure flattening with two new methods _field_reference_unpacker and _nested_dict_transformer.
  • Changed CustomerGaugeToADLS to containing new arguments.

Shortcut

New Contributors

Full Changelog: v0.4.21...v0.4.22

Viadot 0.4.21

26 Oct 15:42
584fa05
Compare
Choose a tag to compare

[0.4.21] - 2023-10-26

Added

  • Added validate_df task to task_utils.
  • Added SharepointList source class.
  • Added SharepointListToDF task class.
  • Added SharepointListToADLS flow class.
  • Added tests for SharepointList.
  • Added get_nested_dict to untils.py.

Fixed

Changed

  • Changed GenesysToCSV logic for end_point == "conversations". Added new fields to extraction.

Viadot 0.4.20

12 Oct 13:42
a6ced4d
Compare
Choose a tag to compare

What's Changed

Added

  • Added Office365-REST-Python-Client library to requirements.
  • Added the GetSalesQuotationData view in the BusinessCore source.
  • Added new ViewType queue_interaction_detail_view to Genesys.
  • Added new column _viadot_source to BigQuery extraction.

Changed

  • Changed the flow name from TransformAndCatalog to TransformAndCatalogToLuma.
  • Modified add_viadot_metadata_columns to apply a parameter source_name to the decorator for the to_df function or function where the DataFrame is generated.
  • Changed the SharepointToDF task in order to implement add_viadot_metadata_columns with value source_name="Sharepoint" after changes.
  • Changed Mindful credentials passed by the auth parameter, instead of the header.

Shortcut

Full Changelog: v0.4.19...v0.4.20

Viadot 0.4.19

31 Aug 10:07
7d5cfd4
Compare
Choose a tag to compare

What's Changed

Added

  • Added add_viadot_metadata_columns function that will be used as a decorator for to_df class methods.
  • Added TransformAndCatalog flow.
  • Added CloneRepo task.
  • Added LumaIngest task.

Fixed

  • Updated Dockerfile - Changed Linux (RPM/DEB/APK) installer packages.

Shortcut

Full Changelog: v0.4.18...v0.4.19

Viadot 0.4.18

27 Jul 11:46
a2d32f2
Compare
Choose a tag to compare

What's Changed

Added

  • Added SQLServerToParquet flow.
  • Added SAPBW source class.
  • Added SAPBWToDF task class.
  • Added SAPBWToADLS flow class.
  • Added a new end_point parameter in genesys_api_connection to make it more generic.
  • Added VidClubToADLS flow class.

Fixed

  • Fixed a bug in subject (extra separator) and in receivers (long strings) parameters in Outlook connector.
  • Fixed issue with credentials handling in VidClub source class.
  • Fixed issue with missing arguments in VidClubToDF task class.

Changed

  • Genesys API call method and the name changed from genesys_generate_exports to genesys_api_connection.
  • Added GET connection inside the method genesys_api_connection.
  • Added new parameters in the GenesysToCSV task to be able to extract web message files.
  • Changed looping structure for API calls in VidClub source class to use time intervals.
  • Changed VidClubToDF task class to use total_load function from source.

Removed

  • Removed methods never used in production: get_analitics_url_report, get_all_schedules_job, schedule_report,
    to_df, delete_scheduled_report_job and generate_reporting_export.

Shortcut

New Contributors

Full Changelog: v0.4.17...v0.4.18

Viadot 0.4.17

15 Jun 15:00
434bb89
Compare
Choose a tag to compare

Fixed

  • Fixed issue with tzlocal version for O365 package