Viadot 0.4.19

What's Changed

Added

• Added add_viadot_metadata_columns function that will be used as a decorator for to_df class methods.
• Added TransformAndCatalog flow.
• Added CloneRepo task.
• Added LumaIngest task.

Fixed

• Updated Dockerfile - Changed Linux (RPM/DEB/APK) installer packages.

Shortcut

• Added to .gitignore new assumptions by @burzekj in #739
• Add TransformAndCatalog flow by @Rafalz13 in #720
• ✨ Added add_viadot_metadata_columns function by @Rafalz13 in #735
• ✨ Updated Dockerfile by @Rafalz13 in #741
• Release 0.4.19 PR by @Rafalz13 in #742

Full Changelog: v0.4.18...v0.4.19

Viadot 0.4.18

What's Changed

Added

• Added SQLServerToParquet flow.
• Added SAPBW source class.
• Added SAPBWToDF task class.
• Added SAPBWToADLS flow class.
• Added a new end_point parameter in genesys_api_connection to make it more generic.
• Added VidClubToADLS flow class.

Fixed

• Fixed a bug in subject (extra separator) and in receivers (long strings) parameters in Outlook connector.
• Fixed issue with credentials handling in VidClub source class.
• Fixed issue with missing arguments in VidClubToDF task class.

Changed

• Genesys API call method and the name changed from genesys_generate_exports to genesys_api_connection.
• Added GET connection inside the method genesys_api_connection.
• Added new parameters in the GenesysToCSV task to be able to extract web message files.
• Changed looping structure for API calls in VidClub source class to use time intervals.
• Changed VidClubToDF task class to use total_load function from source.

Removed

• Removed methods never used in production: get_analitics_url_report, get_all_schedules_job, schedule_report,
  to_df, delete_scheduled_report_job and generate_reporting_export.

Shortcut

• Outlook bug by @Diego-H-S in #710
• Genesys New Queue Performance Method by @Diego-H-S in #715
• added new connector SAPBW by @gwieloch in #707
• Added SQLServerToParquet flow. by @angelika233 in #725
• ✨ Added new function for the GET method by @burzekj in #723
• Update logic to VidClub source and task by @hhnks in #728
• Improve tests coverage for BigQuery and Sharepoint source classes by @Rafalz13 in #716
• Add VidClubToADLS flow by @hhnks in #730
• Release 0.4.18 PR by @Rafalz13 in #731

New Contributors

• @burzekj made their first contribution in #723

Full Changelog: v0.4.17...v0.4.18

Viadot 0.4.17

Fixed

• Fixed issue with tzlocal version for O365 package

Viadot 0.4.16

What's changed

Added

• Added VidClub source class
• Added VidClubToDF task class
• Added GetPendingSalesOrderData, GetSalesInvoiceData, GetSalesReturnDetailData
GetSalesOrderData endpoints in BusinessCore() source.
• Added url parameter to CustomerGauge source, and endpoint_url parameter to CustomerGaugeToDF task
  and CustomerGaugeToADLS flow. This parameter enables to pass the endpoint URL by the user.
• Added new parameter outbox_list at all levels in the Outlook connector to tag mailbox folders.

Fixed

• Fixed to_parquet() from base.py when there is no directory specified in the path

Changed

• Changed loop when retrieving email in Outlook source file, to cover all possible folders and subfolders.

Shortcut

• Update PULL_REQUEST_TEMPLATE.md by @trymzet in #694
• ✨ Added GetPendingSalesOrderData view by @angelika233 in #689
• Dev add url customer gauge by @hhnks in #679
• Genesys all queues together by @Diego-H-S in #691
• Vid club connector by @cgildenia in #686
• 🐛 Fixed dir path by @angelika233 in #688
• New Outlook approximation by @Diego-H-S in #690
• Release 0.4.16 PR by @Rafalz13 in #705

New Contributors

• @cgildenia made their first contribution in #686

Full Changelog: v0.4.15...v0.4.16

Viadot 0.4.15

Added

• Added BusinessCore source class
• Added BusinessCoreToParquet task class
• Added verify parameter to handle_api_response().
• Added to_parquet() in base.py
• Added new source class SAPRFCV2 in sap_rfc.py with new approximation.
• Added new parameter rfc_replacement to sap_rfc_to_adls.py to replace
  an extra separator character within a string column to avoid conflicts.
• Added rfc_unique_id in SAPRFCV2 to merge chunks on this column.
• Added close_connection() to SAPRFC and SAPRFCV2

Fixed

• Removed try-except sentence and added a new logic to remove extra separators in sap_rfc.py
  source file, to avoid a mismatch in columns length between iterative connections to SAP tables.
• When SAP tables are updated during sap_rfc.py scrip running, if there are chunks, the
  columns in the next chunk are unrelated rows.
• Fixed sap_rfc.py source file to not break down by both,
  and extra separator in a row and adding new rows in the SAP table between iterations.

Viadot 0.4.14

Added

• Added anonymize_df task function to task_utils.py to anonymize data in the dataframe in selected columns.
• Added Hubspot source class
• Added HubspotToDF task class
• Added HubspotToADLS flow class
• Added CustomerGauge source class
• Added CustomerGaugeToDF task class
• Added CustomerGaugeToADLS flow class

Viadot 0.4.13

Added

• Added validate_date_filter parameter to Epicor source, EpicorOrdersToDF task and EpicorOrdersToDuckDB flow.
  This parameter enables user to decide whether or not filter should be validated.
• Added Mediatool source class
• Added MediatoolToDF task class
• Added MediatoolToADLS flow class
• Added option to disable check_dtypes_sort in ADLSToAzureSQL flow.
• Added query parameter to BigQueryToADLS flow and BigqueryToDF task to be able to enter custom SQL query.
• Added new end point conversations/details/query connection to Genesys task.
• Added new task filter_userid in GenesysToADLS flow to filter out by user Ids list, previously passed by the user.

Changed

• Changed parameter name in BigQueryToADLS flow - from credentials_secret to credentials_key

Viadot 0.4.12

Added

• Added view_type_time_sleep to the Genesys queue_performance_detail_view.
• Added FileNotFoundError to catch up failures in MindfulToCSV and when creating SQL tables.
• Added check_dtypes_sort task into ADLSToAzureSQL to check if dtypes is properly sorted.
• Added timeout parameter to all Tasks where it can be added.
• Added timeout parameter to all Flows where it can be added.
• Added adls_bulk_upload task function to task_utils.py
• Added get_survey_list into Mindful Source file.

Changed

• Updated genesys_to_adls.py flow with the adls_bulk_upload task
• Updated mindful_to_adls.py flow with the adls_bulk_upload task
• Changed MindfulToCSV task to download surveys info.

Viadot 0.4.11

Added

• Added into Genesys the new view type AGENT.

Changed

• Changed data extraction logic for Outlook data.

Viadot 0.4.10

Added

• Added credentials_loader function in utils
• Added new columns to Epicor source - RequiredDate and CopperWeight
• Added timeout to DuckDBQuery and SAPRFCToDF
• Added support for SQL queries with comments to DuckDB source
• Added "WITH" to query keywords in DuckDB source
• Added avro-python3 library to requirements

Changed

• Changed duckdb version to 0.5.1
• Added new column into Data Frames created with Mindful.
• Added region parameter as an entry argument in MindfulToADLS.

Fixed

• Fixed incorrect if_exists="delete" handling in DuckDB.create_table_from_parquet()
• Fixed test_duckdb_to_sql_server.py tests - revert to a previous version
• Removed test__check_if_schema_exists() test