Viadot 0.4.22

What's Changed

Added

• Added TM1 source class.
• Added TM1ToDF task class.
• Added set_prefect_kv parameter to BigQueryToADLS with False as a default. If there is a need to create new pair in KV Store the parameter can be changed to True.
• Added _rename_duplicated_fields method to SharepointListToDF task class for finding and rename duplicated columns.
• Added new view type agent_interaction_view_type in Genesyssource.
• Added new logic for endpoint users in Genesystask.
• Added libraries nltk and sklearn to requirements.

Fixed

• Fixed bug for endpoint conversations in GET method in Genesys Task.

Changed

• Splitted test for Eurostat on source tests and task tests.
• Modified SharepointList source class:
  -> Docstrings update.
• Modified SharepointToADLS flow class:
  -> Docstrings update.
  -> Changed set_prefect_kv: bool = False to prevent forced KV store append.
• Modified SharepointListToADLS flow class:
  -> Changed set_prefect_kv: bool = False to prevent forced KV store append.
• Modified SharepointList source class:
  -> Docstrings update.
  -> Changed _unpack_fields method to handle Sharepoint MultiChoiceField type + small improvements.
  -> Changed get_fields method to handle special characters - different approach to call get() and execute_query().
  -> Renamed method from select_expandable_user_fields to select_fields + update for MultiChoiceField type.
  -> Changed check_filters method errors messages and more checks added.
  -> Changed operators_mapping method errors messages.
  -> Changed make_filter_for_df method errors messages.
• Modified SharepointListToDF task class:
  -> Docstrings update
• Splitted test for Eurostat on source tests and task tests.
• Modified CustomerGauge source class with simplified logic to return json structure.
• Expanded CustomerGaugeToDF task class with separate cleaning functions and handling nested json structure flattening with two new methods _field_reference_unpacker and _nested_dict_transformer.
• Changed CustomerGaugeToADLS to containing new arguments.

Shortcut

• Genesys agent interaction view type by @Diego-H-S in #803
• Added TM1 connector by @angelika233 in #801
• Eurostat tests improvments by @adrian-wojcik in #806
• ⚡️ Added set_prefect_kv parameter to BigQueryToADLS flow by @Rafalz13 in #809
• Improve test coverage for viadot source package by @Rafalz13 in #807
• Dev customer gauge unpacker by @hhnks in #800
• ✨ Add new requirements by @malgorzatagwinner in #813
• Feature/sharepoint list extension kpi5 by @marcinpurtak in #810
• web_msg logic fix by @burzekj in #812
• Fixed typo in tests by @marcinpurtak in #814
• ✨ new logic to extracting users from genesys by @burzekj in #811
• Revert of changes for desired columns by @marcinpurtak in #815
• Release 0.4.22 PR by @Rafalz13 in #816

New Contributors

• @malgorzatagwinner made their first contribution in #813

Full Changelog: v0.4.21...v0.4.22