Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ Add orchestration module #917

Merged
merged 276 commits into from
Aug 22, 2024
Merged

✨ Add orchestration module #917

merged 276 commits into from
Aug 22, 2024

Conversation

djagoda881
Copy link

@djagoda881 djagoda881 commented Jun 6, 2024 •
edited by trymzet
Loading

Summary

Closes #915, #948

Importance

Checklist

This PR:

  • follows the guidelines laid out in CONTRIBUTING.md
  • links relevant issue(s)
  • adds/updates tests (if appropriate)
  • adds/updates docstrings (if appropriate)
  • adds an entry in CHANGELOG.md

@djagoda881 djagoda881 force-pushed the 2.0-new-repository-structure branch from 418c1e3 to e1415d0 Compare June 6, 2024 11:31
@djagoda881 djagoda881 added 2.0 viadot 2.0 documentation Improvements or additions to documentation labels Jun 6, 2024
@djagoda881
🔖 Bumped version to 2.0.0-alpha.2
5f50f97
@djagoda881
♻️ Synchronized prefect-viadot with orchiestration/prefect
d7922f0
@djagoda881
🐛 Fixed import in test_git.py
c39ff5c
@djagoda881
🧱 Updated docker-compose.yml
674343b
@djagoda881
🚚 Moved prefect_viadot to src/viadot/orchestration
df3e4ce
@djagoda881
🚚 Changes imports in prefect-viadot
bb7f65c
@djagoda881
⬆️ Added prefect-viadot dependencies to viadot
bec9d3e
@djagoda881
⬆️ Upgraded prefect dependencie
fb62dc2
@djagoda881
🔧 Updated Dockerfile
deedb07
@djagoda881
⬆️ Upgraded dependecies
d89f1e4
@djagoda881
🔥 Depreacted datahub.py
f97c97c
@djagoda881
➕ Added viadot-azure and viadot-aws dependecies
aee4d0b
@djagoda881
🧱 Added viadot-azure.Dockerfile
0a148c7
@djagoda881
🐛 Added import error handlig to all optional sources
035589e
@djagoda881
🐛 Fixed adls import
e3e7c17
@djagoda881
🧱 Added viadot-aws.Dockerfile
cadf4ea
@djagoda881
🐛 Fixed import errors in prefect-viadot
363541c
@djagoda881
✅ Added prefect-viadot test and refactored viadot tests
7ba3915
@djagoda881
🙈 Updated .gitignore file
b1619e2
@djagoda881
➕ Added new dev dependencies
1d49a83
@djagoda881
🧱 Removed not needed packages from viadot-azure.Dockerfile
4c19a04
@djagoda881
➕ Added dependecies to pyproject.toml
a39c1d9
Rafalz13 and others added 2 commits August 7, 2024 13:49
@Rafalz13
✨ Added 0365 (#969)
3303435 
* ✨ Added 0365

* 🚧  Moved `0365` to dependencies
@trymzet
Orchestration last changes (#953)
ebc1a79 
* 🚚 Move manual actions to a subfolder

* 🐛 Fix the incorrect test dir structure & duplication

* 👷 Add CD workflow

* 📌 Update MSSQL driver version & pin mssql-tools

* 📝 Update container instructions

* 👷 Add linter rules

* 🐛 Fix typo

* ♻️ Do not install Databricks by default

* ♻️ Use standard file name

* ♻️ Readd commented out test

* ♻️ Refactor some weird stuff

* ♻️ Fix typo

* ♻️ Remove duplicated docstrings

* ♻️ Remove clutter

* 🎨 Linting

* 🚨 Docs & linting

* 📝 Improve the user guide

* 📝 Further docs improvements

* 📝 Further docs improvements

* 📝 Docs some more

* 📝 Docs - final touches

* 📝 Improve example

* 🚚 Move to correct path

* 🔥 Remove duplicate tests

* ♻️ Update tests

* 📝 Minor improvement

* 🚨 Lint utils

* 🔒️ Remove the insecure `credentials` param

* 🚨 More linting

* 📝 Add SAP RFC installation instructions

* 🚨 More linting

* ⬆️ Bump pyarrow
Fixes #970

* 📌 Update lockfiles

* ✅ Fix all unit tests

* 🔥 Remove dead code

* 🚨 Lint tests

* ✅ Skip broken tests

* 🚨 Lint all remaining tests

* 🚨 Fix remaining linter warnings

* 📌 Update lock files

* ♻️ Minor fixes

* 🧑‍💻 Also publish `latest` tags for all images
For use eg. in the docker-compose file.

* 🐛 Fix typo
@trymzet trymzet changed the title ✨ viadot:2.0.0 new repository structure ✨ Add orchestration module Aug 14, 2024
@trymzet trymzet marked this pull request as ready for review August 14, 2024 12:00
@trymzet
♻️ Update some sources' test configuration to match rest of lib
38dde53
@trymzet
📝 Add more docs on contributing
fac1183
@trymzet
📝 Update a link
662ad38
@trymzet
🐛 Update lock files, removing optional deps
17e91a5
@trymzet
Merge branch '2.0' into 2.0-new-repository-structure
f752987
@trymzet
⬆️ Update dependencies
91d185d
@trymzet
🚨 Linting
04912f4
@trymzet
🐛 Add TOML support to coverage
c2c9d41
@trymzet
✅ Fix _cast_df() test failing on datetimes in pandas 2.0
5b81385
@trymzet
⬆️ Run CI on Python 3.12
090d6bc
@trymzet
➖ Remove unused pytest-cov
1972760
@trymzet
⬆️ Upgrade Python version so Rye CI action uses 3.12
5ecad24
@trymzet
⬆️ Upgrade Python to 3.12 in the images
ee3b66e
@trymzet
📝 Improve container env docs
bf32428
@trymzet
⬇️ Rollback pyarrow to v10.x
795c895 
Also roll back Python to 3.10 as this `pyarrow` version is not compatible with Python 3.12.
@trymzet
♻️ Use a skip_test_on_missing_extra() utils to simplify life
51f8c2d
@trymzet
🧑‍💻 Install dev dependencies in local containers
ad42983
@trymzet
🐛 Fix for broken numpy version
9db5a06
@trymzet
🚧 RedshiftSpectrum source unit tests - WIP
075b333
@trymzet trymzet mentioned this pull request Aug 21, 2024
Open
1 task
@trymzet trymzet merged commit 46b51b7 into 2.0 Aug 22, 2024
3 checks passed
@trymzet trymzet deleted the 2.0-new-repository-structure branch August 22, 2024 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@trymzet trymzet trymzet left review comments

Assignees
No one assigned
Labels
2.0 viadot 2.0 documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@djagoda881 @trymzet @Diego-H-S @angelika233 @fdelgadodyvenia @judynah @Rafalz13