Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding git workflows with test matrix and common workflows #363

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

chethanuk
Copy link

  1. Adding GitHub workflows to test across various matrix while merging PR or pushing to master or even before releasing new version
          - java: 8
            hadoop: 2.7
            spark-version: 2.4.7
          - java: 8
            hadoop: 3.2
            spark-version: 3.0.2
          - java: 11
            hadoop: 3.2
            spark-version: 3.0.2
          - java: 8
            hadoop: 2.7
            spark-version: 3.0.2
          - java: 11
            hadoop: 2.7
            spark-version: 3.0.2
          - java: 8
            hadoop: 3.2
            spark-version: 3.1.1
          - java: 11
            hadoop: 3.2
            spark-version: 3.1.1
          - java: 8
            hadoop: 2.7
            spark-version: 3.1.1
          - java: 11
            hadoop: 2.7
            spark-version: 3.1.1
  1. Adding workflows for some common git workflows (Github action to mark as stale and close issues) etc.

cc: @twollnik @TammoR

@twollnik
Copy link
Contributor

Hi, Thanks for introducing these changes! We will have a look within the next two weeks.

@chethanuk
Copy link
Author

Hi, Thanks for introducing these changes! We will have a look within the next two weeks.

Sure, I will update it. There are few changes, will push it in weekend [when I get some time]

@chethanuk
Copy link
Author

@twollnik Also, When will be Spark 3.1.1 supported by deequ?

@aviatesk
Copy link
Contributor

see #366

Copy link
Contributor

@twollnik twollnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks super useful overall, thanks! We would like to add just the testing workflow for now to keep the scope of the PR small (and to be honest because we are no github experts and don't feel comfortable maintaining a lot of github config code). Thanks!

hadoop: [2.7, 3.2]
java: [11]
spark-version: [2.4.7, 3.0.2] # 3.1.1 - not supported yet
python-version: [3.7, 3.8]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please clarify why we need different python version and why do we need python at all?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Internally I had mono repo both deequ and python-deequ, I will remove it

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Side note, Having one mono repo for both is better

@chethanuk chethanuk requested a review from twollnik June 10, 2021 06:59
@twollnik
Copy link
Contributor

Thanks so much for your quick reaction on this and for making changes. It will likely take some time for us to react to the updated PR, but we will get around to it!

@chethanuk
Copy link
Author

@twollnik Any update? and Also waiting for Spark 3.1 support as well.

@chethanuk
Copy link
Author

Ping!.

@twollnik
Copy link
Contributor

Thanks for your patience on this.

@sscdotopen could you please take a look and let us know what you think about the proposed changes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants