Enhanced Great Expectations Integration

[cicdw]

In a joint effort with the great folks on the Great Expectations team, we are planning on enhancing our current integration with GE. There are a few aspects to this work that should be very straightforward:

Prior Art

@lauralorenz successfully researched and built an initial integration as a Great Expectations checkpoint task that is being used by a subset of our users. This resulted in a still outstanding set of discussion points / future directions of work:

• High-level discussion: Design/discuss integration of Great Expectations with Prefect (potentially in Results feature) #2436
• Particular pain point about how GE config is handled: Improve or create new GE task library task to run a checkpoint against in-memory config #3057

Because Prefect tasks can also be built on-the-fly using the @task decorator, there are likely other GE-specific uses and patterns out there that we might not be aware of.

Concrete Goals

In order to improve on the current state of affairs, we want to:

• make any improvements to the currently existing task that take recent GE configuration / data context changes into account (this "audit" should probably come from the GE side)
• possibly add other tasks to the task library that perform other GE functions other than just checkpointing, or at least document recommended patterns and gotchas
• the Prefect team is already working on a new API for publishing task "artifacts", which includes arbitrary markdown that will be natively rendered in the UI; we want to improve the current checkpoint task to automatically publish the GE markdown report for UI visibility (cc: @joshmeek / @znicholasbrown )

Notes

One of the biggest pain points from our earlier work was the differences in execution paradigms: it seemed GE was emphasizing an interactive workflow on a fixed machine, whereas Prefect is built on a deferred execution model in which Flows can be executed on any number of machines in the future, including a distributed network of machines (#3057 describes this well).

Here is a minimal example of using the current GE task within a Flow: https://github.com/PrefectHQ/prefect/blob/master/examples/task_library/great_expectations/great_expectations_flow.py

This discussion can serve as a landing ground for ideas, action items, updates, etc. If we need to sync up on any sticking points, I'm happy to arrange for that as well.

Last but certainly not least, tagging @spbail for helping triage!

Please comment on this discussion if you are a user using (or planning to use) Great Expectations + Prefect and let us know what you're doing, what's going well and what pain points you may be experiencing!

[joshmeek]

Currently working on the artifacts API and I want to run some initial tests with posting markdown using a real world example. Looking through some of the great expectations docs/code now and one thing I'm wondering is what is the recommended way to turn an expectation suite into markdown? A quick look at the code revealed something like render_multiple_validation_result_pages_markdown but it seems to be deprecated now with a different solution in the warning (which appears to be the contents of the function) and I'm wondering if this is the new recommended approach:

validation_results_page_renderer = ValidationResultsPageRenderer(run_info_at_end=run_info_at_end)
rendered_document_content_list = validation_results_page_renderer.render_validation_operator_result(validation_operator_result=validation_operator_result)
return " ".join(DefaultMarkdownPageView().render(rendered_document_content_list))

cc @anthonyburdi

[anthonyburdi]

That is correct @joshmeek!
We deprecated the utility function render_multiple_validation_result_pages_markdown() in favor of creating ValidationResultsPageRenderer.render_validation_operator_result() which can render ValidationOperatorResult objects by calling the render method for each ExpectationSuiteValidationResult. A ValidationOperatorResultcan have multiple ExpectationSuiteValidationResults, for example by running a checkpoint with multiple expectation suites.

[anthonyburdi]

The current Great Expectations checkpoint task looks great! To allow for a bit more flexible configuration of validation, we can potentially allow for the following:

1. Allow for passing in an in-memory data context in lieu of reading from disk. The Prefect task could take optional parameters for either the context_root_dir or an in-memory data context - and house the logic / error checking to make sure only a single one is used. Currently we do support an in memory DataContext (e.g. in Spark / Databricks notebooks), but we are currently in the process of enhancing the API and defaults to make this quite a bit less verbose. See here and here for our current implementation.

2. Add additional configuration options for defining the validation. Similar to 1., give the user the option to pass in one of the following (using optional parameters, with logic / error checking to make sure only one option is used) to define the validation: A. checkpoint, B. expectation_suite + batch_kwargs, C. assets_to_validate (e.g. [{"suite_name": "my_suite", "batch_kwargs": {...}}, {"suite_name": "my_suite_2", "batch_kwargs": {...}}, ...])

[znicholasbrown]

Hi @anthonyburdi ! The UI team is picking up the rendering portion of this, where we're building a generic markdown renderer that we can use in a few different places - in this case for rendering task and flow run artifacts. Since we'll be using this for other purposes, I was hoping we could use GitHub-flavored markdown but whatever Great Expectations uses might be a better place to start; do you have documentation or some info we can use as a starting point for this? What flavor of markdown does GE use and is it extendible/configurable?

cc @ThatGalNatalie

[anthonyburdi]

Hi @znicholasbrown and @ThatGalNatalie!! Nice to meet you. We are using jinja templates to render our validation results to markdown, trying to adhere to GitHub-flavored markdown. You can take a look at all the files prefixed with markdown_ in our templates directory. This reference doc may also be useful as it describes customizing data docs which render to html but the concepts are similar for markdown.

Under the hood, we first render our validation results to a RenderedDocumentContent object (can be exported to json), and then consume it in our templates. The actual rendering happens in DefaultMarkdownPageView where we do things like bold parameter names and add icons.

Hope that is helpful! Please don't hesitate to let me know if you have any questions.

[anthonyburdi]

Hi all! Just checking in to see if there is anything I can help with. Also, just so I can make sure to be available going forward: what is the timeline that you are working towards for this effort?

[cicdw]

Yup that sounds good to me! I'll reach out next week 👍

[joshmeek]

@anthonyburdi the fix works flawlessly! Thanks for making that!

[spbail]

Hi folks, just wanted to check, is there a feature branch for the upcoming release? I'd love to check it out!

[joshmeek]

Hey @spbail @anthonyburdi I just pushed up a PR (#3753) to enhance the RunGreatExpectationsCheckpoint task in the task library w/ new kwargs and make use of markdown artifacts! (there's also a screenshot of the artifacts API in the UI there) I believe the UI branch will be available soon 🙌

[zanieb]

UI draft PR available at PrefectHQ/legacy-ui#484

[spbail]

Howdy! Thanks @cicdw for taking the time to chat and demo today! I went over the code again and would suggest a few tweaks to make the input parameters (batch_kwargs+expectation suite, checkpoint, assets_to_validate) mutually exclusive, since a checkpoint name shouldn't be required when batch_kwargs+expectation suite are provided. Is it ok if I submit a PR to get started with this?

[cicdw]

Yes, absolutely! The new kwargs haven't been released yet so shouldn't be breaking for anyone 👍

[spbail]

I've just opened a draft PR: #3766

I don't have a local dev environment set up to actually test this yet, just wanted to get the code out to start the conversation. If anyone is able to commandeer the PR and run it, that would be super helpful - happy to make tweaks or provide input as needed.

[cicdw]

Sounds good, thanks @spbail! I can make some tweaks on formatting, etc. Because the task name was released in previous versions of Prefect I'll probably deprecate the old name and replace it with the new one instead of removing it in one fell swoop.

[spbail]

That sounds good re: deprecating the old name. Thanks!

[znicholasbrown]

A quick update on this for the GE folks (and anyone else who might be following along!): Artifacts are now released in the UI, meaning reports created with GE Tasks or more general markdown artifacts will now be visible on the Flow and Task Run pages in Cloud (and the latest version of Server).

For the GE folks, you can test the integration by signing up for a Cloud account, and creating any flow with a GE task (or tasks). You'll be able to see the output of the GE task on the Artifacts tabs of either the flow run or task run pages.

Let me know if you have any questions!

cc: @spbail @anthonyburdi @joshmeek @cicdw

[nishuvr]

Hey how exactly did u create this report can u please share the code?

[melgo-tb]

Hey @znicholasbrown is the artifacts UI available only on the prefect Cloud UI or is it open sourced?

[zanieb]

Artifacts is available in Prefect 1 Cloud and Server.

Artifacts are on the roadmap for Prefect 2 and will likely be open source (but I cannot guarantee).