Delta housekeeping notebooks #95
Conversation
|
|
There was a problem hiding this comment.
@lorenzorubi-db Nice work!
I added a number of comments, they are mostly intended to align this functionality with the rest of the library.
This looks very promising!
+ improved unit tests
|
@edurdevic thanks for your comments, I'd say that all of them are covered now, please let me know if you find anything missing or new ideas come up note that I've removed the implementation of |
|
|
||
| # DBTITLE 1,Run the discoverx DeltaHousekeeping operation -generates an output object you can apply operations to | ||
| output = ( | ||
| dx.from_tables("lorenzorubi.*.*") |
There was a problem hiding this comment.
Uh, I missed this.
Could you please move "lorenzorubi.." to a widget, and if you want replace it with another example catalog name?
| returns a pandas DataFrame, and converts Spark internal dfs to pandas as soon as they are manageable | ||
| the reason being that DESCRIBE HISTORY / DESCRIBE DETAIL cannot be cached | ||
|
|
||
| TODO reconsider if it is better outside of the class |
There was a problem hiding this comment.
Please remove this TODO
| Would make sense only if using map_chunked from the `DataExplorer` object | ||
| (otherwise tables are writen one by one into Delta with overhead) | ||
|
|
||
| TODO create function in `DataExplorer` that uses this for a chunked |
There was a problem hiding this comment.
Please remove the TODO
| need_optimize_df.reset_index().loc[:, ["catalog", "database", "tableName"]], | ||
| expected_need_optimize.loc[:, ["catalog", "database", "tableName"]], | ||
| ) | ||
| # TODO complete all the tests |
There was a problem hiding this comment.
Please remove the TODO
| def test_process_describe_history_empty_history(spark, dd_click_sales, dh_click_sales): | ||
| dh = DeltaHousekeeping(spark) | ||
| describe_detail_df = spark.createDataFrame(dd_click_sales) | ||
| describe_history_df = spark.createDataFrame(dh_click_sales) |
There was a problem hiding this comment.
NIT: I like to define the DFs inline inside the tests, in order to make the tests more readable.
But that's for the next time :)
|
Closing this PR so that @lorenzorubi-db can re-open from his account |
No description provided.