Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add mechanism to check for processing progress at schema level #1043

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

ttngu207
Copy link
Contributor

Add new method progress for the schema object, returning the populate progress for all imported/computed tables for a given schema (also accounts for reserved, error, ignore jobs)

@ttngu207 ttngu207 changed the title add mechanism to call .progress() at schema-level add mechanism to check for processing progress at schema-level Aug 11, 2022
@ttngu207 ttngu207 changed the title add mechanism to check for processing progress at schema-level add mechanism to check for processing progress at schema level Aug 11, 2022
@guzman-raphael
Copy link
Collaborator

@ttngu207 Perhaps we should consider utilizing populate's progress? You should be able to call this to determine the table populate status.

@ttngu207
Copy link
Contributor Author

@guzman-raphael , the problem with table.progress() is that it is not taking into account entries in the jobs table (e.g. reserved, error, ignore).

@guzman-raphael
Copy link
Collaborator

Right, just thinking how we can report progress consistently at the table level and the schema level.

@ttngu207
Copy link
Contributor Author

Yeah, ideally we'd fix the .progress() at the table level to account for reserve, error and ignore jobs, but that would break backward compatibility.

@guzman-raphael
Copy link
Collaborator

We can introduce an option to return as pandas too but have it default to the current behavior.

@ttngu207
Copy link
Contributor Author

You mean for the new schema.progress() to return a query expression instead of pandas dataframe?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants