Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importer performance #1841

Merged

Conversation

richardebeling
Copy link
Member

Best reviewed by commit. Fixes #1840.

Openpyxl no longer spends half a minute on determining the worksheet dimensions.
We prevent some N+1 query behavior, making the importer and the staff semester page (which the importer redirects to) a bit faster.

On my machine, import times for a 6000 row sheet go from 32-40s to 3-6s.

@richardebeling richardebeling mentioned this pull request Dec 30, 2022
@richardebeling richardebeling force-pushed the 1840-importer-performance-fix branch 2 times, most recently from 333b4d2 to e61b2b1 Compare December 30, 2022 15:25
@richardebeling
Copy link
Member Author

moving the staff semester page N+1 query fix to #1842

Copy link
Member

@niklasmohrin niklasmohrin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

haven't measured or thought about the prefetches yet (more prefetching = more performance, right?); I generally trust you with this, but if you want I can also take a closer look

evap/staff/importers/base.py Outdated Show resolved Hide resolved
evap/staff/importers/enrollment.py Outdated Show resolved Hide resolved
@richardebeling richardebeling merged commit 123b63c into e-valuation:main Jan 16, 2023
@richardebeling richardebeling deleted the 1840-importer-performance-fix branch January 16, 2023 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

Openpyxl max_column computation bottlenecks importers
3 participants