Parallelise Report Generator #696
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I've had a go at trying to parallelise the - aimed particularly at helping GitHub Actions / Azure DevOps where the report generator can be quite slow, presumably due to disk IO. I've read some issues about previous attempts and realised it's not as simple as processing classes in parallel as not all the `IReportBuilder` implements support concurrency. With this in mind, I believe two parts of the report generation process (`ReportGenerator`) can be parallelised 1. The initial File Analysis can be done concurrently with the report generation 2. Introduced `IParallelisableReportBuilder` - `IReportBuilder`s that also implement `IParallelisableReportBuilder` can then be processed in parallel this PR has a draft implementation of this, and with the change, I have been able to get report generation (measuring time spent in `Generator.GenerateReport`) down from 21 secs to down as low as 4 seconds with extreme concurrency, and 6secs with more reasonable concurrency levels. This PR is still a bit of a work in progress - in particular I need to plumb the concurrency level through as a config option, and I also want to do some testing with GitHub Actions / Azure Devops. However I wanted to raise it in it's current form for some initial thoughts on the approach. Note, this parallelisation change highly benefits form #TODO
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've had a go at trying to parallelise the report generation - aimed particularly at helping GitHub Actions / Azure DevOps where the report generator can be quite slow, presumably due to disk IO. I've read some issues about previous attempts and realised it's not as simple as processing classes in parallel as not all the
IReportBuilderimplements support concurrency.With this in mind, I believe two parts of the report generation process (
ReportGenerator) can be parallelisedIParallelisableReportBuilder-IReportBuilders that also implementIParallelisableReportBuildercan then be processed in parallelthis PR has a draft implementation of this, and with the change, I have been able to get report generation (measuring time spent in
Generator.GenerateReport) down from 21 secs to down as low as 4 seconds with extreme concurrency, and 6secs with more reasonable concurrency levels.This PR is still a bit of a work in progress - in particular I need to plumb the concurrency level through as a config option, and I also want to do some testing with GitHub Actions / Azure Devops. However I wanted to raise it in it's current form for some initial thoughts on the approach.
Note, this parallelisation change highly benefits form #695