Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up HasUserStopwatch & GetActiveStopwatch #23051

Merged
merged 15 commits into from
Feb 27, 2023

Conversation

zeripath
Copy link
Contributor

GetActiveStopwatch & HasUserStopwatch is a hot piece of code that is repeatedly called and on examination of the cpu profile for TestGit it represents 0.44 seconds of CPU time. This PR reduces this time to 80ms.

GetActiveStopwatch & HasUserStopwatch is a hot piece of code that is
repeatedly called and on examination of the cpu profile for TestGit it
represents 0.44 seconds of CPU time. This PR reduces this time to 80ms.

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath added type/testing performance/speed performance issues with slow downs labels Feb 21, 2023
@codecov-commenter

This comment was marked as off-topic.

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Feb 21, 2023
@lunny
Copy link
Member

lunny commented Feb 22, 2023

Just compared CI spent time with other PRs. One time is 27 minutes, another time is 35 minutes.
At the moment, the last two time spent is 24 minutes and 24 minutes. So I don't think there is an obvious time reduction from this PR.

@wxiaoguang
Copy link
Contributor

Just compared CI spent time with other PRs. One time is 27 minutes, another time is 35 minutes. At the moment, the last two time spent is 24 minutes and 24 minutes. So I don't think there is an obvious time reduction from this PR.

I think those are different/independent problems.

For this PR, it could improve the all Gitea servers performance during daily usage.

For CI speed, it's not related to this PR.

  • The times collected from CI jobs are inaccurate from one to another, and these times are heavily affected by other factors, for example, the server load when the CI runs, or the network, etc.
  • If there is a plan to optimize the CI speed, IMO the first step is to have a well-measured benchmark. The second step is to find the bottle necks (the server load, the slowest tests, etc). Making changes is the last step.

@lunny
Copy link
Member

lunny commented Feb 22, 2023

Maybe I'm wrong to mention CI because you mentioned TestGit. If it's not for CI speed up, which page/feature is the purpose for?

@zeripath
Copy link
Contributor Author

Maybe I'm wrong to mention CI because you mentioned TestGit. If it's not for CI speed up, which page/feature is the purpose for?

There should be a small CI improvement but most of my current PRs are going to be improvements in clock-time of the range of less than a minute. Our variance in CI is more than that.

My opening comment mentions CPU time - this is slightly different from clock-time - it's possible that there is no clock-time improvement as the bulk of time is IO time - however, reducing CPU time is still useful. If go is measuring something as CPU time on the cpuprofile then that may be cpu time the goroutine runner could be using elsewhere.

@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Feb 22, 2023
@zeripath
Copy link
Contributor Author

zeripath commented Feb 22, 2023

  • If there is a plan to optimize the CI speed, IMO the first step is to have a well-measured benchmark. The second step is to find the bottle necks (the server load, the slowest tests, etc). Making changes is the last step.

You've just mentioned the problems with coming up with a well-measured benchmark - there's too many interrelated factors and noise here that I don't think it's fundamentally possible. That's why I'm mostly stabbing in the dark here looking at things that are common based on clocktime and cputime. HasUserStopwatch repeatedly stood out on the cpuprofile for TestGit and given that's really hot code I think it's a good case for optimisation in any case.

Interpretting the FGProf clock time profile for TestGit is difficult but it appears heavily IO bound (unsurprising) so I'm just not certain what else we can do for that apart from just drop the big commit and/or not include the big commit in the merges etc.

@zeripath
Copy link
Contributor Author

@lunny I just want to note that there is a slight change in this code - the use of INNER JOIN for repository and issue means that if there is an invalid stopwatch i.e. one that points to a non-existing issue or repository then this code won't see it.

My opinion is that this is fine for HasUserStopwatch but it is a change.

models/issues/stopwatch.go Outdated Show resolved Hide resolved
Signed-off-by: Andrew Thornton <art27@cantab.net>
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Feb 22, 2023
@delvh delvh added the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Feb 22, 2023
@lunny
Copy link
Member

lunny commented Feb 22, 2023

@lunny I just want to note that there is a slight change in this code - the use of INNER JOIN for repository and issue means that if there is an invalid stopwatch i.e. one that points to a non-existing issue or repository then this code won't see it.

My opinion is that this is fine for HasUserStopwatch but it is a change.

I think that's not the scope of this PR.

@jolheiser jolheiser added this to the 1.20.0 milestone Feb 22, 2023
@lunny
Copy link
Member

lunny commented Feb 23, 2023

CI failure is related.

@jolheiser jolheiser removed the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Feb 23, 2023
Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath
Copy link
Contributor Author

CI failure is related.

Sorry, mistake when dealing with comments - the change induced overriding of the provided issue - have changed to use the name otherIssue instead.

@delvh delvh added the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Feb 26, 2023
@techknowlogick techknowlogick merged commit ef4fc30 into go-gitea:main Feb 27, 2023
@zeripath zeripath deleted the speed-up-get-active-stopwatch branch February 27, 2023 21:29
@jolheiser jolheiser removed the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Feb 27, 2023
zjjhot added a commit to zjjhot/gitea that referenced this pull request Feb 28, 2023
* giteaofficial/main:
  [skip ci] Updated translations via Crowdin
  Change button text for commenting and closing an issue at the same time (go-gitea#23135)
  Don't run unnecessary steps when only docs changed (go-gitea#23103)
  Add word-break to sidebar-item-link (go-gitea#23146)
  Speed up HasUserStopwatch & GetActiveStopwatch (go-gitea#23051)
  Add InsecureSkipVerify to Minio Client for Storage (go-gitea#23166)
  Fix Fomantic UI's `touchstart` fastclick, always use `click` for click events (go-gitea#23065)
  Remove useless comment in go-gitea#23114 (go-gitea#23173)
  Remove xin-u from maintainers (go-gitea#23170)
@go-gitea go-gitea locked and limited conversation to collaborators May 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. performance/speed performance issues with slow downs type/testing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants