Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retry mechanism for rustdoc GUI tests to reduce flakyness #106449

Merged
merged 1 commit into from
Jan 8, 2023

Conversation

GuillaumeGomez
Copy link
Member

@GuillaumeGomez GuillaumeGomez commented Jan 4, 2023

Part of #93784.

I added 3 retries for failing GUI tests. An important note: if more than half of total tests fail, I don't retry because it's very likely not flakyness anymore at this point but a missing update after changes.

@rustbot
Copy link
Collaborator

rustbot commented Jan 4, 2023

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. labels Jan 4, 2023
@GuillaumeGomez GuillaumeGomez force-pushed the rustdoc-gui-retry-mechanism branch 2 times, most recently from 418cb4b to e36b5bf Compare January 4, 2023 15:44
@GuillaumeGomez
Copy link
Member Author

I added some documentation on the NB_RETRY constant too.

@jyn514 jyn514 mentioned this pull request Jan 5, 2023
9 tasks
@Mark-Simulacrum
Copy link
Member

@bors r+ rollup=iffy

@bors
Copy link
Contributor

bors commented Jan 7, 2023

📌 Commit f902200 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jan 7, 2023
@bors
Copy link
Contributor

bors commented Jan 8, 2023

⌛ Testing commit f902200 with merge cc47b06...

@bors
Copy link
Contributor

bors commented Jan 8, 2023

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing cc47b06 to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Jan 8, 2023
@bors bors merged commit cc47b06 into rust-lang:master Jan 8, 2023
@rustbot rustbot added this to the 1.68.0 milestone Jan 8, 2023
@GuillaumeGomez GuillaumeGomez deleted the rustdoc-gui-retry-mechanism branch January 8, 2023 20:51
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (cc47b06): comparison URL.

Overall result: ❌ regressions - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
0.7% [0.5%, 1.1%] 8
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
- - 0
All ❌✅ (primary) - - 0

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)
- - 0
Regressions ❌
(secondary)
- - 0
Improvements ✅
(primary)
- - 0
Improvements ✅
(secondary)
-1.5% [-1.7%, -1.3%] 2
All ❌✅ (primary) - - 0

Cycles

This benchmark run did not return any relevant results for this metric.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants