Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add caching to lychee links checker #5160

Merged
merged 8 commits into from
Apr 10, 2024

Conversation

dmathieu
Copy link
Member

@dmathieu dmathieu commented Apr 5, 2024

This adds caching to the links checker, to hopefully reduce the false positive failures due to 429s.

@dmathieu dmathieu added the Skip Changelog PRs that do not require a CHANGELOG.md entry label Apr 5, 2024
@dmathieu dmathieu marked this pull request as ready for review April 5, 2024 09:12
@dmathieu
Copy link
Member Author

dmathieu commented Apr 5, 2024

I'm currently basing the cache on the GIT commit, but we can make it global if needed.

@MrAlias
Copy link
Contributor

MrAlias commented Apr 5, 2024

Won't caching on the git sha mean a new cache is used for each commit? That will make the cache ineffective as it will be new for each commit, right?

.github/workflows/links-fail-fast.yml Show resolved Hide resolved
.github/workflows/links-fail-fast.yml Outdated Show resolved Hide resolved
.github/workflows/links-fail-fast.yml Show resolved Hide resolved
@dmathieu
Copy link
Member Author

dmathieu commented Apr 8, 2024

Won't caching on the git sha mean a new cache is used for each commit? That will make the cache ineffective as it will be new for each commit, right?

Based on the lychee and actions/cache documentations, I don't think that's accurate.
https://github.com/lycheeverse/lychee-action?tab=readme-ov-file#utilising-the-cache-feature

We do store the sha in the cache name, but if there is no cache hit, the restore-keys argument means we'll still restore the most recent one, no matter the sha.
See https://github.com/actions/cache?tab=readme-ov-file#inputs

@dmathieu
Copy link
Member Author

dmathieu commented Apr 8, 2024

Discussing with @pellared, we have decided to only cache if the run is fully successful, because lychee caches everything, including failures.
See lycheeverse/lychee#1400

Co-authored-by: Robert Pająk <pellared@hotmail.com>
@MrAlias
Copy link
Contributor

MrAlias commented Apr 8, 2024

Won't caching on the git sha mean a new cache is used for each commit? That will make the cache ineffective as it will be new for each commit, right?

Based on the lychee and actions/cache documentations, I don't think that's accurate. https://github.com/lycheeverse/lychee-action?tab=readme-ov-file#utilising-the-cache-feature

We do store the sha in the cache name, but if there is no cache hit, the restore-keys argument means we'll still restore the most recent one, no matter the sha. See https://github.com/actions/cache?tab=readme-ov-file#inputs

Doesn't this mean that we will always use the last cache from "restore-keys" in the normal operation? We almost never re-run the link checker on the same commit.

If this is the case using a global cache would be equivalent and reduce the number of caches stored in actions.

@pellared
Copy link
Member

pellared commented Apr 8, 2024

Doesn't this mean that we will always use the last cache from "restore-keys" in the normal operation? We almost never re-run the link checker on the same commit.

Yes

If this is the case using a global cache would be equivalent and reduce the number of caches stored in actions.

Can you please describe what do you mean by "a global cache"?

Do you mean using a single cache key value? The problem with it is that the action does not save an updates the cache when the cache was restored using the key. See #5160 (comment)

I think this is the main reason why lychee recommends such way of caching.

@MrAlias
Copy link
Contributor

MrAlias commented Apr 8, 2024

Do you mean using a single cache key value? The problem with it is that the action does not save an updates the cache when the cache was restored using the key. See #5160 (comment)

👍

@pellared
Copy link
Member

pellared commented Apr 9, 2024

Waiting for an non-Splunker approval.

Copy link
Contributor

@dashpole dashpole left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked too deeply into lychee, but approving since this follows https://github.com/lycheeverse/lychee-action?tab=readme-ov-file#utilising-the-cache-feature, and based on other approvals.

@pellared pellared merged commit fb02927 into open-telemetry:main Apr 10, 2024
26 checks passed
@dmathieu dmathieu deleted the lychee-cache branch April 11, 2024 06:50
@MrAlias MrAlias added this to the v1.26.0 milestone Apr 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Skip Changelog PRs that do not require a CHANGELOG.md entry
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants