Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate crawler #4385

Merged
merged 23 commits into from
Nov 3, 2023
Merged

Automate crawler #4385

merged 23 commits into from
Nov 3, 2023

Conversation

JKarlavige
Copy link
Collaborator

@JKarlavige JKarlavige commented Nov 1, 2023

What are you changing in this pull request and why?

Asana task

Adds Git workflow to trigger an Algolia crawl on PR merges.

To trigger the crawler, add the trigger-crawl label to a PR. Once the PR is merged, the GitHub Action will:

  • Check if the trigger-crawl label is set
  • If so, wait 8 minutes to allow time for production build to complete
  • Start the Algolia crawl

Example run from this PR

Web Team Testing

To test:

  • Open sandbox PR here
  • This PR has the trigger-crawl label set.
  • Open the workflow runs in separate tab
  • Open Algolia crawler dashboard
  • Merge PR, and verify workflow starts and finishes successfully
  • Once the workflow is complete, a new crawl should be running for the docs site (this is set to crawl the live site, so no issue with letting it run through.)

Notes

Rather than using the sleep 480 command to wait 8 minutes before triggering the crawl, I looked into using this git action to watch for a Vercel deployment. However, this watches for preview deploys, so if a previous deploy preview was built from an earlier commit, this step will complete instantly and will allow the crawl to start before the live docs site is rebuilt.

There's another git action for watching Vercel deployments. However their section on dynamically resolving a deployment URL has an example workflow which i'd rather not do due to the complexity of it.

@JKarlavige JKarlavige requested a review from a team as a code owner November 1, 2023 16:54
Copy link

vercel bot commented Nov 1, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
docs-getdbt-com ✅ Ready (Inspect) Visit Preview 💬 Add feedback Nov 3, 2023 11:14am

@github-actions github-actions bot added the size: small This change will take 1 to 2 days to address label Nov 1, 2023
Copy link
Contributor

github-actions bot commented Nov 1, 2023

Check your created Crawler

Check your created index on your Algolia Application

Copy link
Contributor

@mirnawong1 mirnawong1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey @JKarlavige ! this looks good to me - i couldn't test this out in this pr but the testing you provided looks amazing! thank you!

Copy link
Contributor

@john-rock john-rock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

Confirmed opening a PR on the sandbox repo and tagging with trigger-crawl will initiate a crawl after the build has completed.

@JKarlavige
Copy link
Collaborator Author

Will go ahead and merge this. If all works as expected a crawl should be triggered after this production build is complete!

@JKarlavige JKarlavige merged commit d08a2d4 into current Nov 3, 2023
7 checks passed
@JKarlavige JKarlavige deleted the automate-crawler branch November 3, 2023 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size: small This change will take 1 to 2 days to address trigger-crawl
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants