Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(sitemap): filter all routes with robots meta containing noindex #7964

Merged
merged 2 commits into from
Aug 17, 2022

Conversation

slorber
Copy link
Collaborator

@slorber slorber commented Aug 17, 2022

Motivation

Bug:

This page meta leads to the page being filtered correctly from sitemap.xml:

<head>
  <meta name="robots" content="noindex" />
</head>

But with this page meta, the page stays in sitemap.xml

<head>
  <meta name="robots" content="noindex, nofollow" />
</head>

Both pages should be filtered consistently.

Also Robots meta directives are case insensitive (https://developers.google.com/search/docs/advanced/robots/robots_meta_tag#directives)

Adding an example and improving our SEO doc a little

Test Plan

unit tests

Compared preview sitemap with a previous preview sitemap and didn't notice anything wrong

Test links

Deploy preview:

Related issues/PRs

#7963 (review)

@slorber slorber added the pr: bug fix This PR fixes a bug in a past release. label Aug 17, 2022
@facebook-github-bot facebook-github-bot added the CLA Signed Signed Facebook CLA label Aug 17, 2022
@netlify
Copy link

netlify bot commented Aug 17, 2022

[V2]

Name Link
🔨 Latest commit d9e41c6
🔍 Latest deploy log https://app.netlify.com/sites/docusaurus-2/deploys/62fce8493671ff0008c86f9f
😎 Deploy Preview https://deploy-preview-7964--docusaurus-2.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@github-actions
Copy link

⚡️ Lighthouse report for the deploy preview of this PR

URL Performance Accessibility Best Practices SEO PWA Report
/ 🟢 93 🟢 98 🟢 100 🟢 100 🟠 80 Report
/docs/installation 🟠 82 🟢 100 🟢 100 🟢 100 🟢 90 Report

@github-actions
Copy link

Size Change: +270 B (0%)

Total Size: 813 kB

Filename Size Change
website/build/assets/js/main.********.js 609 kB +270 B (0%)
ℹ️ View Unchanged
Filename Size
website/.docusaurus/globalData.json 52.5 kB
website/build/assets/css/styles.********.css 111 kB
website/build/index.html 40.8 kB

compressed-size-action

@Josh-Cena Josh-Cena changed the title fix(sitemap-plugin): should filter routes with robots meta "noiNdeX, noFFoLow" from sitemap.xml fix(sitemap): filter all routes with robots meta containing noindex from sitemap Aug 17, 2022
@Josh-Cena Josh-Cena changed the title fix(sitemap): filter all routes with robots meta containing noindex from sitemap fix(sitemap): filter all routes with robots meta containing noindex Aug 17, 2022
Copy link
Collaborator

@Josh-Cena Josh-Cena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@slorber slorber added the to backport This PR is planned to be backported to a stable version of Docusaurus label Aug 17, 2022
@slorber slorber merged commit a335a69 into main Aug 17, 2022
@slorber slorber deleted the slorber/no-Index-sitemap-bug branch August 17, 2022 14:41
@slorber slorber added backported This PR has been backported to a stable version of Docusaurus and removed to backport This PR is planned to be backported to a stable version of Docusaurus labels Sep 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backported This PR has been backported to a stable version of Docusaurus CLA Signed Signed Facebook CLA pr: bug fix This PR fixes a bug in a past release.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants