Recommended Setup until Rehydration is Possible

[hhff]

Hi !

I'm looking at using Fastboot for an upcoming build - because strong SEO & metatags is a must. Until Rehydration is working - is there a recommended way of handling the routing:

--> "hi request, you're a crawler, go to my fastboot server for an index.html"
--> "hi request, you're a browser, here's my real app server that will give you an index.html"

Or is this an exercise best left to the developer? Working on the infra stuffs 👍

Thankyou!!

[RuslanZavacky]

@hhff I do that via nginx configuration, as it acts like a proxy in between request & fastboot or prerender. Check user agent and pass to respective backend.

[tomdale]

@tylr is doing this for Bustle, he can probably give you the details on how to detect the Googlebot.

[hhff]

Thanks both - would love some pointers on the finer points of Fastboot SEO & Infra. Bustle is for sure killing it in that world

Sitemap / Full site crawling tips welcome too!!

Will be happy to contribute my findings back to the repo as docs!

[RuslanZavacky]

server {
   ...
   location / {
          try_files $uri @prerender;
        }

        location @prerender {
          set $prerender 0;
          if ($http_user_agent ~* "googlebot|bot|bingbot|adsbot-google|mediapartners-google|baiduspider|twitterbot|facebookexternalhit|rogerbot|linkedinb$
            set $prerender 1;
          }
          if ($args ~ "_escaped_fragment_") {
            set $prerender 1;
          }
          if ($http_user_agent ~ "Prerender") {
            set $prerender 0;
          }
          if ($uri ~ "\.(js|css|xml|less|png|jpg|jpeg|gif|pdf|doc|txt|ico|rss|zip|mp3|rar|exe|wmv|doc|avi|ppt|mpg|mpeg|tif|wav|mov|psd|ai|xls|mp4|m4a|swf$
            set $prerender 0;
          }

          #resolve using Google's DNS server to force DNS resolution and prevent caching of IPs
          resolver 8.8.8.8;

          if ($prerender = 1) {
            #setting prerender as a variable forces DNS resolution since nginx caches IPs and doesnt play well with load balancing
            set $prerender "**prerender.website-url.com:3000**";
            rewrite .* /https://$host$request_uri? break;
            proxy_pass http://$prerender;
          }
          if ($prerender = 0) {
            rewrite .* /index.php;
          }
        }
   ...
}

@hhff this is how I do it with nginx & prerender. It could be similar later on with fastboot

[hhff]

amazing @RuslanZavacky !

[tylr]

While Bustle still uses _escaped_fragment_, you shouldn't use it for new projects and we're migrating away from it as soon as possible.

Google has officially deprecated and dropped support for it and encourages you not to use it.
See here: https://googlewebmastercentral.blogspot.com/2015/10/deprecating-our-ajax-crawling-scheme.html

We do however use (read: need) server-side rendering for social media bots and services. We're also investigating ways of serving a non-rehydratable DOM / skeleton to browsers if we can perfect the re-draw when the app actually boots and provide a satisfactory UX. Which might make use out of it pre-rehydration.

If you want good google SEO, simply follow their best practices and submit a sitemap to webmaster tools.

[tomdale]

Gonna close this for now but we should definitely make sure we put some "Best Practices" documentation together. @hhff maybe you can tackle this once you've tried it out. :)

[hhff]

totally, will do!

feel like having a static full-page loader in the non-rehdratable skeleton HTML and then pulling the loader away shortly after a full-redraw might be a good half-step for this.

[ianpetzer]

@RuslanZavacky ... I'm just curious.. Have you noticed any ill effects from serving the pre-rendered HTML to the googlebot.. According to google docs, you should serve them the same content as your users.

Prerender.io recommend that you don't serve the pre-rendered HTML, as you can see if their config here

I would love to serve them up the pre-rendered HTML from prerender.io as it renders so much faster.

[RuslanZavacky]

@ianpetzer my solution is a bit more complicated, than just serving simple HTML. I store static HTML until I do any changes or information of specific page changes. If for example, I change information on 1 page, I schedule it for HTML cache update. So I am always trying to keep HTML cache up to date. My next improvements would be to place Varnish in front of prerender (or fastboot) so Varnish cache everything, instead I do it manually :)

There is a potential risk, but it mainly relates to the popups, modal windows jumping over, or content disappearing. Like if you want to boost your SEO, you show Google totally different content. In our case, its fine, because we are not doing anything bad :)