Skip to content

SEO Server is a command line tool that runs a server that allows GoogleBot, as well as other crawlers, to crawl Javascript heavy websites.

License

Notifications You must be signed in to change notification settings

bakerac4/seoserver

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

32 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Welcome!

Seo Server is a command line tool that runs a server that allows GoogleBot(and any other crawlers) to crawl your heavily Javascript built websites. The tool works with very little changes to your server or client side code.

This entire site is driven by Javascript(view the source or see the code). Click the `What does Google see?` button at the bottom of each page to see Seo Server in action.

How it works



Seo Server runs PhantomJs(headless webkit browser) which renders the page fully and returns the fully executed code to GoogleBot.

Getting started

1) you must install PhantomJs(http://phantomjs.org/) and link into your bin so that Seo Server can call it.

2) Seo Server is an NPM module so install via

sudo npm install -g seoserver

3) Now we have access to the Seo Server command line tool

seoserver start

Which starts an Express server on port 3000 or

seoserver -p 4000 start

Start it as a background process and log the output

seoserver -p 4000 start > seoserver.log &

Telling GoogleBot to fetch from Seo Server

To tell GoogleBot that we are using ajaxed content we simply add to our sites index.html file the Google specific meta tag. If you view the source of this page you can see we have included the tag below.

<meta name="fragment" content="!">

Now whenever GoogleBot visits any of our pages it will try to load ?_escaped_fragment_=pathname

So if we were using Apache with mod rewrite and mod proxy, we can include in our .htaccess

RewriteCond %{QUERY_STRING} ^_escaped_fragment_=(.*)$
RewriteRule (.*) http://address-of-seoserver:3000/%1? [P]

Now all request from GoogleBot will be returned fully rendered. How GoogleBot sees the page can be tested with Google WebMasters(they allow you to simulate Google crawls and see the result instantly).

For other crawlers

Using mod rewrite, we can send other crawlers to Seo Server also

RewriteCond %{HTTP_USER_AGENT} ^DuckDuckBot/1.0;
RewriteRule (.*) http://address-of-seoserver:3000/%1? [P]

FAQ

Nothing here yet, but check out the examples on the left to see different types of ajaxed content. Also ask questions and give feedback on GitHub issues.

About

SEO Server is a command line tool that runs a server that allows GoogleBot, as well as other crawlers, to crawl Javascript heavy websites.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • JavaScript 100.0%