New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Search engine crawl improvements #144

Open

sendtogeo opened this issue Sep 6, 2019 · 0 comments

sendtogeo commented Sep 6, 2019

Rotation of general user agents
Ref: http://www.networkinghowtos.com/howto/common-user-agent-list/

Google Chrome

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36

Firefox:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:53.0) Gecko/20100101 Firefox/53.0

Create a new class to process the regex parsing. Add more general regex for search engines, if one is failed
Implement rss feed type for bing to get the results

https://www.bing.com/search?q=seo+panel&format=rss&count=20&setmkt=de-de

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment