503 error (access denied) for Google Scholar, i.e. how to deal with "Please show you're not a robot"

[GerHobbelt]

Possibly relevant info from another tool which uses Scholar as well: ckreibich/scholar.py#66

Related issue: #2

[GerHobbelt]

Related material from others / same type of trouble for Zotero et al:

• JabRef: https://help.jabref.org/en/GoogleScholar ("traffic limitations")

• Zotero: https://forums.zotero.org/discussion/56179/unable-to-download-more-citations-fro-google-scholar/

• Google Support pages: https://support.google.com/websearch/forum/AAAAgtjJeM4f8CPeypuN_4/?hl=en ("Google Scholar Error 403 after clicking "Import to Bibtex")

• https://piunikaweb.com/2019/05/22/google-scholar-error-403-with-endnote-citations-has-a-workaround/ ("Google Scholar error 403 with EndNote citations has a workaround") -- which says the same as this original Google Support page (Jack Grady answering):

• https://support.google.com/chrome/thread/3959611?hl=en ("Error 403 when trying to export citation from google scholar to EndNote" --> Jack Grady says:

  Jack Grady (5/5/2019)

  I have the same issue, luckily I found someone who knew who to get around this problem...mostly... What you need to do save it to your google library (click the little star next to the cite option). You can then download the citation from there. Still sucks but better than what you had to do before.

• https://stackoverflow.com/questions/8217769/is-there-a-way-to-download-bibtex-from-google-scholar-using-php

[GerHobbelt]

Analysis for Qiqqa: this depends on #2, where we should change to using Chrome+CefSharp so that the Google "I am not a bot" Captchas will be working again.

[GerHobbelt]

There's an additional work-around that I didn't know before: in Scholar, click on the "cite" link (often shown as a very large double-quote icon below the item, next to links to "related articles", etc.) and then a popup is shown in the page where you can view a few different citation formats **and below those you'll see a couple of links to 'download' the citation info in various formats, the first of which is: BibTeX. 🥳 yay!

Source: https://texblog.org/2014/04/22/using-google-scholar-to-download-bibtex-citations/

[Mocabl3nd]

I'm also experiencing this issue. I cant still get pass through the captcha :(

[GerHobbelt]

This makes it a tough nut as it's dependent on #2, which is still some way away (= adapting Qiqqa to use a totally different embedded browser than the old fireFox (a.k.a. XULrunner) in there)

Have you tried running one of the latest experimental v82* releases? I cannot guarantee that the captcha problem will be gone (I get it myself at the oddest times), but at least I don't suffer from it most of the time; automatic BibTeX access is a severe problem on my own machine (for which the work-around posted above is some help at least), so I wonder what the difference in our setups really is that makes Google even more aggressive on your box.

Please check Qiqqa version and if it's not a recent v82, fetch the latest and try that one for a while. If you're not satisfied, you can always "downgrade" back to your old version by re-installing that one: any Qiqqa installer will replace the Qiqqa software already on your machine, while user configuration will be kept intact.

[Mocabl3nd]

Hi! I was able to find a fix for this issue, sort of... haha.  I tried to use my default browser (Google chrome) to search for the Bibtex info for my journal articles because of the captcha problem in Bibtex sniffer. One problem I encountered is that I can't also access the Bibtex file of some articles in Google scholar using my browser. it shows error 403 (Please see photo below). I don't know why it occurs.  So I did a bit of a research to know why this happens. I learned that it has something to do with the IP address. So because of this, I thought of using VPN. Then, IT WORKED!  I was able to retrieve the Bibtex file both from my browser and in BibTeX sniffer. But the downside is, whenever I turn off my VPN, I can't access the Bibtex file from google scholar. it shows that error 403 again.  At least I was able to find a temp solution for this. I hope this information may help you to fix this issue.  Warm regards, Monica On Tuesday, April 21, 2020, 06:39:11 AM GMT+8, Ger Hobbelt <notifications@github.com> wrote: This makes it a tough nut as it's dependent on #2, which is still some way away (= adapting Qiqqa to use a totally different embedded browser than the old fireFox (a.k.a. XULrunner) in there) Have you tried running one of the latest experimental v82* releases? I cannot guarantee that the captcha problem will be gone (I get it myself at the oddest times), but at least I don't suffer from it most of the time; automatic BibTeX access is a severe problem on my own machine (for which the work-around posted above is some help at least), so I wonder what the difference in our setups really is that makes Google even more aggressive on your box. Please check Qiqqa version and if it's not a recent v82, fetch the latest and try that one for a while. If you're not satisfied, you can always "downgrade" back to your old version by re-installing that one: any Qiqqa installer will replace the Qiqqa software already on your machine, while user configuration will be kept intact. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[Leepee]

As another workaround for this, you can login to the browser, and Scholar allows you a few more cracks at it, to get some more docs cited. If you have more issues, you can also "star" the document, and then go to "My Library" and cite from there. That's what I did and managed to cite 50 something articles just now.

[Mocabl3nd]

This is noted. I'll try this one, for sure. Thank you for the tip! Warm regards,Monica  On Monday, July 20, 2020, 09:20:22 PM GMT+8, Leepee <notifications@github.com> wrote: As another workaround for this, you can login to the browser, and Scholar allows you a few more cracks at it, to get some more docs cited. If you have more issues, you can also "star" the document, and then go to "My Library" and cite from there. That's what I did and managed to cite 50 something articles just now. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[SimonDedman]

I've found the VPN (tunnelbear is good since it's free but has a low allowance but you hardly need any for this purpose) + cite (quotation marks) links, works quite well, and used to for original qiqqa. However it only worked for about 50 papers (3 successful VPN nation changes) before giving up. I'll try again tomorrow.

Edit: confirmed working. Bit of a grind but I've completed my ~150 paper backlog using this method. I found better results VPN-ing to western countries (I'm based in the US and it lasted longer today with Canada, UK, Ireland, than yesterday when I included Mexico & Brazil, but that might be random variance). In any case: this is a viable approach. Though like I say, I manually check each one, I don't know if others somehow try to let it work automatically for all at once somehow.

[m0rxy]

Hi,
just wanted to let you know that this is still an issue as one cannot complete the CAPTCHA in the sniffer browser.

Keep up the great works people :)

[GerHobbelt]

@m0rxy: Yup, still an issue. While the Scholar captcha problem won't go away, the underlying problem that's aggravating this is #2. Which will be addressed after I've taken care of the PDF background processes upgrade (PDF rendering, text extraction, OCR). Ergo: this will take a while before it is addressed, unfortunately.

[GerHobbelt]

Additional work-around has been posted in #310.