Error while posting: "Too many requests"

[tichter]

Hi @eduardoboucas,

I found this great framework for audio listening tasks https://github.com/deeuu/listen, using github pages and staticman. I love how easy it works and I implemented my own version of it.
I set up a task and send it to 20 people. Everything worked fine most of the time, but sometimes people and myself got an error while submitting the task via staticman. You can find a screenshot of the error below.

The problem is, that I can't reproduce the error consistently. When it occurs, after reloading the page and redoing the task, the submit works fine.
The only way I could consistently reproduce the error is when I submit the task, click the back button of the browser, in order to get back to the popup and then click submit again.

You can find a test version at https://tichter.github.io/pwlisten/. You can enter the page with the codes "test1" or "test2" (without the "). The site is still under development so you can test as much as you want and post as many requests as you want. I will reset everything before I launch the project.

Do you know where this error comes from? Is it because of the staticman api, or is it an error coming from github? Do you have any solutions for this problem?

Thank you in advance for your help,
Alex Tichter

[jloisel]

We also have this issue, despite the fact we only have a comment once every few days. That looks like an issue on staticman side (ddos attack?).

[onwingslikeeagles]

Hello. My team is using Staticman and we are seeing this issue as well. The issue is happening intermittently as we are submitting data requests to the tool. Since there is a variable that denotes the next time when submitting a request would be valid, there must be a reason why the request are failing. Could a job queue be stored within Staticman that is causing the delay?

[tmadeira]

+1

[leonp]

Similar on my site:

{"error":{"text":"Too many requests in this time frame.","nextValidRequestDate":"2018-08-27T13:14:57.224Z"}}

Intermittent. Assumed it was spambots sending comments and the API stopping requests until a specific time. However, I was able to post a comment before nextValidRequestDate (assuming it's UTC).

Or maybe Staticman is getting too many comments and throttling?

[StephanWeinhold]

I'm afraid it's the throttling. Because I've experienced this yesterday on a site that was live for 5 mins while sending a test comment.

[eduardoboucas]

Staticman itself does not implement any throttling. I'm afraid this must be a limitation from the GitHub API, as everyone using the public instance is pinging the GitHub API using the same API key. Not sure if there's anything I can do.

[leonp]

Thanks Eduardo. Guess it's best we host our own instances :-)

[jloisel]

Thanks Eduardo. Guess it's best we host our own instances :-)

Well, hosting your own comments server pretty much cancels the benefits of a statically generated website stored on Github / S3 + Cloudfront (or equivalent).

[InsOpDe]

@eduardoboucas Actually staticman does the throttling - with the help of ExpressBrute - you may add an option-object as the second parameter when creating a new instance of ExpressBrute in server.js:36 - see: https://github.com/AdamPflug/express-brute and if the threshold is reached, it fires the 429 response and lets you wait 15 minutes.

[casually-creative]

I'm a part of the same team as @unpreceden7ed mentioned. We're still receiving this issue day to day and it has been a major blocking issue for us. Our content people who use the tool report this issue very frequently (know that they do use the tool a lot though). I think that it wouldn't be a very unwise idea to implement some sort of queue at the side of staticman? When the github api returns an error, you simply wait untill you can send again, asuring that all requests in the queue succeed eventually. Time is not an issue for us, and I think for most other implementations either. So it doesn't really matter if it happens instantly, after 2 minutes or even after 10 minutes. As long as we can be quaranteed every single request goes through with a 100% success rate, that would be incredibly helpfull! Cheers!

[aivuk]

@MichielDP, see @InsOpDe answer. I had the same problem because of the ExpressBrute and solved removing its use from https://github.com/eduardoboucas/staticman/blob/master/server.js. Just remove this.initialiseBruteforceProtection() and all the four lines with this.bruteforce.prevent from the server.js and you will not receive the error "Too many requests in this time frame" anymore.

You can see the changes here:

aivuk@0fe57eb

[casually-creative]

@aivuk Thanks for your reply. Do you mind enlightening me about express brute? Not sure what it is or where it steps in exactly. We don't use express brute neither do we have a back-end nor do we host our own staticman instance..

[alexwaibel]

Duplicate of #317