-
Notifications
You must be signed in to change notification settings - Fork 765
PoET not accepting any requests after getting to many requests error 429 #2384
Comments
Hi- There is a problem with PoET, after we reach 10 concurrent users, it stops working and it says too many requests error. This usually happens after 5 minutes of working. This might be due to the backpressure module in the sawtooth. But the point is that we have the same backpressure for both RAFT and PBFT. We have only the issue when we use PoET consensus. So generally, PoET is useless because we keep getting too many request errors and the system crashes. But for both PBFT and Raft and it keeps working. |
Hi @agunde406 @vaporos and @rbuysse , I would need your help as I am going to use Hyperledger Sawtooth for a big project for our company. We have the issue with PoET but neither for PBFT nor Raft. After 5 to 10 min maximum, when the users start sending more requests and sending transactions. The system crashes for PoET with the error of 429 error. The issue is the whole system crashes. We have the error with both PBFT and raft but after the users reduce the error will go. But for PoET, it never continues working. Can you please check this bug with the PoET? In this case we have to ignore using of PoET and stick with either Raft or PBFT but we prefer to use PoET as it is a large scale project in Canada. I appreciate your kind consideration on this matter and looking forward hearing form you |
What is the transaction rate that you are submitting against your validators? Are you spreading them across the network or firing them at a single node? How many blocks deep are you when this occurs? |
Thank you in advance @peterschwarz @agunde406 @vaporos and @rbuysse
|
Dear @arsulegai, could you please check this bug above? We have no problem with either Raft or PBFT in Hyperledger Sawtooth. But we have issue that after 5-10 mins when we start sending transactions, with 5 nodes, the system starts crashes with the error 429 too many requests. Do you have any suggestions? I assume that you work with this repository and you might have experienced some idea to cope with such a problem. Thanks for your kind consideration. |
@rowaisi it was long ago, but sure. I need logs from the PoET engine and the Validator to analyze in detail. Following is a possibility, only logs can confirm The tag I observed one such behavior as you are facing, it was long back and hyperledger-archives/sawtooth-poet@754766d was the fix for that issue. |
Hello @arsulegai @peterschwarz @agunde406 @vaporos and @rbuysse, this is the logs of sawtooth-validator-default-0 container when I got the error, same thing for the other validators
|
Dear Members: @arsulegai @vaporos @peterschwarz @buysse and @isabeltomb, We are very interested in to use Hypereldger Sawtooth for Canadian SupplyChain project and the issue with PoET is stopping us. I will be glad if someone can run the image of PoET and explore the issue. This backpressure causes PoET to crash and it does not work after we start using it with more users in 5 - 10 min max. This issue does not happen on both Raft and PBFT in Sawtooth. We would be very much appreciate if somebody can take a look on logs and images. Your consideration is highly appreciated. |
Hello, I was not able to replicate this issue locally. The commit that @arsulegai mentioned above has not been released but should be available in the nightly build. I would suggest the following:
Any other errors that show up would also be helpful in figuring out what may be wrong, especially from the poet-engine. For example when you say crash is there an error message? |
Dear @vaporos, could you please help us regarding the specified issue. Please kindly look at the images that we use to run our Sawtooth network with PoET and after 5-10 of running network and sending transactions, the system is saturated with too many requests errors and even by reducing the transactions it wont be recovered. This issue does not exist in both PBFT and Raft. I hope that you can guide us and let us know an option to see how can we surpass this issue. |
1. Issue
I am running a heavy workload on Sawtooth network for test purpose. When I run the network using PBFT or Raft consensus I have Too many requests error but the network continue to accept requests. However using PoET the network stop accepting any request after I got 429 error.
2. System information
This is the validator configuration for PoET
I am running the sawtooth network on a AWS VM, with 8GO of RAM and 2CPU running ubuntu 18.04
2. Question
Any idea how to solve this issue?
Is there a way to disable back pressure test ?
The text was updated successfully, but these errors were encountered: