Implement new blocks subscriber and submit data to kafka #34
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…turn (break the loop)
…ock) to ChainEvent (emit for every block)
…actionHash in synced logs
…ong condition in entrypoint.sh
ducthotran2010
requested changes
Nov 8, 2021
ducthotran2010
approved these changes
Nov 9, 2021
There was a problem hiding this comment.
Node logs from this PR, please delete unnecessary logs:
Forcing update chain config.
Using account 8209b2f9f5ea2c09a75ba90e8206385488849470
checking subscriber= kafka_url=
dump: 8209b2f9f5ea2c09a75ba90e8206385488849470 enode://a166ab6437cf370bc604097529a0fb6a8a4836bb85833fbf588b130cb73fe0517940d10c5d89c0e3e1c2800a774ac1ae2108d62cb4608556e41bc1fc4482241a@10.3.0.18:30303
params: --rpc.allow-unprotected-txs --http.api eth,web3,personal,txpool --networkid 2020 --unlock 8209b2f9f5ea2c09a75ba90e8206385488849470 --bootnodes enode://a166ab6437cf370bc604097529a0fb6a8a4836bb85833fbf588b130cb73fe0517940d10c5d89c0e3e1c2800a774ac1ae2108d62cb4608556e41bc1fc4482241a@10.3.0.18:30303 --ethstats vm-rpc-skymavis-8:xQj2MZPaN6@stats.roninchain.com:443 --miner.gasprice 0 --keystore /ronin/keystore --password /ronin/keystore/password --mine
+ echo 'params: --rpc.allow-unprotected-txs --http.api eth,web3,personal,txpool --networkid 2020 --unlock 8209b2f9f5ea2c09a75ba90e8206385488849470 --bootnodes enode://a166ab6437cf370bc604097529a0fb6a8a4836bb85833fbf588b130cb73fe0517940d10c5d89c0e3e1c2800a774ac1ae2108d62cb4608556e41bc1fc4482241a@10.3.0.18:30303 --ethstats vm-rpc-skymavis-8:xQj2MZPaN6@stats.roninchain.com:443 --miner.gasprice 0 --keystore /ronin/keystore --password /ronin/keystore/password --mine'
DNK90
added a commit
that referenced
this pull request
Dec 10, 2021
* Implement new blocks subscriber and submit data to kafka * Fix job is added to job queue continuously * Add SCRAM-SHA-256 authentication type and update description of authentication flag * Continue the loop in worker when job reaches its maxTry instead of return (break the loop) * Change subscribe function from ChainHeadEvent (only emit canonical block) to ChainEvent (emit for every block) * Add missing queueSize flag * Apply sending batch to optimize memory and fix deadlock when exiting * Add handle reorg event * Send message to ChainEvent after finish syncing block using snap or fast * Add Remove and Rebirth logs event, Fix empty blockHash, Number, TransactionHash in synced logs * Implement get and publish past blocks in state * Add subscribe removedLogEvent and rebirthLogsEvent codes * Update Dockerfile and command usage * Add contractAddress to transaction * Add checkConnection function to check connectivity to kafka, Fixed wrong condition in entrypoint.sh * Correct docker-compose volumes path * Correct mine condition * Add number of transactions to block * Add CumulativeGasUsed to transaction * Add Verbosity variable to docker-compose * Add comments for new ideas of resetting blockchain's state if bad block happen * Add fromHeight validation for reOrg, rebirthLog and removeLog * Add RONIN_PARAMS to docker-compose * Update sending confirmed block which behind 10 blocks with current block * Handle nil confirmed block * Add write timeout and log while publishing error * Update the latest go-kafka * Fix typo, send wrong topic in confirmBlockTopic * Update flags default value description * Enable miner by default in Dockerfile and entrypoint * Remove unused commented code * Add publishedTime to newBlock/newTx/newLogs to specify the exact time event is sent from blockchain * Update entrypoint that allows users custom their datadir * Remove unnecessary comments in entrypoint.sh
DNK90
added a commit
that referenced
this pull request
Dec 10, 2021
* Implement new blocks subscriber and submit data to kafka * Fix job is added to job queue continuously * Add SCRAM-SHA-256 authentication type and update description of authentication flag * Continue the loop in worker when job reaches its maxTry instead of return (break the loop) * Change subscribe function from ChainHeadEvent (only emit canonical block) to ChainEvent (emit for every block) * Add missing queueSize flag * Apply sending batch to optimize memory and fix deadlock when exiting * Add handle reorg event * Send message to ChainEvent after finish syncing block using snap or fast * Add Remove and Rebirth logs event, Fix empty blockHash, Number, TransactionHash in synced logs * Implement get and publish past blocks in state * Add subscribe removedLogEvent and rebirthLogsEvent codes * Update Dockerfile and command usage * Add contractAddress to transaction * Add checkConnection function to check connectivity to kafka, Fixed wrong condition in entrypoint.sh * Correct docker-compose volumes path * Correct mine condition * Add number of transactions to block * Add CumulativeGasUsed to transaction * Add Verbosity variable to docker-compose * Add comments for new ideas of resetting blockchain's state if bad block happen * Add fromHeight validation for reOrg, rebirthLog and removeLog * Add RONIN_PARAMS to docker-compose * Update sending confirmed block which behind 10 blocks with current block * Handle nil confirmed block * Add write timeout and log while publishing error * Update the latest go-kafka * Fix typo, send wrong topic in confirmBlockTopic * Update flags default value description * Enable miner by default in Dockerfile and entrypoint * Remove unused commented code * Add publishedTime to newBlock/newTx/newLogs to specify the exact time event is sent from blockchain * Update entrypoint that allows users custom their datadir * Remove unnecessary comments in entrypoint.sh
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.