Set Odysseus hardfork for mainnet #50

ducthotran2010 · 2022-01-13T10:44:28Z

No description provided.

* ethdb: use pebble Co-authored-by: Gary Rong <garyrong0905@gmail.com> foo update * apply suggested changes * flags: go format node: fix ddir lookup mistake accounts/abi/bind: fix go.mod replacement for generated binding deps: update pebble + with fix 32-bit build * ethdb/pebble: respect max memtable size * core/rawdb, ethdb: enable pebble on non-32bit platforms only * core/rawdb: fix build tags, fix some review concerns * core/rawdb: refactor methods for database opening * core/rawdb: remove erroneous build tag * cmd/geth: fix the flag default handling + testcase * cmd/geth: improve testing regarding custom backends * ethdb/pebble, deps: update pebble dependency * core/rawdb: replace method with Open * ethdb/pebble: several updates for pebble (axieinfinity#49) * ethdb/pebble: fix size count in batch * ethdb/pebble: disable seek compaction * ethdb/pebble: more fixes * ethdb, core, cmd: polish and fixes (axieinfinity#50) * cmd/utils, core/rawdb, ethdb/pebble: address some review concerns * Update flags.go * ethdb/pebble: minor refactors * ethdb/pebble: avoid copy on batch replay * ethdb: fix compilation flaw * cmd: fix test fail due to mismatching error message * cmd/geth, node: rename backingdb to db.engine --------- Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com> Co-authored-by: rjl493456442 <garyrong0905@gmail.com> Co-authored-by: Péter Szilágyi <peterke@gmail.com>

* core, ethdb, tests, trie: implement NewBatchWithSize API for batcher (#24392) This PR adds an addtional API called `NewBatchWithSize` for db batcher. It turns out that leveldb batch memory allocation is super inefficient. The main reason is the allocation step of leveldb Batch is too small when the batch size is large. It can take a few second to build a leveldb batch with 100MB size. Luckily, leveldb also offers another API called MakeBatch which can pre-allocate the memory area. So if the approximate size of batch is known in advance, this API can be used in this case. It's needed in new state scheme PR which needs to commit a batch of trie nodes in a single batch. Implement the feature in a seperate PR. * core, ethdb, tests, trie: introduce database snapshot (#24486) * ethdb: pebble backend (64bit platforms only) (#26517) * ethdb: use pebble Co-authored-by: Gary Rong <garyrong0905@gmail.com> foo update * apply suggested changes * flags: go format node: fix ddir lookup mistake accounts/abi/bind: fix go.mod replacement for generated binding deps: update pebble + with fix 32-bit build * ethdb/pebble: respect max memtable size * core/rawdb, ethdb: enable pebble on non-32bit platforms only * core/rawdb: fix build tags, fix some review concerns * core/rawdb: refactor methods for database opening * core/rawdb: remove erroneous build tag * cmd/geth: fix the flag default handling + testcase * cmd/geth: improve testing regarding custom backends * ethdb/pebble, deps: update pebble dependency * core/rawdb: replace method with Open * ethdb/pebble: several updates for pebble (#49) * ethdb/pebble: fix size count in batch * ethdb/pebble: disable seek compaction * ethdb/pebble: more fixes * ethdb, core, cmd: polish and fixes (#50) * cmd/utils, core/rawdb, ethdb/pebble: address some review concerns * Update flags.go * ethdb/pebble: minor refactors * ethdb/pebble: avoid copy on batch replay * ethdb: fix compilation flaw * cmd: fix test fail due to mismatching error message * cmd/geth, node: rename backingdb to db.engine --------- Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com> Co-authored-by: rjl493456442 <garyrong0905@gmail.com> Co-authored-by: Péter Szilágyi <peterke@gmail.com> * ethdb/pebble: fix nil callbacks (#26650) * ethdb: add benchmark test suite (#26659) * ethdb/pebble: Fix `MemTableStopWritesThreshold` (#26692) MemTableStopWritesThreshold was set to the max size of all memtables before blocking writing but should be set to the max number of memtables. This is documented [here](https://github.com/cockroachdb/pebble/blob/master/options.go#L738-L742). * ethdb/pebble: fix range compaction (#26771) * ethdb/pebble: fix range compaction * ethdb/pebble: add comment * ethdb/pebble: fix max memorytable size (#26776) * core/rawdb, ethdb/pebble: disable pebble on openbsd (#26801) * ethdb/pebble: use atomic type (#27014) * cmd/utils, node: switch to Pebble as the default db if none exists (#27136) * cmd/utils, node: switch to Pebble as the default db if none exists * node: fall back to LevelDB on platforms not supporting Pebble * core/rawdb, node: default to Pebble at the node level * cmd/geth: fix some tests explicitly using leveldb * ethdb/pebble: allow double closes, makes tests simpler * ethdb/pebble: prevent shutdown-panic (#27238) One difference between pebble and leveldb is that the latter returns error when performing Get on a closed database, the former does a panic. This may be triggered during shutdown (see #27237) This PR changes the pebble driver so we check that the db is not closed already, for several operations. It also adds tests to the db test-suite, so the previously implicit assumption of "not panic:ing at ops on closed database" is covered by tests. * ethdb/pebble: fix NewBatchWithSize to set db (#27350) * ethdb/pebble: fsync for batch writes (#27522) This is likely the culprit behind several data corruption issues, e.g. where data has been written to the freezer, but the deletion from pebble does not go through due to process crash. * ethdb/pebble: use sync mode for pebble writes (#27615) * core/rawdb, ethdb/pebble: avoid fsync db in tests (#27836) Adds an option to disable fsync for database operations. This is to make tests faster. * ethdb, internal/ethapi: support exposing Pebble stats too, beside LevelDB (#28224) ethdb, internal/ethapi: support exposing Pebble stats too, besinde LevelDB * ethdb/pebble: upgrade pebble to master (aa077af62593) (#28070) * ethdb/pebble: upgrade pebble * ethdb/pebble, go.mod: update pebble to master (aa077af62593) --------- Co-authored-by: Péter Szilágyi <peterke@gmail.com> * ethdb/pebble: luv you linter * ethdb/pebble: add level file metrics (#28271) * cmd, core, ethdb: enable Pebble on 32 bits and OpenBSD too (#28335) * cmd, core, ethdb: enable Pebble on 32 bits and OpenBSD too * ethdb/pebble: use Pebble's internal constant calculation * ethdb/pebble: cap memory table size as maxMemTableSize-1 (#28444) * ethdb/pebble: add `Errorf` function to panicLogger (#28491) cockroachdb/pebble@422dce9 added Errorf to the Logger interface, this change makes it possible to compile geth with that version of pebble by adding the corresponding method to panicLogger. * ethdb/pebble: don't double-close iterator inside pebbleIterator (#28566) Adds 'released' flag to pebbleIterator to avoid double closing cockroachdb/pebble.Iterator as it is an invalid operation. Fixes #28565 * ethdb/pebble: remove a dependency (#28627) The dependency was not really used anyway, so we can get rid of it. Co-authored-by: Felix Lange <fjl@twurst.com> * cmd: add db.engine flags to Ronin commands This commit adds --db.engine to database related commands in Ronin to specify the engine to use between leveldb and pebble. * entrypoint: add DB_ENGINE environment variable --------- Co-authored-by: rjl493456442 <garyrong0905@gmail.com> Co-authored-by: Martin Holst Swende <martin@swende.se> Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com> Co-authored-by: Péter Szilágyi <peterke@gmail.com> Co-authored-by: Patrick O'Grady <prohb125@gmail.com> Co-authored-by: s7v7nislands <s7v7nislands@gmail.com> Co-authored-by: Jim McDonald <Jim@mcdee.net> Co-authored-by: Maciej Kulawik <10907694+magicxyyz@users.noreply.github.com> Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de> Co-authored-by: Felix Lange <fjl@twurst.com>

…ty#385) * core, ethdb, tests, trie: implement NewBatchWithSize API for batcher (#24392) This PR adds an addtional API called `NewBatchWithSize` for db batcher. It turns out that leveldb batch memory allocation is super inefficient. The main reason is the allocation step of leveldb Batch is too small when the batch size is large. It can take a few second to build a leveldb batch with 100MB size. Luckily, leveldb also offers another API called MakeBatch which can pre-allocate the memory area. So if the approximate size of batch is known in advance, this API can be used in this case. It's needed in new state scheme PR which needs to commit a batch of trie nodes in a single batch. Implement the feature in a seperate PR. * core, ethdb, tests, trie: introduce database snapshot (#24486) * ethdb: pebble backend (64bit platforms only) (#26517) * ethdb: use pebble Co-authored-by: Gary Rong <garyrong0905@gmail.com> foo update * apply suggested changes * flags: go format node: fix ddir lookup mistake accounts/abi/bind: fix go.mod replacement for generated binding deps: update pebble + with fix 32-bit build * ethdb/pebble: respect max memtable size * core/rawdb, ethdb: enable pebble on non-32bit platforms only * core/rawdb: fix build tags, fix some review concerns * core/rawdb: refactor methods for database opening * core/rawdb: remove erroneous build tag * cmd/geth: fix the flag default handling + testcase * cmd/geth: improve testing regarding custom backends * ethdb/pebble, deps: update pebble dependency * core/rawdb: replace method with Open * ethdb/pebble: several updates for pebble (axieinfinity#49) * ethdb/pebble: fix size count in batch * ethdb/pebble: disable seek compaction * ethdb/pebble: more fixes * ethdb, core, cmd: polish and fixes (axieinfinity#50) * cmd/utils, core/rawdb, ethdb/pebble: address some review concerns * Update flags.go * ethdb/pebble: minor refactors * ethdb/pebble: avoid copy on batch replay * ethdb: fix compilation flaw * cmd: fix test fail due to mismatching error message * cmd/geth, node: rename backingdb to db.engine --------- Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com> Co-authored-by: rjl493456442 <garyrong0905@gmail.com> Co-authored-by: Péter Szilágyi <peterke@gmail.com> * ethdb/pebble: fix nil callbacks (#26650) * ethdb: add benchmark test suite (#26659) * ethdb/pebble: Fix `MemTableStopWritesThreshold` (#26692) MemTableStopWritesThreshold was set to the max size of all memtables before blocking writing but should be set to the max number of memtables. This is documented [here](https://github.com/cockroachdb/pebble/blob/master/options.go#L738-L742). * ethdb/pebble: fix range compaction (#26771) * ethdb/pebble: fix range compaction * ethdb/pebble: add comment * ethdb/pebble: fix max memorytable size (#26776) * core/rawdb, ethdb/pebble: disable pebble on openbsd (#26801) * ethdb/pebble: use atomic type (#27014) * cmd/utils, node: switch to Pebble as the default db if none exists (#27136) * cmd/utils, node: switch to Pebble as the default db if none exists * node: fall back to LevelDB on platforms not supporting Pebble * core/rawdb, node: default to Pebble at the node level * cmd/geth: fix some tests explicitly using leveldb * ethdb/pebble: allow double closes, makes tests simpler * ethdb/pebble: prevent shutdown-panic (#27238) One difference between pebble and leveldb is that the latter returns error when performing Get on a closed database, the former does a panic. This may be triggered during shutdown (see #27237) This PR changes the pebble driver so we check that the db is not closed already, for several operations. It also adds tests to the db test-suite, so the previously implicit assumption of "not panic:ing at ops on closed database" is covered by tests. * ethdb/pebble: fix NewBatchWithSize to set db (#27350) * ethdb/pebble: fsync for batch writes (#27522) This is likely the culprit behind several data corruption issues, e.g. where data has been written to the freezer, but the deletion from pebble does not go through due to process crash. * ethdb/pebble: use sync mode for pebble writes (#27615) * core/rawdb, ethdb/pebble: avoid fsync db in tests (#27836) Adds an option to disable fsync for database operations. This is to make tests faster. * ethdb, internal/ethapi: support exposing Pebble stats too, beside LevelDB (#28224) ethdb, internal/ethapi: support exposing Pebble stats too, besinde LevelDB * ethdb/pebble: upgrade pebble to master (aa077af62593) (#28070) * ethdb/pebble: upgrade pebble * ethdb/pebble, go.mod: update pebble to master (aa077af62593) --------- Co-authored-by: Péter Szilágyi <peterke@gmail.com> * ethdb/pebble: luv you linter * ethdb/pebble: add level file metrics (#28271) * cmd, core, ethdb: enable Pebble on 32 bits and OpenBSD too (#28335) * cmd, core, ethdb: enable Pebble on 32 bits and OpenBSD too * ethdb/pebble: use Pebble's internal constant calculation * ethdb/pebble: cap memory table size as maxMemTableSize-1 (#28444) * ethdb/pebble: add `Errorf` function to panicLogger (#28491) cockroachdb/pebble@422dce9 added Errorf to the Logger interface, this change makes it possible to compile geth with that version of pebble by adding the corresponding method to panicLogger. * ethdb/pebble: don't double-close iterator inside pebbleIterator (#28566) Adds 'released' flag to pebbleIterator to avoid double closing cockroachdb/pebble.Iterator as it is an invalid operation. Fixes #28565 * ethdb/pebble: remove a dependency (#28627) The dependency was not really used anyway, so we can get rid of it. Co-authored-by: Felix Lange <fjl@twurst.com> * cmd: add db.engine flags to Ronin commands This commit adds --db.engine to database related commands in Ronin to specify the engine to use between leveldb and pebble. * entrypoint: add DB_ENGINE environment variable --------- Co-authored-by: rjl493456442 <garyrong0905@gmail.com> Co-authored-by: Martin Holst Swende <martin@swende.se> Co-authored-by: Jared Wasinger <j-wasinger@hotmail.com> Co-authored-by: Péter Szilágyi <peterke@gmail.com> Co-authored-by: Patrick O'Grady <prohb125@gmail.com> Co-authored-by: s7v7nislands <s7v7nislands@gmail.com> Co-authored-by: Jim McDonald <Jim@mcdee.net> Co-authored-by: Maciej Kulawik <10907694+magicxyyz@users.noreply.github.com> Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de> Co-authored-by: Felix Lange <fjl@twurst.com>

Set Odysseus hardfork for mainnet

41063a8

ducthotran2010 requested a review from DNK90 January 13, 2022 10:51

DNK90 approved these changes Jan 13, 2022

View reviewed changes

ducthotran2010 merged commit 763ee57 into axieinfinity:master Jan 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set Odysseus hardfork for mainnet #50

Set Odysseus hardfork for mainnet #50

ducthotran2010 commented Jan 13, 2022

Set Odysseus hardfork for mainnet #50

Set Odysseus hardfork for mainnet #50

Conversation

ducthotran2010 commented Jan 13, 2022