Add database CLI tools

[weiihann]

Description

This PR adds a new CLI tool for database-related operations (i.e. juno db [commands]). Under the db category, there are 2 subcommand:

1. juno db info - display the blockchain information stored in the database

{
  "network": "sepolia",
  "chain_height": 71958,
  "latest_block_hash": "0x69fa4f792a98cab20d5fe5040401a8377ae75234ee2446590e1b68d3abfa9e2",
  "latest_state_root": "0x7c5dd3a676f5b6f603227b1f1b77f8d95a96b05abc93c09f0715d27c7169f1c",
  "l1_height": 66367,
  "l1_block_hash": "0x335b4136b1da8317a1a72450ddb2df813ae828cb2ee4347d4c21d2d8adee0dc",
  "l1_state_root": "0x4ff66961f76f33cedc65124cbb3021e370cd966434c13245f1d75015e136ba8"
}

3. juno db size - display storage information for each bucket in the db

Calculating size of StateTrie, remaining buckets: 23
Calculating size of Peer, remaining buckets: 22
Calculating size of ContractClassHash, remaining buckets: 21
Calculating size of ContractStorage, remaining buckets: 20
Calculating size of Class, remaining buckets: 19
Calculating size of ContractNonce, remaining buckets: 18
Calculating size of ChainHeight, remaining buckets: 17
Calculating size of BlockHeaderNumbersByHash, remaining buckets: 16
Calculating size of BlockHeadersByNumber, remaining buckets: 15
Calculating size of TransactionBlockNumbersAndIndicesByHash, remaining buckets: 14
Calculating size of TransactionsByBlockNumberAndIndex, remaining buckets: 13
Calculating size of ReceiptsByBlockNumberAndIndex, remaining buckets: 12
Calculating size of StateUpdatesByBlockNumber, remaining buckets: 11
Calculating size of ClassesTrie, remaining buckets: 10
Calculating size of ContractStorageHistory, remaining buckets: 9
Calculating size of ContractNonceHistory, remaining buckets: 8
Calculating size of ContractClassHashHistory, remaining buckets: 7
Calculating size of ContractDeploymentHeight, remaining buckets: 6
Calculating size of L1Height, remaining buckets: 5
Calculating size of SchemaVersion, remaining buckets: 4
Calculating size of Pending, remaining buckets: 3
Calculating size of BlockCommitments, remaining buckets: 2
Calculating size of Temporary, remaining buckets: 1
Calculating size of SchemaIntermediateState, remaining buckets: 0
+-----------------------------------------+------------+----------+
|                 BUCKET                  |    SIZE    |  COUNT   |
+-----------------------------------------+------------+----------+
| StateTrie                               | 9.44 MiB   |   143032 |
| Peer                                    | 0.00 B     |        0 |
| ContractClassHash                       | 4.43 MiB   |    71516 |
| ContractStorage                         | 662.14 MiB |  5970298 |
| Class                                   | 5.80 GiB   |    11139 |
| ContractNonce                           | 4.43 MiB   |    71516 |
| ChainHeight                             | 9.00 B     |        1 |
| BlockHeaderNumbersByHash                | 2.88 MiB   |    73536 |
| BlockHeadersByNumber                    | 115.86 MiB |    73536 |
| TransactionBlockNumbersAndIndicesByHash | 91.42 MiB  |  1956435 |
| TransactionsByBlockNumberAndIndex       | 4.03 GiB   |  1956435 |
| ReceiptsByBlockNumberAndIndex           | 4.82 GiB   |  1956435 |
| StateUpdatesByBlockNumber               | 688.26 MiB |    73536 |
| ClassesTrie                             | 1.14 MiB   |    17370 |
| ContractStorageHistory                  | 848.86 MiB |  8477125 |
| ContractNonceHistory                    | 43.72 MiB  |   628061 |
| ContractClassHashHistory                | 57.53 KiB  |      807 |
| ContractDeploymentHeight                | 2.80 MiB   |    71516 |
| L1Height                                | 0.00 B     |        0 |
| SchemaVersion                           | 9.00 B     |        1 |
| Pending                                 | 1.56 MiB   |        1 |
| BlockCommitments                        | 8.56 MiB   |    73536 |
| Temporary                               | 0.00 B     |        0 |
| SchemaIntermediateState                 | 2.00 B     |        1 |
+-----------------------------------------+------------+----------+
|                  TOTAL                  | 17.08 GIB  | 21625833 |
+-----------------------------------------+------------+----------+
+-----------------+----------+----------+
|      STATE      |   SIZE   |  COUNT   |
+-----------------+----------+----------+
| Without history | 6.46 GiB |  6267501 |
| With history    | 7.33 GiB | 15373494 |
+-----------------+----------+----------+

[codecov]

Codecov Report

Attention: Patch coverage is 72.46377% with 38 lines in your changes missing coverage. Please review.

Project coverage is 75.59%. Comparing base (7205822) to head (c15d540).

Files	Patch %	Lines
cmd/juno/dbcmd.go	73.68%	17 Missing and 13 partials ⚠️
utils/size.go	45.45%	6 Missing ⚠️
db/pebble/db.go	83.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1902      +/-   ##
==========================================
+ Coverage   75.45%   75.59%   +0.14%     
==========================================
  Files         100      101       +1     
  Lines        8983     9064      +81     
==========================================
+ Hits         6778     6852      +74     
+ Misses       1602     1593       -9     
- Partials      603      619      +16     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kirugan]

Can we also show to which network db corresponds to? it would be great if there is some snapshot stored without identification

[weiihann]

Can we also show to which network db corresponds to? it would be great if there is some snapshot stored without identification

I think that's a good addition. From my understanding, the network ID isn't stored directly in the db. Let me figure out how to pull that information.

[kirugan]

Can we also show to which network db corresponds to? it would be great if there is some snapshot stored without identification

I think that's a good addition. From my understanding, the network ID isn't stored directly in the db. Let me figure out how to pull that information.

That's right you can modify existing approach for your case:

// Verify that cfg.Network is compatible with the database.
	head, err := chain.Head()
	if err != nil && !errors.Is(err, db.ErrKeyNotFound) {
		return nil, fmt.Errorf("get head block from database: %v", err)
	}
	if head != nil {
		// We assume that there is at least one transaction in the block or that it is a pre-0.7 block.
		if _, err = core.VerifyBlockHash(head, &cfg.Network); err != nil {
			return nil, errors.New("unable to verify latest block hash; are the database and --network option compatible?")
		}
	}

You should iterate over networks to figure out (_, err := core.VerifyBlockHash(...); err == nil) to which one it corresponds

[weiihann]

You should iterate over networks to figure out (_, err := core.VerifyBlockHash(...); err == nil) to which one it corresponds

Added the network field.

[weiihann]

TODO for myself: do not find the target directory using working directory, binary will not be able to run anywhere. Perhaps use embed directive.

[weiihann]

Calculating DB size has some differences between this implementation and #1773:

1. Formats the data size into metric prefixes (i.e. MiB, GiB, etc)
2. Adds count of each KV pair
3. Outputs result in table format
4. Put this subcommand under the subcommand of db ./juno db size instead of ./juno db-size

[weiihann]

Tests fail because #1978 has to be merged.