Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem: memiavl is not integrated with state machine #950

Merged
merged 37 commits into from
Apr 19, 2023

Conversation

yihuang
Copy link
Collaborator

@yihuang yihuang commented Apr 3, 2023

Closes: #938

Solution:

  • implement custom rootmulti store and kvstore wrappers

👮🏻👮🏻👮🏻 !!!! REFERENCE THE PROBLEM YOUR ARE SOLVING IN THE PR TITLE AND DESCRIBE YOUR SOLUTION HERE !!!! DO NOT FORGET !!!! 👮🏻👮🏻👮🏻

PR Checklist:

  • Have you read the CONTRIBUTING.md?
  • Does your PR follow the C4 patch requirements?
  • Have you rebased your work on top of the latest master?
  • Have you checked your code compiles? (make)
  • Have you included tests for any non-trivial functionality?
  • Have you checked your code passes the unit tests? (make test)
  • Have you checked your code formatting is correct? (go fmt)
  • Have you checked your basic code style is fine? (golangci-lint run)
  • If you added any dependencies, have you checked they do not contain any known vulnerabilities? (go list -json -m all | nancy sleuth)
  • If your changes affect the client infrastructure, have you run the integration test?
  • If your changes affect public APIs, does your PR follow the C4 evolution of public contracts?
  • If your code changes public APIs, have you incremented the crate version numbers and documented your changes in the CHANGELOG.md?
  • If you are contributing for the first time, please read the agreement in CONTRIBUTING.md now and add a comment to this pull request stating that your PR is in accordance with the Developer's Certificate of Origin.

Thank you for your code, it's appreciated! :)

store/rootmulti/store.go Fixed Show fixed Hide fixed
store/rootmulti/store.go Fixed Show fixed Hide fixed
store/rootmulti/store.go Fixed Show fixed Hide fixed
store/rootmulti/store.go Fixed Show fixed Hide fixed
store/rootmulti/store.go Fixed Show fixed Hide fixed
memiavl/tree.go Fixed Show fixed Hide fixed
store/rootmulti/store.go Fixed Show fixed Hide fixed
store/rootmulti/store.go Fixed Show fixed Hide fixed
store/rootmulti/store.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
Comment on lines +117 to +125
for k, v := range rs.stores {
store := types.KVStore(v)
// Wire the listenkv.Store to allow listeners to observe the writes from the cache store,
// set same listeners on cache store will observe duplicated writes.
if rs.ListeningEnabled(k) {
store = listenkv.NewStore(store, k, rs.listeners[k])
}
stores[k] = store
}

Check failure

Code scanning / gosec

the value in the range statement should be _ unless copying a map: want: for key := range m

expected exactly 1 statement (either append, delete, or copying to another map) in a range with a map, got 3
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/changeset.go Fixed Show fixed Hide fixed
memiavl/tree.go Fixed Show fixed Hide fixed
Comment on lines +117 to +125
for k, v := range rs.stores {
store := types.KVStore(v)
// Wire the listenkv.Store to allow listeners to observe the writes from the cache store,
// set same listeners on cache store will observe duplicated writes.
if rs.ListeningEnabled(k) {
store = listenkv.NewStore(store, k, rs.listeners[k])
}
stores[k] = store
}

Check warning

Code scanning / CodeQL

Iteration over map

Iteration over map may be a possible source of non-determinism
Solution:
- implement custom rootmulti store and kvstore wrappers

add flag --memiavl to enable memiavl

kvstore use change set api
store/rootmulti/store.go Fixed Show fixed Hide fixed
memiavl/db.go Show resolved Hide resolved
Comment on lines 62 to 79
for key, store := range rs.stores {
if memiavlStore, ok := store.(*memiavlstore.Store); ok {
changeSets = append(changeSets, &memiavl.NamedChangeSet{
Name: key.Name(),
Changeset: memiavlStore.PopChangeSet(),
})
} else {
commitID := store.Commit()
// to keep the root hash compatible with cosmos-sdk 0.46
if store.GetStoreType() != types.StoreTypeTransient {
si := types.StoreInfo{
Name: key.Name(),
CommitId: commitID,
}
extraStoreInfos = append(extraStoreInfos, si)
}
}
}

Check failure

Code scanning / gosec

the value in the range statement should be _ unless copying a map: want: for key := range m

the value in the range statement should be _ unless copying a map: want: for key := range m
memiavl/export.go Show resolved Hide resolved
memiavl/db.go Show resolved Hide resolved
memiavl/import.go Outdated Show resolved Hide resolved
Co-authored-by: mmsqe <tqd0800210105@gmail.com>
Signed-off-by: yihuang <huang@crypto.com>
memiavl/import.go Fixed Show fixed Hide fixed
memiavl/import.go Fixed Show fixed Hide fixed
if importer != nil {
importer.Close()
}
importer = NewTreeImporter(filepath.Join(dir, snapshotDir, item.Store.Name), int64(height))

Check failure

Code scanning / gosec

Potential integer overflow by integer type conversion

Potential integer overflow by integer type conversion
if height > math.MaxUint32 {
return snapshottypes.SnapshotItem{}, fmt.Errorf("version overflows uint32: %d", height)
}
snapshotDir := snapshotName(uint32(height))

Check failure

Code scanning / gosec

Potential integer overflow by integer type conversion

Potential integer overflow by integer type conversion
memiavl/import.go Fixed Show fixed Hide fixed
return fmt.Errorf("height overflows uint32: %d", height)
}

mtree, err := LoadMultiTree(snapshotPath(db.dir, uint32(height)))

Check failure

Code scanning / gosec

Potential integer overflow by integer type conversion

Potential integer overflow by integer type conversion
@yihuang yihuang force-pushed the multistore branch 3 times, most recently from 1918e48 to 1b274f7 Compare April 19, 2023 09:24
@codecov
Copy link

codecov bot commented Apr 19, 2023

Codecov Report

Merging #950 (2cd0a41) into main (454cfe2) will increase coverage by 22.10%.
The diff coverage is 15.56%.

Impacted file tree graph

@@             Coverage Diff             @@
##             main     #950       +/-   ##
===========================================
+ Coverage   21.57%   43.67%   +22.10%     
===========================================
  Files          48       80       +32     
  Lines        3004     6537     +3533     
===========================================
+ Hits          648     2855     +2207     
- Misses       2293     3379     +1086     
- Partials       63      303      +240     
Impacted Files Coverage Δ
memiavl/export.go 31.57% <0.00%> (ø)
memiavl/proof.go 90.51% <ø> (ø)
memiavl/tree.go 80.20% <0.00%> (ø)
memiavl/import.go 31.42% <1.23%> (ø)
memiavl/multitree.go 71.16% <50.00%> (ø)
memiavl/db.go 55.61% <58.62%> (ø)

... and 41 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Problem: memiavl is not integrated into rootmulti store APIs
3 participants