Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: State Sync from External Storage #8789

Merged
merged 89 commits into from
Apr 14, 2023

Commits on Apr 13, 2023

  1. feat: Dump state to S3

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    0039c20 View commit details
    Browse the repository at this point in the history
  2. fix

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    e9cb32c View commit details
    Browse the repository at this point in the history
  3. Metrics

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    95ff6dc View commit details
    Browse the repository at this point in the history
  4. Metrics

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    c3a0112 View commit details
    Browse the repository at this point in the history
  5. Metrics

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    85a8ded View commit details
    Browse the repository at this point in the history
  6. Metrics

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    4f0d69c View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    556248f View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    b15dd00 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    3e17aba View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    8105005 View commit details
    Browse the repository at this point in the history
  11. Syncing works

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    cb2731e View commit details
    Browse the repository at this point in the history
  12. Metrics

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    98cd7cf View commit details
    Browse the repository at this point in the history
  13. Fix

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    712ca55 View commit details
    Browse the repository at this point in the history
  14. Fields are not optional

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    bc953ea View commit details
    Browse the repository at this point in the history
  15. .

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    61671f5 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    0667d8b View commit details
    Browse the repository at this point in the history
  17. Too much output

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    0063fa8 View commit details
    Browse the repository at this point in the history
  18. Timeouts

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    f9b7f24 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    2195391 View commit details
    Browse the repository at this point in the history
  20. Fix DB corruption

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    8b1f311 View commit details
    Browse the repository at this point in the history
  21. feat: use split store in the view client (near#8656)

    - use split store in the view client - when configured
    - removed the get_store(Temperature) method and replaced usages with safer get_cold_store and get_hot_store
    - added SplitStorageConfig and used it to configure a few things
    wacban authored and nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    b224412 View commit details
    Browse the repository at this point in the history
  22. feat: Dump state of every epoch to S3 (near#8661)

    * Start a thread per shard to do the dumping
    * AWS credentials are provided as environment variables: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
    * In `config.json` specify both `config.state_sync.s3_bucket` and `config.state_sync.s3_region` to enable the new behavior.
    * No changes to the behavior of the node if those options are not enabled in `config.json`.
    * State is persisted to RocksDB such that restarts of the node are well handled.
    * Some useful metrics are exported.
    * The node assumes it's the only node in the this and all alternative universes that does the dumping.
    * * Unclear how to use multiple nodes to complete the dump faster
    * TODO: Speed this up by doing things in parallel: obtain parts, upload parts, set tags
    * * Do we even need tags?
    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    5846e1a View commit details
    Browse the repository at this point in the history
  23. feat: Dump state to S3

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    4cb5139 View commit details
    Browse the repository at this point in the history
  24. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    afcce1b View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    8408ce7 View commit details
    Browse the repository at this point in the history
  26. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    c61ff2a View commit details
    Browse the repository at this point in the history
  27. Print state_root

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    b682424 View commit details
    Browse the repository at this point in the history
  28. Print state_root

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    4f43f8c View commit details
    Browse the repository at this point in the history
  29. Changelog

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    161a357 View commit details
    Browse the repository at this point in the history
  30. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    020d7f9 View commit details
    Browse the repository at this point in the history
  31. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    4910fcc View commit details
    Browse the repository at this point in the history
  32. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    c2fa4a0 View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    14e202f View commit details
    Browse the repository at this point in the history
  34. More output

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    0da5f3b View commit details
    Browse the repository at this point in the history
  35. More output

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    610e820 View commit details
    Browse the repository at this point in the history
  36. Configuration menu
    Copy the full SHA
    296dda3 View commit details
    Browse the repository at this point in the history
  37. fmt

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    11e965a View commit details
    Browse the repository at this point in the history
  38. fmt

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    95c1ad5 View commit details
    Browse the repository at this point in the history
  39. fmt

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    dc89915 View commit details
    Browse the repository at this point in the history
  40. read-state-header

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    f14e8aa View commit details
    Browse the repository at this point in the history
  41. Configuration menu
    Copy the full SHA
    15d1fbb View commit details
    Browse the repository at this point in the history
  42. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    f390b19 View commit details
    Browse the repository at this point in the history
  43. Configuration menu
    Copy the full SHA
    89134c5 View commit details
    Browse the repository at this point in the history
  44. Tune verbosity

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    5a1acff View commit details
    Browse the repository at this point in the history
  45. Timing

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    4a9aa86 View commit details
    Browse the repository at this point in the history
  46. S3 downloads should now work.

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    56dc868 View commit details
    Browse the repository at this point in the history
  47. Metrics for part application.

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    74773e0 View commit details
    Browse the repository at this point in the history
  48. Metrics for part application.

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    81d511f View commit details
    Browse the repository at this point in the history
  49. Metrics for part application.

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    6db2638 View commit details
    Browse the repository at this point in the history
  50. More metrics for part download

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    51dbd55 View commit details
    Browse the repository at this point in the history
  51. Ready for review

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    6022323 View commit details
    Browse the repository at this point in the history
  52. Rename config field

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    cca8bdc View commit details
    Browse the repository at this point in the history
  53. Fix retry metric

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    b800bdc View commit details
    Browse the repository at this point in the history
  54. Minimize

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    1cd3a72 View commit details
    Browse the repository at this point in the history
  55. fix: apply log_summary_style to state sync phase message (near#8735)

    * Use colour for state sync phase display
    
    * fix
    
    * fix
    
    * debug verbosity
    
    * fix
    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    b319c51 View commit details
    Browse the repository at this point in the history
  56. [Runtime Epoch Split] (3/n) Add ability to get Arc<EpochManagerAdapte…

    …r> out of an &RuntimeWithEpochManagerAdapter. (near#8768)
    robin-near authored and nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    6b09e0d View commit details
    Browse the repository at this point in the history
  57. feat: Disable state sync by default because it's unreliable (near#8730)

    Add a config option `state_sync_enabled`. Default value is `false`.
    
    If the option is `true`, state sync works as usual, but prints this message every 10 seconds:
    ```
    Mar 15 07:31:11 nikurt-4 neard[48582]: 2023-03-15T07:31:11.352458Z  WARN stats: The node is syncing its State. The current implementation of this mechanism is known to be unreliable. It may never complete, or fail randomly and corrupt the DB.
    Mar 15 07:31:11 nikurt-4 neard[48582]: Suggestions:
    Mar 15 07:31:11 nikurt-4 neard[48582]: * Download a recent data snapshot and restart the node.
    Mar 15 07:31:11 nikurt-4 neard[48582]: * Disable state sync in the config. Add `"state_sync_enabled": false` to `config.json`.
    Mar 15 07:31:11 nikurt-4 neard[48582]: A better implementation of State Sync is work in progress.
    ```
    
    If the option is `false`, the node proceeds to download and apply blocks.
    Started a node from a month-old snapshot. After about 24 hours of running I see this, which confirms that the Block Sync is enabled and State Sync wasn't enabled:
    ```
    Mar 15 07:32:27 nikurt-3 neard[36289]: 2023-03-15T07:32:27.565700Z  INFO stats: #118440788 Downloading blocks 4.70% (1951625 left; at 118440788) 23 peers ⬇ 638 kB/s ⬆ 86.9 kB/s 2.00 bps 70.7 Tgas/s CPU: 62%, Mem: 4.27 GB
    ```
    
    Fix near#8719
    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    fdaa79e View commit details
    Browse the repository at this point in the history
  58. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    89297d4 View commit details
    Browse the repository at this point in the history
  59. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    699954f View commit details
    Browse the repository at this point in the history
  60. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    1244ac3 View commit details
    Browse the repository at this point in the history
  61. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    ad7ded8 View commit details
    Browse the repository at this point in the history
  62. Doc

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    6691bf7 View commit details
    Browse the repository at this point in the history
  63. Fix test

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    1cc6f05 View commit details
    Browse the repository at this point in the history
  64. Fix test

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    0ad29cd View commit details
    Browse the repository at this point in the history
  65. Move some code to near#8794

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    438e343 View commit details
    Browse the repository at this point in the history
  66. chore: update changelog with 1.32.1 and 1.32.2 (near#8869)

    Not entirely clear if we want minor releases tracked in changelog, do we? If so here is the PR for 1.32.1 and 1.32.2.
    wacban authored and nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    ea0b82a View commit details
    Browse the repository at this point in the history
  67. feat: override NetworkConfig from JSON config (near#8871)

    Added config.experimental.network_config_overrides field. It contains the overrides for the currently default values from NetworkConfig.
    
    The JSON config override is done before the CLI overrides.
    VanBarbascu authored and nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    e3baa8e View commit details
    Browse the repository at this point in the history
  68. feat: Dump state to S3

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    5a0a4fb View commit details
    Browse the repository at this point in the history
  69. Fix DB corruption

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    526225d View commit details
    Browse the repository at this point in the history
  70. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    f5d9545 View commit details
    Browse the repository at this point in the history
  71. Changelog

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    cc49285 View commit details
    Browse the repository at this point in the history
  72. More output

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    8067add View commit details
    Browse the repository at this point in the history
  73. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    4c53eac View commit details
    Browse the repository at this point in the history
  74. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    cd34b4a View commit details
    Browse the repository at this point in the history
  75. .

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    8f2c9b4 View commit details
    Browse the repository at this point in the history
  76. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    faa8a10 View commit details
    Browse the repository at this point in the history
  77. Configuration menu
    Copy the full SHA
    cfce87c View commit details
    Browse the repository at this point in the history
  78. merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    e71a0ea View commit details
    Browse the repository at this point in the history
  79. merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    b01050f View commit details
    Browse the repository at this point in the history
  80. Configuration menu
    Copy the full SHA
    b61749a View commit details
    Browse the repository at this point in the history
  81. .

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    594cafa View commit details
    Browse the repository at this point in the history
  82. Configuration menu
    Copy the full SHA
    a9debc5 View commit details
    Browse the repository at this point in the history
  83. Configuration menu
    Copy the full SHA
    00f70d4 View commit details
    Browse the repository at this point in the history
  84. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    0855002 View commit details
    Browse the repository at this point in the history
  85. Merge

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    b09200c View commit details
    Browse the repository at this point in the history
  86. Merge

    Ekleog-NEAR authored and nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    27423e5 View commit details
    Browse the repository at this point in the history
  87. re-push

    nikurt committed Apr 13, 2023
    Configuration menu
    Copy the full SHA
    2a7454f View commit details
    Browse the repository at this point in the history

Commits on Apr 14, 2023

  1. fix

    nikurt committed Apr 14, 2023
    Configuration menu
    Copy the full SHA
    b29d2b3 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6500c42 View commit details
    Browse the repository at this point in the history