Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make graph-node more robust in the face of shards being down or restarting #2815

Closed
wants to merge 27 commits into from
Closed

Make graph-node more robust in the face of shards being down or restarting #2815

wants to merge 27 commits into from

Commits on Oct 4, 2021

  1. store: Reconnect notification listener when database drops 

    Instead of crashing the process, try to reconnect indefinitely if we lose
the connection to the database.
    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    4008ef6 View commit details
    Browse the repository at this point in the history

  2. store: Don't block startup because notification listener can't connect

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    d874f7a View commit details
    Browse the repository at this point in the history

  3. store: Don't log connection details for 'Connection refused' errors

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    66fe268 View commit details
    Browse the repository at this point in the history

  4. store: Mirror chains and deployment_schemas in all shards 

    Manually insert/delete rows from the primary's deployment_schemas and
chains table. We can't use logical replication because we need to support
Postgres 9.6.

Add a job that refreshes the mirror every 5 minutes
    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    39de144 View commit details
    Browse the repository at this point in the history

  5. store: Allow reading from the primary and mirrored tables 

    Use that facility for the places where the BlockStore reads from the primary
    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    6354d69 View commit details
    Browse the repository at this point in the history

  6. store: Map and mirror subgraph_deployment_assignment to read assignments 

    We read the assignments for a node early during startup; with this change,
a node will start up even when the primary is down
    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    8b37a41 View commit details
    Browse the repository at this point in the history

  7. store: Mirror the 'subgraph' and 'subgraph_version' table

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    eb1814d View commit details
    Browse the repository at this point in the history

  8. store: Use mirrored tables in a lot of readonly SubgraphStore methods

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    40decfa View commit details
    Browse the repository at this point in the history

  9. store: Mark any failure in getting a connection as 'db unavailable'

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    ba6d765 View commit details
    Browse the repository at this point in the history

  10. store: Track db availability to fail fast when db is unavailable

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    0c87b8d View commit details
    Browse the repository at this point in the history

  11. store: Add a gauge for the size of the connection pool

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    c12ee0f View commit details
    Browse the repository at this point in the history

  12. store: Allow bypassing availability logic in the conn pool

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    7dc3cd2 View commit details
    Browse the repository at this point in the history

  13. core, graph, store: Asyncify store methods that returned DynTryFuture

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    bbec57e View commit details
    Browse the repository at this point in the history

  14. all: Return StoreError from store methods more consistently

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    633d047 View commit details
    Browse the repository at this point in the history

  15. store: Make DeploymentStore::transact_entity_operations take references 

    This is in preparation of retrying operations: if we pass owned data, we
need to clone before every attempt. Instead of forcing a clone in the
common case of the first try succeeding, work with references.

This also requires that we use Cow<Entity> because in rare cases we have to
modify the entity to add fulltext search fields; in the common case where
there is no fulltext search, we do not want to clone.
    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    4018b87 View commit details
    Browse the repository at this point in the history

  16. graph: Simple helper to sleep with exponential backoff

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    261f426 View commit details
    Browse the repository at this point in the history

  17. graph, store: Retry all WritableStore ops when database is not available

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    aba537d View commit details
    Browse the repository at this point in the history

  18. all: Pass a subgraph logger to WritableStore und log retry messages t… 

    …here
    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    2aa7381 View commit details
    Browse the repository at this point in the history

  19. store: Fail fast for an unavailable conn pool that's not ready yet, too

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    703973e View commit details
    Browse the repository at this point in the history

  20. all: Pass hashes to ChainStore.blocks by reference

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    ef60ab9 View commit details
    Browse the repository at this point in the history

  21. runtime: Only build test::common in test configs

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    c8a5510 View commit details
    Browse the repository at this point in the history

  22. core: Log more detail after starting all subgraphs

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    d86aa39 View commit details
    Browse the repository at this point in the history

  23. store: Do not treat failing to send store event as error 

    When we transact block operations, or revert a block, ignore errors when
sending store events. Since store events sent for these operations are only
used to update subscriptions, it is better to carry on than to fail the
operation and with that the subgraph.
    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    7e48747 View commit details
    Browse the repository at this point in the history

  24. store: Remove ConnectionPool.get_with_timeout_warning 

    This can block indefinitely and therefore lead to a lot of work on query
nodes queueing up; rather than that, operations should fail when we can not
get a connection.
    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    6b2cde8 View commit details
    Browse the repository at this point in the history

  25. store: Rename DeploymentStore.conn to pool

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    65ec9ea View commit details
    Browse the repository at this point in the history

  26. store: Remove import only needed for tests and not for release

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    6c0fe5d View commit details
    Browse the repository at this point in the history

  27. store: Small code/comment improvements

    @lutter
    lutter committed Oct 4, 2021
    Configuration menu
    Copy the full SHA
    393ce06 View commit details
    Browse the repository at this point in the history