[POC] Cross Cluster Replication as a core component

[ankitkala]

This issue documents the POC for Cross Cluster Replication in core.

Key Highlights:

• This implementation works only for SegRep & remote store backed indices.
• Contrary to existing CCR design, leader is connected with the follower via seed ndoes. So the control flow is essentially push where follower operations are event driven (instead of polling). Data flow is pull as follower will pull the data(and metadata) from remote store.
• This implementation relies on leader and follower indices pointing to same remote store repository. We need to further explore whether we should move to separate remote stores with data replicating between the remote buckets instead.

What's covered:
As a user:

• I'm able to start replication between 2 clusters on SegRep & remote store backed indices.
• I'm able to continuously replicate all the documents from leader to follower index.

What's not covered:

• Ability to add multiple follower.
• Persisting replication states(index and follower level) and metadata.
• Failover between primary & replica.
• Failure handing (e.g. node/cluster outages).
• Integration with security.

---

Data flow from leader to follower:

Start Replication flow:

• User invokes the API to start full cluster replication with follower aliases.
• Rest handler will start a persistent task SupervisorReplicationTask to monitor the overall replication process.
• Upon starting, SupervisorReplicationTask will bootstrap and
  • Create one FollowerReplicationTask per follower. This persistent task will monitor the states of follower cluster and react to events (like Follower cluster not responding, replication state: started/stopped/paused/resumed, etc)(not part of POC)
  • Create one IndexReplicationTask per leader index.
    • Upon starting, IndexReplicationTask will create the follower index with same settings & metadata as leader index.
    • This persistent task will track replication events for the index to be replicated(not part of POC). Events like index open/close/delete, mapping and setting updates, handling primary relocation/failover etc.

New index Creation:

• Follower recieves a request for creating the index. For CCR Follower indices, we
  • override and load the NRTReplicationEngine for follower's primary as well as replica.
  • override the remote store path to the leader index's remote store.
  • disable tranlog manager(NOOP) (not part of POC).
  • with follower's primary and replicas pointing to leader's remote, index recovery happens via remote store.

Data sync flow:

• RemoteStoreRefreshListener.afterRefresh uploads the segments & segmentsInfo snapshot to the remote store.
• After upload is complete, it invokes RemoteStoreSegmentUploadNotificationPublisher.notifySegmentUpload
• RemoteStoreSegmentUploadNotificationPublisher.notifySegmentUpload will:
  • SegmentReplicationCheckpointPublisher.publish for notifying replica.
  • NotifyCCRFollowersAction.publish -> leaderService.syncFollowerSegments invokes SyncFromLeaderAction on follower nodes containing primary and replica shards.
• SyncFromLeaderAction will sync the segments from remote store and update the IndexShard's reader.

Note: Notifying replica and followers can be be parallelized eventually.

[ankitkala]

POC link: Full cluster Replication using Segment Replication and Remote Store

[ankitkala]

@krishna-ggk @saikaranam-amazon @muralikpbhat @nknize @mch2 @Bukhtawar @sachinpkale @andrross

[anasalkouz]

Related Issue (RFC): #2872