[Remote State] Create a common interface for reader/writer in PersistedClusterStateService and RemoteClusterStateService

[soosinha]

Is your feature request related to a problem? Please describe.
PersistedClusterStateService contains method to read and write cluster state to Lucene. RemoteClusterStateService contains method to read and write cluster state to remote blob store. We should refactor these to have a common read/write interface so that it will be easier to switch from local store to remote store in the future.

[msfroh]

I was discussing something along these lines with @owaiskazi19 a couple of days ago.

Specifically, we were talking about storing config in indexes (like many of our plugins do) versus storing config in cluster state (as search + ingest pipelines do).

In my opinion, we could do better than treating this a cluster state problem and instead think of it as a "config store" problem, with a config store interface. Cluster state is one such config store (managed by cluster managers and replicated to all nodes). An index (either managed by the data nodes or in a remote store) could be another config store. Those would be two implementations that would make the most sense to ship with the OpenSearch min distribution. (At that point, plugins that move their config to the new interface could offer users the choice of storing config in an index or cluster state.)

Other config store implementations could be provided via plugins: Apache Cassandra, Zookeeper, DynamoDB, files on an NFS-mount, etc. All could provide trade-offs in terms of update propagation latency, setup / maintenance cost, scalability, etc.