[BUG] [Dev Setup] Assertion failure while creating index with remote_store.enabled=true

[sachinpkale]

Describe the bug

While creating an index with remote store enabled setting in developer setup, following exception is thrown.

WARNUNG: Uncaught exception in thread: Thread[opensearch[node_s1][clusterApplierService#updateTask][T#1],5,TGRP-RemoteStoreIT]
java.lang.AssertionError: Expected current thread [Thread[opensearch[node_s1][clusterApplierService#updateTask][T#1],5,TGRP-RemoteStoreIT]] to be the snapshot or generic thread.
	at __randomizedtesting.SeedInfo.seed([7CFDAEA59544BF28]:0)
	at org.opensearch.repositories.blobstore.BlobStoreRepository.assertSnapshotOrGenericThread(BlobStoreRepository.java:1586)
	at org.opensearch.repositories.blobstore.BlobStoreRepository.blobStore(BlobStoreRepository.java:648)
	at org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory.createRemoteDirectory(RemoteSegmentStoreDirectoryFactory.java:58)
	at org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory.newDirectory(RemoteSegmentStoreDirectoryFactory.java:47)
	at org.opensearch.index.IndexService.createShard(IndexService.java:513)
	at org.opensearch.indices.IndicesService.createShard(IndicesService.java:866)
	at org.opensearch.indices.IndicesService.createShard(IndicesService.java:200)
	at org.opensearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:651)
	at org.opensearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:628)
	at org.opensearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:285)
	at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:606)
	at org.opensearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:593)
	at org.opensearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:561)
	at org.opensearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:484)
	at org.opensearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:186)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747)
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282)
	at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

[andrross]

This has come up on the searchable snapshot work. In digging through the commit history, I believe this assertion was added in a change that allowed lazily loading a repository, such as you could add a repository before cloud resources existed (e.g. an S3 bucket) and it wouldn't fail until you actually needed to interact with the remote repository. As we add more features that interact with repositories this assertion may end up being too strict. However, there probably are times where we don't want to do synchronous calls to remote resources (such as cluster state updates, which looks like it is the case here, though in this case it is just creating the BlobStore object and not doing anything heavy weight with remote resources). I'm interested in what others think here.

[sachinpkale]

Started working on this bug to fix and agree with @andrross 's comment. BlobStoreRepository.blobStore() does not interact with the configured remote store.

[sachinpkale]

@andrross Please review: #5918

[sachinpkale]

Fixed in #5918