[Tracing Instrumentation] Add instrumentation at deep search path

[Gaganjuneja]

Is your feature request related to a problem? Please describe.
Now the distributed tracing framework #7543 is available. We need to add the tracing instrumentation at several places to effectively monitoring the overall OpenSearch. Creating this task to add instrumentation at deep search path like coordinator, shard, segments, lucene calls, etc.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[macohen]

@Gaganjuneja thanks for opening this issue! can you please add more information to your request about what metrics you want to capture? What outcome are you looking for? What analytics do you want? What will you do with this data?

[Gaganjuneja]

@Gaganjuneja thanks for opening this issue! can you please add more information to your request about what metrics you want to capture? What outcome are you looking for? What analytics do you want? What will you do with this data?

Thanks @macohen for your reply, I have created a Meta task #8553 for instrumentations in OpenSearch code paths. I have updated the description. Contributor can add further details while picking it up for implementation.

[kkhatua]

One thing to note is that there is a much higher level tracing being done under #7334
This might appear as an overlap, but the granularity is the primary difference.