Split the remote global metadata file to metadata attribute files #12190

shiv0408 · 2024-02-06T14:54:05Z

Description

We are now uploading the global metadata of a cluster state as a separate file for each metadata attribute like coordination metadata, settings, templates and all of the custom metadata attributes. Remote global state directory will look like below:

base folder/
    |
    |--> index/
    |     | --> index_UUID/
    |              | --> metadata__<inverted_index_metadata_version>__<inverted_codec_version>__<timestamp>.dat
    |              | --> metadata__<inverted_index_metadata_version>__<inverted_codec_version>__<timestamp>.dat  
    |
    |--> global-metadata/
    |       | --> coordination__<inverted_metadata_version>__<inverted_codec_version>__<timestamp>.dat
    |       | --> settings__<inverted_metadata_version>__<inverted_codec_version>__<timestamp>.dat
    |       | --> templates__<inverted_metadata_version>__<inverted_codec_version>__<timestamp>.dat
    |       | --> custom__<type>__<inverted_metadata_version>__<inverted_codec_version>__<timestamp>.dat
    |
    |
    |--> manifest/
    |       | --> manifest__<inverted_term>__<inverted_version>__<inverted_codec_version>__<timestamp>
    |       | --> manifest__<inverted_term>__<inverted_version>__<inverted_codec_version>__<timestamp>

Splitting the global-metadata into multiple files have improved the incremental metadata upload time to S3 by 50-70%, and full metadata upload by upto 5% because of parallel upload of global metadata attribute and index metadata files.
These benchmarks were done by writing a microbenchmark on main (shiv0408/OpenSearch@fe5fad8) and on top of PR branch (shiv0408/OpenSearch@88ab1ac)

Following are the benchmark results:

Benchmark on main

Benchmark                                                                            (indicesAliasesTemplates)  Mode  Cnt     Score    Error  Units
RemoteClusterStateBenchmark.measureFullMetadataUpload                                1000|     100|       100|  avgt   30    60.832 ±  0.642  ms/op
RemoteClusterStateBenchmark.measureFullMetadataUpload                               10000|    1000|      1000|  avgt   30   615.146 ±  1.765  ms/op
RemoteClusterStateBenchmark.measureFullMetadataUpload                               20000|    2000|      2000|  avgt   30  1227.299 ±  4.178  ms/op
RemoteClusterStateBenchmark.measureFullMetadataUpload                               50000|    5000|      5000|  avgt   30  3031.392 ± 19.117  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Coordination        1000|     100|       100|  avgt   30     2.440 ±  0.014  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Coordination       10000|    1000|      1000|  avgt   30    25.849 ±  0.105  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Coordination       20000|    2000|      2000|  avgt   30    52.243 ±  0.476  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Coordination       50000|    5000|      5000|  avgt   30   139.867 ±  1.062  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_IndexMetadata       1000|     100|       100|  avgt   30    32.722 ±  0.541  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_IndexMetadata      10000|    1000|      1000|  avgt   30   311.668 ±  2.694  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_IndexMetadata      20000|    2000|      2000|  avgt   30   622.160 ±  3.091  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_IndexMetadata      50000|    5000|      5000|  avgt   30  1578.523 ±  2.661  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Settings            1000|     100|       100|  avgt   30     2.470 ±  0.007  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Settings           10000|    1000|      1000|  avgt   30    26.391 ±  0.250  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Settings           20000|    2000|      2000|  avgt   30    53.320 ±  0.876  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Settings           50000|    5000|      5000|  avgt   30   144.819 ±  1.237  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Templates           1000|     100|       100|  avgt   30     2.814 ±  0.024  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Templates          10000|    1000|      1000|  avgt   30    29.080 ±  0.160  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Templates          20000|    2000|      2000|  avgt   30    60.032 ±  0.397  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Templates          50000|    5000|      5000|  avgt   30   155.970 ±  2.390  ms/op

Benchmark after splitting the global metadata

Benchmark                                                                            (indicesAliasesTemplates)  Mode  Cnt     Score    Error  Units
RemoteClusterStateBenchmark.measureFullMetadataUpload                                1000|     100|       100|  avgt   30    59.594 ±  0.323  ms/op
RemoteClusterStateBenchmark.measureFullMetadataUpload                               10000|    1000|      1000|  avgt   30   599.334 ±  2.941  ms/op
RemoteClusterStateBenchmark.measureFullMetadataUpload                               20000|    2000|      2000|  avgt   30  1198.450 ±  5.466  ms/op
RemoteClusterStateBenchmark.measureFullMetadataUpload                               50000|    5000|      5000|  avgt   30  2990.730 ± 15.318  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Coordination        1000|     100|       100|  avgt   30     0.800 ±  0.019  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Coordination       10000|    1000|      1000|  avgt   30     8.483 ±  0.059  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Coordination       20000|    2000|      2000|  avgt   30    17.231 ±  0.271  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Coordination       50000|    5000|      5000|  avgt   30    65.734 ±  1.375  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_IndexMetadata       1000|     100|       100|  avgt   30    31.890 ±  0.295  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_IndexMetadata      10000|    1000|      1000|  avgt   30   304.154 ±  0.994  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_IndexMetadata      20000|    2000|      2000|  avgt   30   606.649 ±  1.042  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_IndexMetadata      50000|    5000|      5000|  avgt   30  1530.920 ± 14.235  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Settings            1000|     100|       100|  avgt   30     0.832 ±  0.008  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Settings           10000|    1000|      1000|  avgt   30     8.253 ±  0.226  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Settings           20000|    2000|      2000|  avgt   30    20.208 ±  0.280  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Settings           50000|    5000|      5000|  avgt   30    65.269 ±  0.439  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Templates           1000|     100|       100|  avgt   30     1.166 ±  0.005  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Templates          10000|    1000|      1000|  avgt   30    12.657 ±  0.245  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Templates          20000|    2000|      2000|  avgt   30    26.883 ±  0.419  ms/op
RemoteClusterStateBenchmark.measureIncrementalClusterStateUpdate_Templates          50000|    5000|      5000|  avgt   30    88.283 ±  0.754  ms/op

Related Issues

Resolves #12468
Resolves #10645

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…bute files Signed-off-by: Shivansh Arora <hishiv@amazon.com>

github-actions · 2024-02-06T14:58:01Z

Compatibility status:

Checks if related components are compatible with change 8244c6d

Incompatible components

Skipped components

Compatible components

github-actions · 2024-02-06T15:04:51Z

❌ Gradle check result for 6bf7bc9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/cluster/metadata/Metadata.java

github-actions · 2024-02-15T12:09:34Z

❌ Gradle check result for d0875f9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-02-19T05:32:46Z

❌ Gradle check result for f6a2431: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

github-actions · 2024-02-20T09:40:44Z

❌ Gradle check result for f3e853b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-02-20T10:05:48Z

❌ Gradle check result for 5d6a0ad: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/cluster/metadata/RepositoriesMetadata.java

server/src/main/java/org/opensearch/cluster/metadata/TemplatesMetadata.java

server/src/main/java/org/opensearch/gateway/remote/ClusterMetadataManifest.java

soosinha · 2024-02-20T12:11:27Z

Looks good on a high level. Can you move it out of draft ?

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

server/src/main/java/org/opensearch/gateway/remote/ClusterMetadataManifest.java

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java

github-actions · 2024-02-22T08:00:09Z

❌ Gradle check result for 279dbbe: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-16T01:20:18Z

❌ Gradle check result for 52d5f81: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-16T02:31:40Z

❌ Gradle check result for 4f8a64e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-16T06:08:26Z

✅ Gradle check result for 4f8a64e: SUCCESS

shwetathareja

Approving this change. Lets refactor RemoteClusterStateService.java before making any further changes to this class.

…2190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <hishiv@amazon.com> (cherry picked from commit da3ab92) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <hishiv@amazon.com> (cherry picked from commit da3ab92)

…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <hishiv@amazon.com>

…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <hishiv@amazon.com> (cherry picked from commit da3ab92)

…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <31575408+shiv0408@users.noreply.github.com> (cherry picked from commit da3ab92)

…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <hishiv@amazon.com> (cherry picked from commit da3ab92)

…ibute files (#13703) * Split the remote global metadata file to metadata attribute files (#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <hishiv@amazon.com> (cherry picked from commit da3ab92) * Remove conflicting static method from Metadata.Custom interface Signed-off-by: Shivansh Arora <hishiv@amazon.com> --------- Signed-off-by: Shivansh Arora <hishiv@amazon.com>

…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <hishiv@amazon.com>

…ibute files (opensearch-project#13703) * Split the remote global metadata file to metadata attribute files (opensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <hishiv@amazon.com> (cherry picked from commit da3ab92) * Remove conflicting static method from Metadata.Custom interface Signed-off-by: Shivansh Arora <hishiv@amazon.com> --------- Signed-off-by: Shivansh Arora <hishiv@amazon.com> Signed-off-by: kkewwei <kkewwei@163.com>

Split the cluster state remote global metadata file to metadata attri…

6bf7bc9

…bute files Signed-off-by: Shivansh Arora <hishiv@amazon.com>

soosinha reviewed Feb 7, 2024

View reviewed changes

server/src/main/java/org/opensearch/cluster/metadata/Metadata.java Show resolved Hide resolved

Upload all the metadata attributes in parallel

5d6a0ad

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

shiv0408 force-pushed the cluster_state_split branch from f3e853b to 5d6a0ad Compare February 20, 2024 09:33

soosinha reviewed Feb 20, 2024

View reviewed changes

Added javadocs

279dbbe

Signed-off-by: Shivansh Arora <hishiv@amazon.com>

sarthakaggarwal97 reviewed Feb 21, 2024

View reviewed changes

server/src/main/java/org/opensearch/gateway/remote/ClusterMetadataManifest.java Outdated Show resolved Hide resolved

sarthakaggarwal97 reviewed Feb 21, 2024

View reviewed changes

server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java Show resolved Hide resolved

shiv0408 changed the title ~~Split the cluster state remote global metadata file to metadata attri…~~ Split the remote global metadata file to metadata attribute files Feb 26, 2024

shiv0408 marked this pull request as ready for review February 26, 2024 08:32

shiv0408 requested review from abbashus, adnapibar, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kartg and kotwanikunal as code owners February 26, 2024 08:32

Merge branch 'main' into cluster_state_split

4f8a64e

shiv0408 force-pushed the cluster_state_split branch from 52d5f81 to 4f8a64e Compare May 16, 2024 01:30

shwetathareja approved these changes May 16, 2024

View reviewed changes

shwetathareja merged commit da3ab92 into opensearch-project:main May 16, 2024
28 checks passed

shwetathareja added the backport 2.x Backport to 2.x branch label May 16, 2024

opensearch-trigger-bot bot mentioned this pull request May 16, 2024

[Backport 2.x] Split the remote global metadata file to metadata attribute files #13699

Closed

shiv0408 mentioned this pull request May 16, 2024

[Backport 2.x] Split the remote global metadata file to metadata attribute files #13703

Merged

shwetathareja mentioned this pull request May 30, 2024

Remove conflicting static method from Metadata.Custom interface #13869

Merged

9 tasks

shiv0408 mentioned this pull request Jun 11, 2024

Create Remote Object managers and use them in orchestration from RemoteClusterStateService #13924

Merged

9 tasks

finnegancarroll mentioned this pull request Jun 14, 2024

Remove composeBuild fixture tasks when docker support not found #14357

Merged

3 tasks

prudhvigodithi mentioned this pull request Jun 18, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT prudhvigodithi/OpenSearch#27

Closed

shiv0408 deleted the cluster_state_split branch June 18, 2024 10:46

This was referenced Jun 27, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT #14288

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteReplicaRecoveryIT #14303

Open

[AUTOCUT] Gradle Check Flaky Test Report for AzureBlobContainerRetriesTests #14318

Open

shiv0408 self-assigned this Aug 5, 2024

gbbafna mentioned this pull request Aug 8, 2024

Add Varun Bansal as maintainer #15163

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split the remote global metadata file to metadata attribute files #12190

Split the remote global metadata file to metadata attribute files #12190

shiv0408 commented Feb 6, 2024 •

edited

Loading

github-actions bot commented Feb 6, 2024 •

edited

Loading

github-actions bot commented Feb 6, 2024

github-actions bot commented Feb 15, 2024

github-actions bot commented Feb 19, 2024

github-actions bot commented Feb 20, 2024

github-actions bot commented Feb 20, 2024

soosinha commented Feb 20, 2024

github-actions bot commented Feb 22, 2024

github-actions bot commented May 16, 2024

github-actions bot commented May 16, 2024

github-actions bot commented May 16, 2024

shwetathareja left a comment

Split the remote global metadata file to metadata attribute files #12190

Split the remote global metadata file to metadata attribute files #12190

Conversation

shiv0408 commented Feb 6, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Feb 6, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Feb 6, 2024

github-actions bot commented Feb 15, 2024

github-actions bot commented Feb 19, 2024

github-actions bot commented Feb 20, 2024

github-actions bot commented Feb 20, 2024

soosinha commented Feb 20, 2024

github-actions bot commented Feb 22, 2024

github-actions bot commented May 16, 2024

github-actions bot commented May 16, 2024

github-actions bot commented May 16, 2024

shwetathareja left a comment

Choose a reason for hiding this comment

shiv0408 commented Feb 6, 2024 •

edited

Loading

github-actions bot commented Feb 6, 2024 •

edited

Loading