Add replication factor column to sys table #14403

adarshsanjeev · 2023-06-09T14:06:05Z

Currently, Druid allows segments to be not loaded on any historical. This status is not tracked in the sys.segments table on the broker. This makes it difficult to tell from the segments table why a segment is not available, and if that is expected. This also makes it hard to decide what segments to wait for to load after an ingestion is finished, as those segments may never get loaded due to the load rules.

This PR adds a new column replication factor to the sys virtual table and populates it with information from the coordinator. If this column is 0, the segment is not assigned to any historical and will never be loaded.

This column can be access from the sys table by normal sql queries:
SELECT "segment_id", "replication_factor" FROM sys."segments"

Response:

    [
        "segment_id",
        "replication_factor"
    ],
    [
        "STRING",
        "LONG"
    ],
    [
        "VARCHAR",
        "BIGINT"
    ],
    [
        "kttm_simple_-146136543-09-08T08:23:32.096Z_146140482-04-24T15:36:27.903Z_2023-06-14T03:38:24.502Z",
        -1
    ],
    [
        "kttm_simple_-146136543-09-08T08:23:32.096Z_146140482-04-24T15:36:27.903Z_2023-06-14T03:40:13.463Z",
        2
    ]
]

Release Notes

Adds a new virtual column replication factor to the sys table. This returns the total number of replicants of the segment across all tiers. The column is set to -1 if the information is not available currently.

abhishekrb19

Thanks for the change! I've a few questions related to API backwards compatibility and naming consistency.

docs/api-reference/api-reference.md

sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java

docs/api-reference/api-reference.md

processing/src/main/java/org/apache/druid/timeline/SegmentPlus.java

server/src/main/java/org/apache/druid/server/coordinator/DruidCoordinator.java

server/src/main/java/org/apache/druid/server/coordinator/SegmentReplicantLookup.java

server/src/main/java/org/apache/druid/server/coordinator/DruidCoordinator.java

server/src/main/java/org/apache/druid/server/coordinator/SegmentReplicantLookup.java

abhishekrb19

Overall approach looks good to me. One question related to setReplicationFactor and a few minor nits, thanks!

sql/src/test/java/org/apache/druid/sql/calcite/schema/SystemSchemaTest.java

processing/src/main/java/org/apache/druid/timeline/SegmentStatusInCluster.java

sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java

processing/src/test/java/org/apache/druid/timeline/SegmentStatusInClusterTest.java

server/src/main/java/org/apache/druid/server/coordinator/SegmentReplicantLookup.java

processing/src/main/java/org/apache/druid/timeline/SegmentStatusInCluster.java

kfaraz

Left some final feedback comments.

@adarshsanjeev , could you please also share a web-console screenshot of the query and the results table from your local/cluster testing?

docs/querying/sql-metadata-tables.md

processing/src/main/java/org/apache/druid/timeline/SegmentStatusInCluster.java

processing/src/test/java/org/apache/druid/timeline/SegmentStatusInClusterTest.java

server/src/main/java/org/apache/druid/server/coordinator/SegmentReplicantLookup.java

sql/src/main/java/org/apache/druid/sql/calcite/schema/SystemSchema.java

…tor fails

cryptoe

Left minor comments. Overall LGTM. Will approve once those are addressed.

sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java

server/src/main/java/org/apache/druid/server/coordinator/SegmentReplicantLookup.java

kfaraz

Looks good to me.

@adarshsanjeev , for the coverage issues, you can try adding a simple test for MetadataResource similar to DataSourceResourceTest or tests for other resource classes.

docs/querying/sql-metadata-tables.md

server/src/main/java/org/apache/druid/server/coordinator/DruidCoordinator.java

server/src/test/java/org/apache/druid/server/http/MetadataResourceTest.java

sql/src/main/java/org/apache/druid/sql/calcite/schema/MetadataSegmentView.java

server/src/test/java/org/apache/druid/server/http/MetadataResourceTest.java

kfaraz · 2023-06-16T16:24:38Z

Left a few minor comments, but none of them are blockers for this PR.

This PR catches the console up to all the backend changes for Druid 27 Specifically: Add page information to SqlStatementResource API #14512 Allow empty tiered replicants map for load rules #14432 Adding Interactive API's for MSQ engine #14416 Add replication factor column to sys table #14403 Account for data format and compression in MSQ auto taskAssignment #14307 Errors take 3 #14004

This PR catches the console up to all the backend changes for Druid 27 Specifically: Add page information to SqlStatementResource API apache#14512 Allow empty tiered replicants map for load rules apache#14432 Adding Interactive API's for MSQ engine apache#14416 Add replication factor column to sys table apache#14403 Account for data format and compression in MSQ auto taskAssignment apache#14307 Errors take 3 apache#14004

This PR catches the console up to all the backend changes for Druid 27 Specifically: Add page information to SqlStatementResource API #14512 Allow empty tiered replicants map for load rules #14432 Adding Interactive API's for MSQ engine #14416 Add replication factor column to sys table #14403 Account for data format and compression in MSQ auto taskAssignment #14307 Errors take 3 #14004 Co-authored-by: Vadim Ogievetsky <vadim@ogievetsky.com>

This PR catches the console up to all the backend changes for Druid 27 Specifically: Add page information to SqlStatementResource API apache#14512 Allow empty tiered replicants map for load rules apache#14432 Adding Interactive API's for MSQ engine apache#14416 Add replication factor column to sys table apache#14403 Account for data format and compression in MSQ auto taskAssignment apache#14307 Errors take 3 apache#14004

Add target replica column to sys table

f9ee4af

github-actions bot added the Area - Documentation label Jun 9, 2023

Remove old comment

a397e18

abhishekrb19 reviewed Jun 9, 2023

View reviewed changes

kfaraz requested changes Jun 10, 2023

View reviewed changes

Address review comments

b431b29

abhishekrb19 reviewed Jun 12, 2023

View reviewed changes

processing/src/main/java/org/apache/druid/timeline/SegmentStatusInCluster.java Outdated Show resolved Hide resolved

Address review comments

8f6a470

adarshsanjeev changed the title ~~Add target replica column to sys table~~ Add replication factor column to sys table Jun 12, 2023

adarshsanjeev added 6 commits June 12, 2023 10:21

Update comments

10d14ec

Update comments

c1e98f8

Update docs

7c81ebd

Update doc with correct column name

9c9b975

Fix ITs

e7e0350

Update spellcheck

130e420

adarshsanjeev requested a review from kfaraz June 13, 2023 05:51

kfaraz reviewed Jun 13, 2023

View reviewed changes

adarshsanjeev added 2 commits June 14, 2023 09:07

Rename variables and update comments

0bda368

Cache responses in the broker to prevent incorrect values if coordina…

74473f2

…tor fails

vogievetsky added the Needs web console change Backend API changes that would benefit from frontend support in the web console label Jun 14, 2023

cryptoe reviewed Jun 15, 2023

View reviewed changes

kfaraz approved these changes Jun 15, 2023

View reviewed changes

docs/querying/sql-metadata-tables.md Outdated Show resolved Hide resolved

Add tests

7c43b8b

github-advanced-security bot found potential problems Jun 15, 2023

View reviewed changes

server/src/main/java/org/apache/druid/server/coordinator/DruidCoordinator.java Fixed Show fixed Hide fixed

adarshsanjeev added 4 commits June 15, 2023 21:43

Fix checkstyle

4822492

Resolve checkstyle failures

94e0204

Resolve missing dependency build failure

59fe7b4

Address review comments

ac977a1

kfaraz reviewed Jun 16, 2023

View reviewed changes

Update comments

2d55219

kfaraz merged commit 128133f into apache:master Jun 18, 2023

vogievetsky mentioned this pull request Jul 7, 2023

Web console: catchup to all the backend changes #14540

Merged

AmatyaAvadhanula mentioned this pull request Jul 17, 2023

(Backport #14540) Web console: catchup to all the backend changes #14596

Merged

abhishekagarwal87 added this to the 27.0 milestone Jul 19, 2023

vogievetsky mentioned this pull request Aug 3, 2023

DO NOT MERGE - 27.0.0 WIP release notes #14600

Closed

AmatyaAvadhanula mentioned this pull request Aug 6, 2023

[DRAFT] 27.0.0 release notes #14761

Closed

vogievetsky removed the Needs web console change Backend API changes that would benefit from frontend support in the web console label Aug 8, 2023

cryptoe mentioned this pull request Sep 4, 2023

Add code to wait for segments generated to be loaded on historicals #14322

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add replication factor column to sys table #14403

Add replication factor column to sys table #14403

adarshsanjeev commented Jun 9, 2023 •

edited

Loading

abhishekrb19 left a comment

abhishekrb19 left a comment

kfaraz left a comment

cryptoe left a comment

kfaraz left a comment

kfaraz commented Jun 16, 2023

Add replication factor column to sys table #14403

Add replication factor column to sys table #14403

Conversation

adarshsanjeev commented Jun 9, 2023 • edited Loading

abhishekrb19 left a comment

Choose a reason for hiding this comment

abhishekrb19 left a comment

Choose a reason for hiding this comment

kfaraz left a comment

Choose a reason for hiding this comment

cryptoe left a comment

Choose a reason for hiding this comment

kfaraz left a comment

Choose a reason for hiding this comment

kfaraz commented Jun 16, 2023

adarshsanjeev commented Jun 9, 2023 •

edited

Loading