Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

can not read realtime data when using indexer process instead of middleManager peon process to ingest metrics from kafka #10538

Closed
quenlang opened this issue Oct 28, 2020 · 6 comments
Closed

can not read realtime data when using indexer process instead of middleManager peon process to ingest metrics from kafka #10538

quenlang opened this issue Oct 28, 2020 · 6 comments
Labels
Area - Querying Area - Streaming Ingestion Bug

Comments

@quenlang
Copy link

Hi, all

Affected Version
0.19.0 and 0.20.0

Description

I found that can not query real-time data when using the indexer process instead of the peon process to ingest metrics from the Kafka topic.
Then I terminated the ingestion task, the segments handoff to HDFS and the historical nodes cached the segments, it is queryable.

I downgraded to 0.18.1 but didn't meet this problem, it worked fine.

I think this is a bug after 0.18.1. Any suggestions? @jihoonson
Thanks a lot!
@jihoonson
Copy link
Contributor

Hi @quenlang, I haven't heard of any issues so far, so want to narrow down the scope of potential issue. Can you check the result of the druid/v2/candidates API from the broker (https://druid.apache.org/docs/latest/operations/api-reference.html#queries)? It returns what servers the query will hit. If the result doesn't include the indexer, there might be some problem in segment announcing. Otherwise, the issue might be query execution in indexers.

@quenlang
Copy link
Author

quenlang commented Oct 29, 2020 •
edited
Loading

Hi @jihoonson, as you say, the response of 0.18.1 includes the indexer node, but 0.19.0 doesn't.
Any suggestions?

the response of the same query in 0.18.1:

[ {
  "interval" : "2020-10-28T12:00:00.000Z/2020-10-28T18:00:00.000Z",
  "version" : "2020-10-28T12:30:25.490Z",
  "partitionNumber" : 4,
  "size" : 243155,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T12:00:00.000Z/2020-10-28T18:00:00.000Z",
  "version" : "2020-10-28T12:30:25.490Z",
  "partitionNumber" : 5,
  "size" : 649577,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T12:00:00.000Z/2020-10-28T18:00:00.000Z",
  "version" : "2020-10-28T12:30:25.490Z",
  "partitionNumber" : 6,
  "size" : 712699,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T12:00:00.000Z/2020-10-28T18:00:00.000Z",
  "version" : "2020-10-28T12:30:25.490Z",
  "partitionNumber" : 7,
  "size" : 678307,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T12:00:00.000Z/2020-10-28T18:00:00.000Z",
  "version" : "2020-10-28T12:30:25.490Z",
  "partitionNumber" : 8,
  "size" : 51998961,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T18:00:00.000Z/2020-10-29T00:00:00.000Z",
  "version" : "2020-10-28T18:00:00.275Z",
  "partitionNumber" : 0,
  "size" : 9444651,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T18:00:00.000Z/2020-10-29T00:00:00.000Z",
  "version" : "2020-10-28T18:00:00.275Z",
  "partitionNumber" : 1,
  "size" : 49608689,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-29T00:00:00.000Z/2020-10-29T06:00:00.000Z",
  "version" : "2020-10-29T00:00:00.237Z",
  "partitionNumber" : 0,
  "size" : 9436024,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-29T00:00:00.000Z/2020-10-29T06:00:00.000Z",
  "version" : "2020-10-29T00:00:00.237Z",
  "partitionNumber" : 1,
  "size" : 0,
  "locations" : [ {
    "name" : "app.sg.com:8091",
    "host" : "app.sg.com:8091",
    "hostAndTlsPort" : null,
    "maxSize" : 0,
    "type" : "indexer-executor",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
} ]

in 0.19.0:

[ {
  "interval" : "2020-10-28T12:00:00.000Z/2020-10-28T18:00:00.000Z",
  "version" : "2020-10-28T12:30:25.490Z",
  "partitionNumber" : 4,
  "size" : 243155,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T12:00:00.000Z/2020-10-28T18:00:00.000Z",
  "version" : "2020-10-28T12:30:25.490Z",
  "partitionNumber" : 5,
  "size" : 649577,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T12:00:00.000Z/2020-10-28T18:00:00.000Z",
  "version" : "2020-10-28T12:30:25.490Z",
  "partitionNumber" : 6,
  "size" : 712699,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T12:00:00.000Z/2020-10-28T18:00:00.000Z",
  "version" : "2020-10-28T12:30:25.490Z",
  "partitionNumber" : 7,
  "size" : 678307,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T12:00:00.000Z/2020-10-28T18:00:00.000Z",
  "version" : "2020-10-28T12:30:25.490Z",
  "partitionNumber" : 8,
  "size" : 51998961,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T18:00:00.000Z/2020-10-29T00:00:00.000Z",
  "version" : "2020-10-28T18:00:00.275Z",
  "partitionNumber" : 0,
  "size" : 9444651,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-28T18:00:00.000Z/2020-10-29T00:00:00.000Z",
  "version" : "2020-10-28T18:00:00.275Z",
  "partitionNumber" : 1,
  "size" : 49608689,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
}, {
  "interval" : "2020-10-29T00:00:00.000Z/2020-10-29T06:00:00.000Z",
  "version" : "2020-10-29T00:00:00.237Z",
  "partitionNumber" : 0,
  "size" : 9436024,
  "locations" : [ {
    "name" : "app.sg.com:8083",
    "host" : "app.sg.com:8083",
    "hostAndTlsPort" : null,
    "maxSize" : 1500000000000,
    "type" : "historical",
    "tier" : "_default_tier",
    "priority" : 0
  } ]
} ]

@jihoonson
Copy link
Contributor

Thanks. I think maybe something was broken in #9971, but not 100% sure yet. I will dig into it more.

@dlyoungerman
Copy link

dlyoungerman commented Nov 11, 2020 •
edited
Loading

@jihoonson - We are also experiencing the same issue after upgrading two of our clusters to 0.19. Our temporary solution was to downgrade our indexers to 0.18.1, inspired by this issue: #10258.

I'm curious if there's an ETA for fixing this? It's kind of a big deal for our users. I know indexers are experimental, do you think we should revert to using middle managers and peons for the stability?

@jihoonson jihoonson mentioned this issue Dec 4, 2020
Merged
@gianm
Copy link
Contributor

gianm commented Dec 4, 2020

#10631 should fix this.

@jihoonson
Copy link
Contributor

Closing as #10631 is merged. Please feel free to reopen if you still see the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Area - Querying Area - Streaming Ingestion Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dlyoungerman @gianm @jihoonson @quenlang