-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add IS_MULTIVALUE and COMPLEX_TYPE columns to INFORMATION_SCHEMA.COLUMNS #9707
Comments
+1 on the concept. This would be a great addition. The DruidSchema is actually in a place already where it could know which columns are multivalue — it gets it from SegmentMetadata queries and caches the results to avoid performance issues. It just doesn't plumb the information into the INFORMATION_SCHEMA. I don't recall if SegmentMetadata queries return info about what complex types are in play, but if they don't, they could be extended to do so. @JulianJaffePinterest any interest in contributing something that does this? |
Segment Metadata queries do return the complex type name (e.g. |
After doing some digging, it seems like at least the |
AFAIK @clintropolis hasn't actively been working on the SQL part of the type work. So plumbing through the stuff that already exists (like complex type name) should be open ground. @clintropolis please let us know if that ends up not being accurate. |
Ah yeah, @gianm is right, i haven't worked my way up to the SQL stuff yet. Go ahead if you've got a change in mind, I can work around whatever you get wired up here. I don't believe i'll be removing Even after that change, it still seems useful for the information schema, I'd be happy to review your PR once it exists 👍 |
I believe #11713 addresses the complex type half of this request. Leaving this open to track the IS_MULTIVALUE half. |
Description
Currently, the
INFORMATION_SCHEMA.COLUMNS
table in Druid does not have full information about the columns in a data source, specifically whether or not a column can contain multi-valued entries and detailed information about which complex serde a metric uses. This means that users attempting to interact with Druid metadata must either fall back to (potentially expensive) SegmentMetadataQueries to determine this information or implement complicated handling logic.Motivation
It is currently difficult to obtain complete metadata about a Druid datasource. The only way to determine whether or not a column can contain multiple values or what complex serde was used to encode a metric is to use SegmentMetadataQueries. These queries can be expensive (if aggregating over a large number of segments) and can miss information if not all segments are queried (e.g. columns that don't appear in a given segment won't be returned by a SegmentMetadataQuery, and columns that don't have multiple values in a queried segment will return
..."hasMultipleValues" : false,...
). This can be worked around to some degree via merging results, but this reintroduces the problem that SegmentMetadataQueries can be expensive when run over many segments. If theINFORMATION_SCHEMA.COLUMNS
table were to be extended to include information about whether a column could contain multiple values and what serde was used for a complex metric, tools that interact with Druid metadata such as the Calcite Druid adapter, proposed Spark and Hive readers, and other third party integrations could issue simple SQL-based queries to determine data source metadata instead of needing to rely on SegmentMetadataQueries. This would also align with the Druid recommendation to use theINFORMATION_SCHEMA
tables for metadata if you're using SQL.The text was updated successfully, but these errors were encountered: