-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
explicit outputType for ExpressionPostAggregator, better documentation for the differences between arrays and mvds #15245
Merged
Merged
Changes from 2 commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
4c5a5db
better documentation for the differences between arrays and mvds
clintropolis b729e86
Merge remote-tracking branch 'upstream/master' into arrays-are-not-mvds
clintropolis 1b0ccbb
Merge remote-tracking branch 'upstream/master' into arrays-are-not-mvds
clintropolis 5490ff0
adjustments
clintropolis 8fe01fe
fix link
clintropolis aaa1486
missed a spot
clintropolis bcefe45
doc fixes, add outputType to ExpressionPostAggregator to make docs true
clintropolis 7a25e0a
more test
clintropolis fa2a356
Merge remote-tracking branch 'upstream/master' into arrays-are-not-mvds
clintropolis 7168f50
add output coercion if outputType is defined on ExpressionPostAgg, up…
clintropolis 0e6a862
fix
clintropolis 429f8ab
more test
clintropolis 60dbc6e
adjust
clintropolis File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -88,6 +88,9 @@ When deciding whether to use `REPLACE` or `INSERT`, keep in mind that segments g | |
with dimension-based pruning but those generated with `INSERT` cannot. For more information about the requirements | ||
for dimension-based pruning, see [Clustering](#clustering). | ||
|
||
To insert [ARRAY types](../querying/arrays.md), be sure to set context flag `"arrayIngestMode":"array"` which allows | ||
ARRAY types to be stored in segments. This flag is not enabled by default. | ||
|
||
For more information about the syntax, see [INSERT](./reference.md#insert). | ||
|
||
<a name="replace"></a> | ||
|
@@ -192,10 +195,13 @@ To perform ingestion with rollup: | |
2. Set [`finalizeAggregations: false`](reference.md#context-parameters) in your context. This causes aggregation | ||
functions to write their internal state to the generated segments, instead of the finalized end result, and enables | ||
further aggregation at query time. | ||
3. Wrap all multi-value strings in `MV_TO_ARRAY(...)` and set [`groupByEnableMultiValueUnnesting: | ||
false`](reference.md#context-parameters) in your context. This ensures that multi-value strings are left alone and | ||
remain lists, instead of being [automatically unnested](../querying/sql-data-types.md#multi-value-strings) by the | ||
`GROUP BY` operator. | ||
3. To ingest [Druid multi-value dimensions](../querying/multi-value-dimensions.md), wrap all multi-value strings | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This direction has become too complicated for people to understand, so I think we'll need an example. Or to link to one. |
||
in `MV_TO_ARRAY(...)` in the grouping clause and set [`groupByEnableMultiValueUnnesting: false`](reference.md#context-parameters) in your context. | ||
This ensures that multi-value strings are left alone and remain lists, instead of being [automatically unnested](../querying/sql-data-types.md#multi-value-strings) by the | ||
`GROUP BY` operator. To INSERT these arrays as multi-value strings, wrap the expressions in the SELECT clause with | ||
`ARRAY_TO_MV` to coerce the ARRAY back to a VARCHAR | ||
4. To ingest [ARRAY types](../querying/arrays.md), be sure to set context flag `"arrayIngestMode":"array"` which allows | ||
ARRAY types to be stored in segments. This flag is not enabled by default. | ||
|
||
When you do all of these things, Druid understands that you intend to do an ingestion with rollup, and it writes | ||
rollup-related metadata into the generated segments. Other applications can then use [`segmentMetadata` | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm this seems like the wrong place to put this. It's generic docs about
INSERT
, we don't want to gunk it up with stuff about specific data that might be inserted. (Otherwise this would be, like, 10 times longer.)I suggest cutting it, and relying on the examples and the array docs to guide people.