-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add DATETIMECONVERTWINDOWHOP function #11773
Conversation
4ba5cf2
to
ed32f13
Compare
@Jackie-Jiang - you can try it out yourself, works like a charm on single stage. |
Codecov Report
@@ Coverage Diff @@
## master #11773 +/- ##
============================================
+ Coverage 61.45% 61.49% +0.03%
+ Complexity 1147 1146 -1
============================================
Files 2375 2382 +7
Lines 128500 128633 +133
Branches 19846 19862 +16
============================================
+ Hits 78974 79097 +123
Misses 43815 43815
- Partials 5711 5721 +10
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 13 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
ed32f13
to
e94b8c2
Compare
@Jackie-Jiang Fixed unit test. and FYI I also didn't implement it as a scalar function. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good in general
pinot-common/src/main/java/org/apache/pinot/common/function/TransformFunctionType.java
Outdated
Show resolved
Hide resolved
...rg/apache/pinot/core/operator/transform/function/DateTimeConversionHopTransformFunction.java
Outdated
Show resolved
Hide resolved
private final long _outputGranularityMillis; | ||
protected final long _hopWindowSizeMillis; | ||
|
||
public BaseDateTimeWindowHopTransformer(@Nonnull DateTimeFormatSpec inputFormat, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(convention) We usually only annotate @Nullable
and assume everything non-null if not annotated. Same for other classes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
.../pinot/core/operator/transform/transformer/datetimehop/EpochToEpochWindowHopTransformer.java
Outdated
Show resolved
Hide resolved
d752635
to
191aac4
Compare
191aac4
to
9353de0
Compare
@Jackie-Jiang do you have any other feedback regarding this diff? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! Can you help add documentation for this new feature following instructions here?
Sure! Will do this week |
Adding
DATETIMECONVERTWINDOWHOP
converts the value from a column that contains an epoch timestamp into another time unit and buckets based on the given time granularity and window size.#11775
Usage Examples
These examples are based on the Batch JSON Quick Start.
created_at_timestamp from milliseconds since epoch to seconds since epoch, bucketed to 1 hour window with 15 min granularity:
Moving window of unique user counts per hour with 15 min granularity:
Testing
The feature is tested by unit tests + manual testing
Screenshots run locally