Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Null string is encoded as "null" in incremental index #2765

Merged
merged 1 commit into from
Apr 1, 2016

Conversation

navis
Copy link
Contributor

@navis navis commented Mar 31, 2016

I've found null values for string type is encoded as "null", which seemed not intended behavior.

@gianm
Copy link
Contributor

gianm commented Mar 31, 2016

Is it possible to include a test that would have caught this?

{
return String.valueOf(o);
return o == null ? null : String.valueOf(o);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain in your PR description why this line is not enough to fix the problem. What is the purpose of the remaining changes?

@jon-wei
Copy link
Contributor

jon-wei commented Mar 31, 2016

Can we remove the changes not related to the String null handling?

The code being refactored is being redone in #2760, e.g. those value transformers in IncrementalIndex are removed.

@jon-wei jon-wei added the Bug label Mar 31, 2016
@jon-wei jon-wei added this to the 0.9.1 milestone Mar 31, 2016
@navis
Copy link
Contributor Author

navis commented Apr 1, 2016

@jon-wei sure

@navis navis force-pushed the invalid-encode-nullstring branch from 0f5a845 to f0e55f5 Compare April 1, 2016 00:47
@drcrallen
Copy link
Contributor

Is this a bug that affects 0.9.0, or just 0.9.1?

@navis
Copy link
Contributor Author

navis commented Apr 1, 2016

@drcrallen it's from #2263, which is not included in any released versions.

@jon-wei
Copy link
Contributor

jon-wei commented Apr 1, 2016

@drcrallen it'll just be for 0.9.1, the code being changed was from #2263, which isn't in 0.9.0

@jon-wei
Copy link
Contributor

jon-wei commented Apr 1, 2016

👍

@@ -101,7 +101,7 @@
@Override
public String apply(final Object o)
{
return String.valueOf(o);
return o == null ? null : String.valueOf(o);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be converting "" to null or is something else doing that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's fine, null/empty dim values will end up as "" via a nullToEmpty() call when added to the dictionary:

https://github.com/navis/druid/blob/invalid-encode-nullstring/processing/src/main/java/io/druid/segment/incremental/IncrementalIndex.java#L1125

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok, this looks good then. thanks

@gianm
Copy link
Contributor

gianm commented Apr 1, 2016

👍

@gianm gianm merged commit 23d66e5 into apache:master Apr 1, 2016
try {
return Long.valueOf((String) o);
return s.isEmpty() ? null : Long.valueOf(s);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this change any current behavior that might map empty strings to 0 for numeric values?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it could, since with the old code, Long.valueOf("") would have thrown an exception. The new code would return null.

@jon-wei jon-wei mentioned this pull request Apr 2, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants