-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add REGEXP_REPLACE function. #14460
Add REGEXP_REPLACE function. #14460
Conversation
Replaces all instances of a pattern with a replacement string.
processing/src/main/java/org/apache/druid/query/expression/RegexpReplaceExprMacro.java
Fixed
Show fixed
Hide fixed
if (s == null || pattern == null || replacement == null) { | ||
return ExprEval.of(null); | ||
} else { | ||
final Matcher matcher = Pattern.compile(pattern).matcher(s); |
Check failure
Code scanning / CodeQL
Regular expression injection
testHelper.testExpressionString( | ||
new RegexpReplaceOperatorConversion().calciteOperator(), | ||
ImmutableList.of( | ||
testHelper.makeInputRef("s"), | ||
testHelper.makeLiteral("x(.)"), | ||
testHelper.makeLiteral("z") | ||
), | ||
makeExpression("regexp_replace(\"s\",'x(.)','z')"), | ||
"foo" | ||
); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
testHelper.makeLiteral("x(.)"), | ||
testHelper.makeLiteral("z") | ||
), | ||
makeExpression("regexp_replace(\"s\",'x(.)','z')"), |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
testHelper.testExpressionString( | ||
new RegexpReplaceOperatorConversion().calciteOperator(), | ||
ImmutableList.of( | ||
testHelper.makeInputRef("s"), | ||
testHelper.makeLiteral("(o)"), | ||
testHelper.makeLiteral("z") | ||
), | ||
makeExpression("regexp_replace(\"s\",'(o)','z')"), | ||
"fzz" | ||
); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
testHelper.makeLiteral("(o)"), | ||
testHelper.makeLiteral("z") | ||
), | ||
makeExpression("regexp_replace(\"s\",'(o)','z')"), |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
testHelper.testExpressionString( | ||
new RegexpReplaceOperatorConversion().calciteOperator(), | ||
ImmutableList.of( | ||
testHelper.makeCall( | ||
SqlStdOperatorTable.CONCAT, | ||
testHelper.makeLiteral("Z"), | ||
testHelper.makeInputRef("s") | ||
), | ||
testHelper.makeLiteral("Zf(.)"), | ||
testHelper.makeLiteral("z") | ||
), | ||
makeExpression("regexp_replace(concat('Z',\"s\"),'Zf(.)','z')"), | ||
"zo" | ||
); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
testHelper.makeLiteral("Zf(.)"), | ||
testHelper.makeLiteral("z") | ||
), | ||
makeExpression("regexp_replace(concat('Z',\"s\"),'Zf(.)','z')"), |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
testHelper.testExpressionString( | ||
new RegexpReplaceOperatorConversion().calciteOperator(), | ||
ImmutableList.of( | ||
testHelper.makeInputRef("s"), | ||
testHelper.makeLiteral("f(.)"), | ||
testHelper.makeLiteral("$1") | ||
), | ||
makeExpression("regexp_replace(\"s\",'f(.)','$1')"), | ||
"oo" | ||
); |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
testHelper.makeLiteral("f(.)"), | ||
testHelper.makeLiteral("$1") | ||
), | ||
makeExpression("regexp_replace(\"s\",'f(.)','$1')"), |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
Beast. |
final String patternString = NullHandling.nullToEmptyIfNeeded((String) patternExpr.getLiteralValue()); | ||
|
||
this.arg = args.get(0); | ||
this.pattern = patternString != null ? Pattern.compile(patternString) : null; |
Check failure
Code scanning / CodeQL
Regular expression injection
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a real problem, but.. I'm not really sure the best way to solve the DoS problem since we do want them to be able to provide a pattern, and it can happen in other places where regex are provided by user queries too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
final String patternString = NullHandling.nullToEmptyIfNeeded((String) patternExpr.getLiteralValue()); | ||
|
||
this.arg = args.get(0); | ||
this.pattern = patternString != null ? Pattern.compile(patternString) : null; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is a real problem, but.. I'm not really sure the best way to solve the DoS problem since we do want them to be able to provide a pattern, and it can happen in other places where regex are provided by user queries too.
private static final SqlFunction SQL_FUNCTION = OperatorConversions | ||
.operatorBuilder("REGEXP_REPLACE") | ||
.operandTypes(SqlTypeFamily.CHARACTER, SqlTypeFamily.CHARACTER, SqlTypeFamily.CHARACTER) | ||
.requiredOperands(3) |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
Something got wedged with the test It's not able to build a docker image due to a problem with |
* Add REGEXP_REPLACE function. Replaces all instances of a pattern with a replacement string. * Fixes. * Improve test coverage. * Adjust behavior.
Replaces all instances of a pattern with a replacement string.