Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding dropResults parameter in queryOptions to drop the resultTable field from the response #10419

Merged
merged 3 commits into from
Mar 16, 2023

Conversation

soumitra-st
Copy link
Contributor

This PR adds a parameter to queryOptions to drop the resultTable from the response. This mode can be used to troubleshoot a customer's query (which may have sensitive data in the result) using metadata only.

Manual testing report:

% ./bin/quick-start-batch.sh
...
% curl http://localhost:9000/sql -d '{"sql":"select * from billing limit 10","trace":false}'
{"resultTable":{"dataSchema":{"columnNames":["alternateCharge","alternateUsage","baseCharge","baseUsage","billingMonth","buildingType","city","creditHistory","customerId","firstName","isCarOwner","lastName","levy","maritalStatus","missedPayment","overdueBalance","standingCharge","tax","totalNet","totalToPay"],"columnDataTypes":["BIG_DECIMAL","BIG_DECIMAL","BIG_DECIMAL","BIG_DECIMAL","STRING","STRING","STRING","STRING","INT","STRING","BOOLEAN","STRING","BIG_DECIMAL","STRING","STRING","BIG_DECIMAL","BIG_DECIMAL","BIG_DECIMAL","BIG_DECIMAL","BIG_DECIMAL"]},"rows":[[10.16,112.62,67.44,457.56,"201904","Condominium","Palo Alto","A",341,"Rebecca",false,"Effertz",2.55,"U","Paid",0,7.49,17.18,87.65,104.82],[0,0,150.32,1161.67,"201903","Condominium","Mountain View","A",374,"Reba",true,"OKeefe",4.68,"S","Paid",0,5.62,31.48,160.62,192.1],[0,0,58.13,449.21,"201904","Condominium","Palo Alto","A",398,"Sydnee",false,"Farrell",1.91,"S","Paid",0,5.62,12.87,65.66,78.53],[0,0,114.91,888.02,"201901","Apartment","Cupertino","A",427,"Susanna",false,"Schaefer",3.62,"S","Paid",0,5.62,24.33,124.15,148.48],[0,0,86.72,670.17,"201808","Apartment","Cupertino","B",435,"Stephen",true,"Conn",3.05,"M","Paid",0,14.87,20.51,104.64,125.15],[0,0,115.73,894.34,"201808","Apartment","Palo Alto","A",441,"Rickey",false,"Smitham",3.64,"M","Paid",0,5.62,24.5,124.99,149.49],[0,0,184.78,1427.99,"201808","Apartment","Cupertino","A",465,"Kristina",true,"Towne",5.99,"U","Paid",0,14.87,40.31,205.64,245.95],[0,0,57.59,445.09,"201904","Apartment","Mountain View","A",499,"Alexane",false,"Weber",1.95,"M","Paid",0,7.35,13.11,66.89,80],[0,0,95.55,738.44,"201810","Apartment","Santa Clara","A - Excellent",503,"Elouise",true,"Hickle",3.31,"U","Paid",0,14.87,22.29,113.74,136.03],[0,0,128.86,995.82,"201906","Apartment","Mountain View","A - Excellent",533,"Ansley",true,"Bogisich",4.03,"M","Paid",0,5.62,27.15,138.51,165.66]]},"exceptions":[],"numServersQueried":1,"numServersResponded":1,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numConsumingSegmentsQueried":0,"numConsumingSegmentsProcessed":0,"numConsumingSegmentsMatched":0,"numDocsScanned":10,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":200,"numGroupsLimitReached":false,"totalDocs":694,"timeUsedMs":9,"offlineThreadCpuTimeNs":0,"realtimeThreadCpuTimeNs":0,"offlineSystemActivitiesCpuTimeNs":0,"realtimeSystemActivitiesCpuTimeNs":0,"offlineResponseSerializationCpuTimeNs":0,"realtimeResponseSerializationCpuTimeNs":0,"offlineTotalCpuTimeNs":0,"realtimeTotalCpuTimeNs":0,"segmentStatistics":[],"traceInfo":{},"minConsumingFreshnessTimeMs":0,"numSegmentsPrunedByBroker":0,"numSegmentsPrunedByServer":0,"numSegmentsPrunedInvalid":0,"numSegmentsPrunedByLimit":0,"numSegmentsPrunedByValue":0,"explainPlanNumEmptyFilterSegments":0,"explainPlanNumMatchAllFilterSegments":0,"numRowsResultSet":10}
% curl http://localhost:9000/sql -d '{"sql":"select * from billing limit 10","trace":false,"queryOptions":"dropResults=true"}'
{"exceptions":[],"numServersQueried":1,"numServersResponded":1,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numConsumingSegmentsQueried":0,"numConsumingSegmentsProcessed":0,"numConsumingSegmentsMatched":0,"numDocsScanned":10,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":200,"numGroupsLimitReached":false,"totalDocs":694,"timeUsedMs":6,"offlineThreadCpuTimeNs":0,"realtimeThreadCpuTimeNs":0,"offlineSystemActivitiesCpuTimeNs":0,"realtimeSystemActivitiesCpuTimeNs":0,"offlineResponseSerializationCpuTimeNs":0,"realtimeResponseSerializationCpuTimeNs":0,"offlineTotalCpuTimeNs":0,"realtimeTotalCpuTimeNs":0,"segmentStatistics":[],"traceInfo":{},"minConsumingFreshnessTimeMs":0,"numSegmentsPrunedByBroker":0,"numSegmentsPrunedByServer":0,"numSegmentsPrunedInvalid":0,"numSegmentsPrunedByLimit":0,"numSegmentsPrunedByValue":0,"explainPlanNumEmptyFilterSegments":0,"explainPlanNumMatchAllFilterSegments":0,"numRowsResultSet":10}
% curl http://localhost:9000/sql -d '{"sql":"select * from billing limit 10","trace":false,"queryOptions":"dropResults=TrUE"}'
{"exceptions":[],"numServersQueried":1,"numServersResponded":1,"numSegmentsQueried":1,"numSegmentsProcessed":1,"numSegmentsMatched":1,"numConsumingSegmentsQueried":0,"numConsumingSegmentsProcessed":0,"numConsumingSegmentsMatched":0,"numDocsScanned":10,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":200,"numGroupsLimitReached":false,"totalDocs":694,"timeUsedMs":2,"offlineThreadCpuTimeNs":0,"realtimeThreadCpuTimeNs":0,"offlineSystemActivitiesCpuTimeNs":0,"realtimeSystemActivitiesCpuTimeNs":0,"offlineResponseSerializationCpuTimeNs":0,"realtimeResponseSerializationCpuTimeNs":0,"offlineTotalCpuTimeNs":0,"realtimeTotalCpuTimeNs":0,"segmentStatistics":[],"traceInfo":{},"minConsumingFreshnessTimeMs":0,"numSegmentsPrunedByBroker":0,"numSegmentsPrunedByServer":0,"numSegmentsPrunedInvalid":0,"numSegmentsPrunedByLimit":0,"numSegmentsPrunedByValue":0,"explainPlanNumEmptyFilterSegments":0,"explainPlanNumMatchAllFilterSegments":0,"numRowsResultSet":10}

Added integration tests as well.

if (request.has(Broker.Request.QUERY_OPTIONS)) {
String queryOptions = request.get(Broker.Request.QUERY_OPTIONS).asText();
Map<String, String> optionsFromString = RequestUtils.getOptionsFromString(queryOptions);
if (optionsFromString.getOrDefault(Broker.Request.QueryOptionKey.DROP_RESULTS, "false")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may put this logic into QueryOptionsUtils

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is equivalent to Boolean.parseBoolean(queryOptions.get(QueryOptionKey.DROP_RESULTS))

@codecov-commenter
Copy link

Codecov Report

Merging #10419 (1b65f14) into master (3bad67d) will decrease coverage by 45.90%.
The diff coverage is 61.53%.

@@              Coverage Diff              @@
##             master   #10419       +/-   ##
=============================================
- Coverage     70.32%   24.43%   -45.90%     
+ Complexity     6105       49     -6056     
=============================================
  Files          2055     2028       -27     
  Lines        111389   110638      -751     
  Branches      16939    16850       -89     
=============================================
- Hits          78337    27034    -51303     
- Misses        27566    80780    +53214     
+ Partials       5486     2824     -2662     
Flag Coverage Δ
integration1 24.43% <61.53%> (+0.02%) ⬆️
integration2 ?
unittests1 ?
unittests2 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...va/org/apache/pinot/spi/utils/CommonConstants.java 0.00% <ø> (-24.40%) ⬇️
...roker/requesthandler/BaseBrokerRequestHandler.java 59.27% <50.00%> (-13.83%) ⬇️
...t/common/response/broker/BrokerResponseNative.java 90.58% <50.00%> (-5.87%) ⬇️
...e/pinot/common/utils/config/QueryOptionsUtils.java 68.18% <71.42%> (+0.38%) ⬆️

... and 1538 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Contributor

@Jackie-Jiang Jackie-Jiang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM otherwise

Comment on lines 197 to 209
public static boolean isDropResultsEnabled(final JsonNode request) {
boolean dropResults = false;

if (request.has(CommonConstants.Broker.Request.QUERY_OPTIONS)) {
String queryOptions = request.get(CommonConstants.Broker.Request.QUERY_OPTIONS).asText();
Map<String, String> optionsFromString = RequestUtils.getOptionsFromString(queryOptions);
if (Boolean.parseBoolean(optionsFromString.get(CommonConstants.Broker.Request.QueryOptionKey.DROP_RESULTS))) {
dropResults = true;
}
}

return dropResults;
}
Copy link
Contributor

@Jackie-Jiang Jackie-Jiang Mar 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We want to take the Map here instead of the JsonNode. The logic of extracting the map should be on the caller side. Also suggest rename it

Suggested change
public static boolean isDropResultsEnabled(final JsonNode request) {
boolean dropResults = false;
if (request.has(CommonConstants.Broker.Request.QUERY_OPTIONS)) {
String queryOptions = request.get(CommonConstants.Broker.Request.QUERY_OPTIONS).asText();
Map<String, String> optionsFromString = RequestUtils.getOptionsFromString(queryOptions);
if (Boolean.parseBoolean(optionsFromString.get(CommonConstants.Broker.Request.QueryOptionKey.DROP_RESULTS))) {
dropResults = true;
}
}
return dropResults;
}
public static boolean shouldDropResults(Map<String, String> queryOptions) {
return Boolean.parseBoolean(queryOptions.get(CommonConstants.Broker.Request.QueryOptionKey.DROP_RESULTS));
}

@mayankshriv mayankshriv merged commit dcddb9f into apache:master Mar 16, 2023
@soumitra-st soumitra-st deleted the STP-1713 branch March 16, 2023 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants