docs: query from deep storage #14609

317brian · 2023-07-18T22:44:44Z

WIP but there's enough here to start the review process

Adds a new page for query from deep storage and puts it in the Druid SQL section of Querying
Updates SQL API reference to include the query from deep storage endpoint
Updates mentions of deep storage that previously stated that you couldn't query from it
Moves durable storage info (mostly) out of SQL-based ingestion and to operations since the info is now shared between SQL-based ingestion and query from deep storage. (Note that this is part of breaking up some of the info in there and moving toward an information architecture where SQL-based ingestion is the default for batch.)
Updates webconsole screenshots for datasource and segments

This PR has:

been self-reviewed.

317brian · 2023-07-18T23:05:58Z

docs/operations/durable-storage.md

+
+To enable durable storage, you need to set the following common service properties:
+
+```


@cryptoe are there any new durable storage configs specifically for query from deep storage? Or are results written to the tempDir property?

The tempDir is used more as a staging directory before pushing out bytes to s3. Its not related to results of the query.

docs/api-reference/sql-api.md

docs/querying/query-from-deep-storage.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

cryptoe

Partial review. Will finish the next round by eod.

cryptoe · 2023-07-27T03:38:42Z

docs/api-reference/sql-api.md

+You can use the `sql/statements` endpoint to query segments that exist only in deep storage and are not loaded onto your Historical processes as determined by your load rules. 
+
+Note that at least part of a datasource must be available on a Historical process so that Druid can plan your query.


Suggested change

Note that at least part of a datasource must be available on a Historical process so that Druid can plan your query.

Note that at least one segment of a datasource must be available on a Historical process so that the broker can plan your query. A quick way to check this is that data source should be visible on the druid console.

cryptoe · 2023-07-27T03:43:50Z

docs/api-reference/sql-api.md

+
+- The only supported value for `resultFormat` is JSON.
+- Only the user who submits a query can see the results.


The response why execution mode is async is this pojo: https://github.com/apache/druid/blob/master/extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/sql/entity/SqlStatementResult.java

We might want to mention that in the response payload.

cryptoe · 2023-07-27T03:44:29Z

docs/api-reference/sql-api.md

+```
+
+Returns information about the query associated with the given query ID. The response matches the response from the POST API if the query is accepted or running. The response for a completed query includes the same information as an in-progress query with several additions:


Get query status response and the postReq response is the same when executionMode=async

docs/api-reference/sql-api.md

docs/design/architecture.md

docs/operations/durable-storage.md

docs/operations/rule-configuration.md

docs/querying/query-from-deep-storage.md

docs/operations/durable-storage.md

docs/operations/rule-configuration.md

docs/querying/query-from-deep-storage.md

docs/tutorials/tutorial-query-deep-storage.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

cryptoe

Left some comments.
Rest all LGTM.
Thanks for adding the tutorial page.

docs/api-reference/sql-api.md

docs/multi-stage-query/reference.md

docs/operations/durable-storage.md

cryptoe · 2023-08-01T16:51:19Z

docs/operations/durable-storage.md

+
+To enable durable storage, you need to set the following common service properties:
+
+```


The tempDir is used more as a staging directory before pushing out bytes to s3. Its not related to results of the query.

Co-authored-by: Karan Kumar <karankumar1100@gmail.com>

cryptoe

LGTM. Left 2 comments.

docs/api-reference/sql-api.md

317brian · 2023-08-04T05:21:09Z

Can we go ahead and merge this? That CI failure looks unrelated to my PR since it is complaining about Intellij

cryptoe · 2023-08-04T05:40:20Z

Thanks for the PR @317brian.

* cold tier wip * wip * copyedits * wip * copyedits * copyedits * wip * wip * update rules page * typo * typo * update sidebar * moves durable storage info to its own page in operations * update screenshots * add apache license * Apply suggestions from code review Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * add query from deep storage tutorial stub * address some of the feedback * revert screenshot update. handled in separate pr * load rule update * wip tutorial * reformat deep storage endpoints * rest of tutorial * typo * cleanup * screenshot and sidebar for tutorial * add license * typos * Apply suggestions from code review Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * rest of review comments * clarify where results are stored * update api reference for durablestorage context param * Apply suggestions from code review Co-authored-by: Karan Kumar <karankumar1100@gmail.com> * comments * incorporate apache#14720 * address rest of comments * missed one * Update docs/api-reference/sql-api.md * Update docs/api-reference/sql-api.md --------- Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> Co-authored-by: demo-kratia <56242907+demo-kratia@users.noreply.github.com> Co-authored-by: Karan Kumar <karankumar1100@gmail.com> (cherry picked from commit 3b5b6c6)

* docs: query from deep storage (#14609) * cold tier wip * wip * copyedits * wip * copyedits * copyedits * wip * wip * update rules page * typo * typo * update sidebar * moves durable storage info to its own page in operations * update screenshots * add apache license * Apply suggestions from code review Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * add query from deep storage tutorial stub * address some of the feedback * revert screenshot update. handled in separate pr * load rule update * wip tutorial * reformat deep storage endpoints * rest of tutorial * typo * cleanup * screenshot and sidebar for tutorial * add license * typos * Apply suggestions from code review Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * rest of review comments * clarify where results are stored * update api reference for durablestorage context param * Apply suggestions from code review Co-authored-by: Karan Kumar <karankumar1100@gmail.com> * comments * incorporate #14720 * address rest of comments * missed one * Update docs/api-reference/sql-api.md * Update docs/api-reference/sql-api.md --------- Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> Co-authored-by: demo-kratia <56242907+demo-kratia@users.noreply.github.com> Co-authored-by: Karan Kumar <karankumar1100@gmail.com> (cherry picked from commit 3b5b6c6) * change code tab styles to Docusaurus2 format from Docu1 format * fix spelling file * doc cleanup (cherry picked from commit 2218a13) --------- Co-authored-by: Laksh Singla <lakshsingla@gmail.com>

* cold tier wip * wip * copyedits * wip * copyedits * copyedits * wip * wip * update rules page * typo * typo * update sidebar * moves durable storage info to its own page in operations * update screenshots * add apache license * Apply suggestions from code review Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * add query from deep storage tutorial stub * address some of the feedback * revert screenshot update. handled in separate pr * load rule update * wip tutorial * reformat deep storage endpoints * rest of tutorial * typo * cleanup * screenshot and sidebar for tutorial * add license * typos * Apply suggestions from code review Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> * rest of review comments * clarify where results are stored * update api reference for durablestorage context param * Apply suggestions from code review Co-authored-by: Karan Kumar <karankumar1100@gmail.com> * comments * incorporate apache#14720 * address rest of comments * missed one * Update docs/api-reference/sql-api.md * Update docs/api-reference/sql-api.md --------- Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com> Co-authored-by: demo-kratia <56242907+demo-kratia@users.noreply.github.com> Co-authored-by: Karan Kumar <karankumar1100@gmail.com> (cherry picked from commit 3b5b6c6) (cherry picked from commit ed83c56)

317brian added 13 commits July 12, 2023 12:44

cold tier wip

6dadb9d

Merge remote-tracking branch 'upstream/master' into hybrid-query-docs

04e784d

Merge remote-tracking branch 'upstream/master' into hybrid-query-docs

671d31c

wip

8dbad1d

copyedits

9ae0893

wip

20f808b

copyedits

7dc14c1

copyedits

9d0d4a3

wip

05057df

wip

3c1d839

update rules page

8fb3675

typo

11c749a

typo

9b45bfa

github-actions bot added the Area - Documentation label Jul 18, 2023

317brian added 2 commits July 18, 2023 16:03

update sidebar

52e0d2f

moves durable storage info to its own page in operations

ac0f39e

317brian commented Jul 18, 2023

View reviewed changes

vtlim requested a review from cryptoe July 18, 2023 23:06

317brian added 2 commits July 19, 2023 15:09

update screenshots

ef8039c

add apache license

cf24f72

317brian marked this pull request as ready for review July 20, 2023 17:09

Merge branch 'master' into hybrid-query-docs

a771418

vtlim reviewed Jul 24, 2023

View reviewed changes

Apply suggestions from code review

7880654

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

cryptoe reviewed Jul 27, 2023

View reviewed changes

317brian commented Jul 27, 2023

View reviewed changes

docs/querying/query-from-deep-storage.md Outdated Show resolved Hide resolved

317brian added 4 commits July 26, 2023 21:10

add query from deep storage tutorial stub

a1af5ca

address some of the feedback

f2b1526

revert screenshot update. handled in separate pr

ca26e60

load rule update

61dc630

screenshot and sidebar for tutorial

bc4d974

317brian requested a review from cryptoe July 31, 2023 20:41

317brian added 2 commits July 31, 2023 13:43

add license

110e840

typos

5dd46b1

vtlim reviewed Jul 31, 2023

View reviewed changes

317brian and others added 2 commits July 31, 2023 16:17

Apply suggestions from code review

26a9032

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

rest of review comments

c8ae087

vtlim approved these changes Aug 1, 2023

View reviewed changes

317brian added 2 commits July 31, 2023 17:30

clarify where results are stored

832a58d

update api reference for durablestorage context param

774adca

cryptoe reviewed Aug 1, 2023

View reviewed changes

317brian and others added 3 commits August 1, 2023 13:48

Apply suggestions from code review

ff632a4

Co-authored-by: Karan Kumar <karankumar1100@gmail.com>

comments

9e3109c

incorporate apache#14720

9840cb5

317brian requested a review from cryptoe August 2, 2023 17:26

adarshsanjeev mentioned this pull request Aug 3, 2023

Update durable storage docs #14720

Closed

10 tasks

317brian added 2 commits August 3, 2023 00:31

address rest of comments

a5d1e44

missed one

a17a19d

cryptoe approved these changes Aug 4, 2023

View reviewed changes

docs/api-reference/sql-api.md Outdated Show resolved Hide resolved

docs/api-reference/sql-api.md Outdated Show resolved Hide resolved

cryptoe added 2 commits August 4, 2023 10:04

Update docs/api-reference/sql-api.md

231171b

Update docs/api-reference/sql-api.md

39e3633

cryptoe merged commit 3b5b6c6 into apache:master Aug 4, 2023
9 of 10 checks passed

cryptoe added this to the 27.0 milestone Aug 4, 2023

317brian mentioned this pull request Aug 4, 2023

docs: query from deep storage (#14609) #14754

Merged

LakshSingla mentioned this pull request Aug 4, 2023

Docs cleanup from the PR "docs: query from deep storage" #14755

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: query from deep storage #14609

docs: query from deep storage #14609

317brian commented Jul 18, 2023 •

edited

Loading

317brian Jul 18, 2023

cryptoe Aug 1, 2023

cryptoe left a comment

cryptoe Jul 27, 2023

cryptoe Jul 27, 2023

cryptoe Jul 27, 2023

cryptoe left a comment

cryptoe Aug 1, 2023

cryptoe left a comment

317brian commented Aug 4, 2023 •

edited

Loading

cryptoe commented Aug 4, 2023


		To enable durable storage, you need to set the following common service properties:

		```

		You can use the `sql/statements` endpoint to query segments that exist only in deep storage and are not loaded onto your Historical processes as determined by your load rules.

		Note that at least part of a datasource must be available on a Historical process so that Druid can plan your query.

	Note that at least part of a datasource must be available on a Historical process so that Druid can plan your query.
	Note that at least one segment of a datasource must be available on a Historical process so that the broker can plan your query. A quick way to check this is that data source should be visible on the druid console.


		- The only supported value for `resultFormat` is JSON.
		- Only the user who submits a query can see the results.

		```

		Returns information about the query associated with the given query ID. The response matches the response from the POST API if the query is accepted or running. The response for a completed query includes the same information as an in-progress query with several additions:

docs: query from deep storage #14609

docs: query from deep storage #14609

Conversation

317brian commented Jul 18, 2023 • edited Loading

317brian Jul 18, 2023

Choose a reason for hiding this comment

cryptoe Aug 1, 2023

Choose a reason for hiding this comment

cryptoe left a comment

Choose a reason for hiding this comment

cryptoe Jul 27, 2023

Choose a reason for hiding this comment

cryptoe Jul 27, 2023

Choose a reason for hiding this comment

cryptoe Jul 27, 2023

Choose a reason for hiding this comment

cryptoe left a comment

Choose a reason for hiding this comment

cryptoe Aug 1, 2023

Choose a reason for hiding this comment

cryptoe left a comment

Choose a reason for hiding this comment

317brian commented Aug 4, 2023 • edited Loading

cryptoe commented Aug 4, 2023

317brian commented Jul 18, 2023 •

edited

Loading

317brian commented Aug 4, 2023 •

edited

Loading