apache · cryptoe · Aug 4, 2023 · Jul 7, 2023 · Jul 12, 2023 · Jul 13, 2023
diff --git a/docs/api-reference/sql-api.md b/docs/api-reference/sql-api.md
@@ -186,4 +186,79 @@ Druid returns an HTTP 404 response in the following cases:
   - `sqlQueryId` is incorrect.
   - The query completes before your cancellation request is processed.
 
-Druid returns an HTTP 403 response for authorization failure.
+Druid returns an HTTP 403 response for authorization failure.
+
+## Query from deep storage
+
+> The `/sql/statements` endpoint used to query from deep storage is currently experimental.
+
+You can use the `sql/statements` endpoint to query segments that exist only in deep storage and are not loaded onto your Historical processes as determined by your load rules. 
+
-Note that at least part of a datasource must be available on a Historical process so that Druid can plan your query.
+Note that at least one segment of a datasource must be available on a Historical process so that the broker can plan your query. A quick way to check this is that data source should be visible on the druid console.
-Note that at least part of a datasource must be available on a Historical process so that Druid can plan your query.
+Note that at least one segment of a datasource must be available on a Historical process so that the broker can plan your query. A quick way to check this is that data source should be visible on the druid console.
+Note that at least part of a datasource must be available on a Historical process so that Druid can plan your query.
+
+For more information, see [Query from deep storage](../querying/query-from-deep-storage.md).
+
+### Submit a query
+
+To run a query from deep storage, send your query to the Router using the POST method:
+
+```
+POST https://ROUTER:8888/druid/v2/sql/statements
+```
+
+Submitting a query from deep storage uses the same syntax as any other Druid SQL query where the "query" field in the JSON object within the request payload contains your query. For example:
+
+```json
+{"query" : "SELECT COUNT(*) FROM data_source WHERE foo = 'bar'"}
+```  
+
+Generally, the `sql` and `sql/statements` endpoints support the same response body fields with minor differences. For general information about the available fields, see [submit a query to the `sql` endpoint](#submit-a-query).
+
+Keep the following in mind when submitting queries to the `sql/statements` endpoint:
+
+- There are additional context parameters  for `sql/statements`: 
+
+   - `executionMode`  determines how query results are fetched. The currently supported mode is `ASYNC`. 
+   - `selectDestination` set to `DURABLE_STORAGE` instructs Druid to write the results from SELECT queries to durable storage. Note that this requires you to have [durable storage for MSQ enabled](../operations/durable-storage.md).
+
+- The only supported results format is JSON.
+- Only the user who submits a query can see the results.
+
+
+### Get query status
+
+```
+GET https://ROUTER:8888/druid/v2/sql/statements/{queryID}
+```
+
+Returns the same response as the post API if the query is accepted or running. The response for a completed query includes the same information as an in-progress query with several additions:
+
+- A `result` object that summarizes information about your results, such as the total number of rows and a sample record
+- A `pages` object that includes the following information for each page of results:
+  -  `numRows`: the number of rows in that page of results
+  - `sizeInBytes`: the size of the page
+  - `id`: the page number that you can use to reference a specific page when you get query results
+
+
+### Get query results
+
+```
+GET https://ROUTER:8888/druid/v2/sql/statements/{queryID}/results?page=PAGENUMBER
+```
+
+Results are separated into pages, so you can use the optional `page` parameter to refine the results you get. When you retrieve the status of a completed query, Druid returns information about the composition of each page and its page number (`id`). 
+
+When getting query results, keep the following in mind:
+
+- JSON is the only supported result format
+- If you attempt to get the results for an in-progress query, Druid returns an error. 
+
+### Cancel a query
+
+```
+DELETE https://ROUTER:8888/druid/v2/sql/statements/{queryID}
+```
+
+Cancels a running or accepted query. 
+
+Druid returns an HTTP 202 response for successful cancelation requests. If the query is already complete or can't be found, Druid returns an HTTP 500 error with an error message describing the issue.
diff --git a/docs/design/architecture.md b/docs/design/architecture.md
@@ -70,12 +70,20 @@ Druid uses deep storage to store any data that has been ingested into the system
 storage accessible by every Druid server. In a clustered deployment, this is typically a distributed object store like S3 or
 HDFS, or a network mounted filesystem. In a single-server deployment, this is typically local disk.
 
-Druid uses deep storage only as a backup of your data and as a way to transfer data in the background between
-Druid processes. Druid stores data in files called _segments_. Historical processes cache data segments on
-local disk and serve queries from that cache as well as from an in-memory cache.
-This means that Druid never needs to access deep storage
-during a query, helping it offer the best query latencies possible. It also means that you must have enough disk space
-both in deep storage and across your Historical servers for the data you plan to load.
+Druid uses deep storage for the following purposes:
+
+- As a backup of your data, including those that get loaded onto Historical processes.
+- As a way to transfer data in the background between
+Druid processes. Druid stores data in files called _segments_. 
+- As the source data for queries that run against segments stored only in deep storage and not in Historical processes as determined by your load rules.
+
+Historical processes cache data segments on
+local disk and serve queries from that cache as well as from an in-memory cache. Segments on disk for Historical processes provide the low latency querying performance Druid is known for. You can query directly from deep storage though, which allows you to query segments that exist only in deep storage. This trades some performance to provide you with the ability to query more of your data without necessarily having to scale your Historical processes.
+
+When determining sizing for your storage, keep the following in mind:
+
+- Deep storage needs to be able to hold all the data that you ingest into Druid
+- On disk storage for Historical processes need to be able to accommodate the data you want to load onto them to run queries on data you access frequently and need low latency for
 
 Deep storage is an important part of Druid's elastic, fault-tolerant design. Druid bootstraps from deep storage even
 if every single data server is lost and re-provisioned.
@@ -210,8 +218,7 @@ available before they are published, since they are only published when the segm
 any additional rows of data.
 2. **Deep storage:** Segment data files are pushed to deep storage once a segment is done being constructed. This
 happens immediately before publishing metadata to the metadata store.
-3. **Availability for querying:** Segments are available for querying on some Druid data server, like a realtime task
-or a Historical process.
+3. **Availability for querying:** Segments are available for querying on some Druid data server, like a realtime task, directly from deep storage, or a Historical process.
 
 You can inspect the state of currently active segments using the Druid SQL
 [`sys.segments` table](../querying/sql-metadata-tables.md#segments-table). It includes the following flags:

diff --git a/docs/design/deep-storage.md b/docs/design/deep-storage.md
@@ -25,7 +25,13 @@ title: "Deep storage"
 
 Deep storage is where segments are stored.  It is a storage mechanism that Apache Druid does not provide.  This deep storage infrastructure defines the level of durability of your data, as long as Druid processes can see this storage infrastructure and get at the segments stored on it, you will not lose data no matter how many Druid nodes you lose.  If segments disappear from this storage layer, then you will lose whatever data those segments represented.
 
-## Local
+In addition to being the backing store for segments, you can use  [query from deep storage](#querying-from-deep-storage) and run queries against segments stored primarily in deep storage. Whether segments exist primarily in deep storage or in deep storage and on Historical processes, is determined by the [load rules](../operations/rule-configuration.md#load-rules) you configure.
+
+## Deep storage options
+
+Druid supports multiple options for deep storage, including blob storage from major cloud providers. Select the one that fits your environment.
+
+### Local
 
 Local storage is intended for use in the following situations:
 
@@ -55,22 +61,28 @@ druid.storage.storageDirectory=/tmp/druid/localStorage
 The `druid.storage.storageDirectory` must be set to a different path than `druid.segmentCache.locations` or
 `druid.segmentCache.infoDir`.
 
-## Amazon S3 or S3-compatible
+### Amazon S3 or S3-compatible
 
 See [`druid-s3-extensions`](../development/extensions-core/s3.md).
 
-## Google Cloud Storage
+### Google Cloud Storage
 
 See [`druid-google-extensions`](../development/extensions-core/google.md).
 
-## Azure Blob Storage
+### Azure Blob Storage
 
 See [`druid-azure-extensions`](../development/extensions-core/azure.md).
 
-## HDFS
+### HDFS
 
 See [druid-hdfs-storage extension documentation](../development/extensions-core/hdfs.md).
 
-## Additional options
+### Additional options
 
 For additional deep storage options, please see our [extensions list](../configuration/extensions.md).
+
+## Querying from deep storage
+
+Although not as performant as querying segments stored on disk for Historicals processes, you can query from deep storage to access segments that you may not need frequently or with the extreme low latency Druid queries traditionally provide. You trade some performance for a total lower storage cost because you can access more of your data without the need to increase the number or capacity of your Historical processes.
+
+For information about how to run queries, see [Query from deep storage](../querying/query-from-deep-storage.md)
diff --git a/docs/multi-stage-query/reference.md b/docs/multi-stage-query/reference.md
@@ -341,41 +341,11 @@ CLUSTERED BY user
 
 The context parameter that sets `sqlJoinAlgorithm` to `sortMerge` is not shown in the above example.
 
-## Durable Storage
+## Durable storage
 
-Using durable storage with your SQL-based ingestion can improve their reliability by writing intermediate files to a storage location temporarily. 
+SQL-based ingestion supports using durable storage to store intermediate files temporarily. Enabling it can improve reliability. For more information, see [Durable storage](../operations/durable-storage.md).
 
-To prevent durable storage from getting filled up with temporary files in case the tasks fail to clean them up, a periodic
-cleaner can be scheduled to clean the directories corresponding to which there isn't a controller task running. It utilizes
-the storage connector to work upon the durable storage. The durable storage location should only be utilized to store the output
-for cluster's MSQ tasks. If the location contains other files or directories, then they will get cleaned up as well.
-
-Enabling durable storage also enables the use of local disk to store temporary files, such as the intermediate files produced
-by the super sorter.  Tasks will use whatever has been configured for their temporary usage as described in [Configuring task storage sizes](../ingestion/tasks.md#configuring-task-storage-sizes)
-If the configured limit is too low, `NotEnoughTemporaryStorageFault` may be thrown.
-
-### Enable durable storage
-
-To enable durable storage, you need to set the following common service properties:
-
-```
-druid.msq.intermediate.storage.enable=true
-druid.msq.intermediate.storage.type=s3
-druid.msq.intermediate.storage.bucket=YOUR_BUCKET
-druid.msq.intermediate.storage.prefix=YOUR_PREFIX
-druid.msq.intermediate.storage.tempDir=/path/to/your/temp/dir
-```
-
-For detailed information about the settings related to durable storage, see [Durable storage configurations](#durable-storage-configurations).
-
-
-### Use durable storage for queries
-
-When you run a query, include the context parameter `durableShuffleStorage` and set it to `true`.
-
-For queries where you want to use fault tolerance for workers,  set `faultTolerance` to `true`, which automatically sets `durableShuffleStorage` to `true`.
-
-## Durable storage configurations
+### Durable storage configurations
 
 The following common service properties control how durable storage behaves:
 

diff --git a/docs/operations/durable-storage.md b/docs/operations/durable-storage.md
@@ -0,0 +1,47 @@
+---
+id: durable-storage
+title: "Durable storage for the multi-stage query engine"
+sidebar_label: "Durable storage"
+---
+
+You can use durable storage to improve querying from deep storage and SQL-based ingestion.
+
+> Note that only S3 is supported as a durable storage location.
+
+Durable storage for queries from deep storage provides a location where you can write the results of deep storage queries to. Durable storage for SQL-based ingestion is used to temporarily house intermediate files, which can improve reliability.
+
+## Enable durable storage
+
+To enable durable storage, you need to set the following common service properties:
+
+```
+druid.msq.intermediate.storage.enable=true
+druid.msq.intermediate.storage.type=s3
+druid.msq.intermediate.storage.bucket=YOUR_BUCKET
+druid.msq.intermediate.storage.prefix=YOUR_PREFIX
+druid.msq.intermediate.storage.tempDir=/path/to/your/temp/dir
+```
+
+For detailed information about the settings related to durable storage, see [Durable storage configurations](../multi-stage-query/reference.md#durable-storage-configurations).
+
+
+## Use durable storage for SQL-based ingestion queries
+
+When you run a query, include the context parameter `durableShuffleStorage` and set it to `true`.
+
+For queries where you want to use fault tolerance for workers,  set `faultTolerance` to `true`, which automatically sets `durableShuffleStorage` to `true`.
+
+## Use durable storage for queries from deep storage
+
+When you run a query, include the context parameter `selectDestination` and set it to `DURABLE_STORAGE`. This context parameter configures queries from deep storage to write their results to durable storage.
+
+## Durable storage clean up
+
+To prevent durable storage from getting filled up with temporary files in case the tasks fail to clean them up, a periodic
+cleaner can be scheduled to clean the directories corresponding to which there isn't a controller task running. It utilizes
+the storage connector to work upon the durable storage. The durable storage location should only be utilized to store the output
+for cluster's MSQ tasks. If the location contains other files or directories, then they will get cleaned up as well.
+
+Enabling durable storage also enables the use of local disk to store temporary files, such as the intermediate files produced
+by the super sorter.  Tasks will use whatever has been configured for their temporary usage as described in [Configuring task storage sizes](../ingestion/tasks.md#configuring-task-storage-sizes)
+If the configured limit is too low, `NotEnoughTemporaryStorageFault` may be thrown.
diff --git a/docs/operations/rule-configuration.md b/docs/operations/rule-configuration.md
@@ -107,7 +107,7 @@ In the web console you can use the up and down arrows on the right side of the i
 
 ## Load rules
 
-Load rules define how Druid assigns segments to [historical process tiers](./mixed-workloads.md#historical-tiering), and how many replicas of a segment exist in each tier.
+Load rules define how Druid assigns segments to [Historical process tiers](./mixed-workloads.md#historical-tiering), and how many replicas of a segment exist in each tier.
 
 If you have a single tier, Druid automatically names the tier `_default`. If you define an additional tier, you must define a load rule to specify which segments to load on that tier. Until you define a load rule, your new tier remains empty.
 
@@ -167,7 +167,7 @@ Set the following properties:
   - the segment interval starts any time after the rule interval starts.
 
   You can use this property to load segments with future start and end dates, where "future" is relative to the time when the Coordinator evaluates data against the rule. Defaults to `true`.
-- `tieredReplicants`: a map of tier names to the number of segment replicas for that tier.
+- `tieredReplicants`: a map of tier names to the number of segment replicas for that tier. If you set the replicants for a period to 0 on all tiers, you can still [query the data from deep storage](../querying/query-from-deep-storage.md)
 - `useDefaultTierForNull`: This parameter determines the default value of `tieredReplicants` and only has an effect if the field is not present. The default value of `useDefaultTierForNull` is true.
 
 ### Interval load rule
@@ -190,7 +190,7 @@ Interval load rules have type `loadByInterval`. The following example places one
 Set the following properties:
 
 - `interval`: the load interval specified as an [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) range encoded as a string.
-- `tieredReplicants`: a map of tier names to the number of segment replicas for that tier.
+- `tieredReplicants`: a map of tier names to the number of segment replicas for that tier. If you set the replicants for an interval to 0 on all tiers, you can still [query the data from deep storage](../querying/query-from-deep-storage.md). 
 - `useDefaultTierForNull`: This parameter determines the default value of `tieredReplicants` and only has an effect if the field is not present. The default value of `useDefaultTierForNull` is true.
 
 ## Drop rules
@@ -256,7 +256,7 @@ Set the following property:
 
 ### Interval drop rule
 
-You can use a drop interval rule to prevent Druid from loading a specified range of data onto any tier. The range is typically your oldest data. The dropped data resides in cold storage, but is not queryable. If you need to query the data, update or remove the interval drop rule so that Druid reloads the data.
+You can use a drop interval rule to prevent Druid from loading a specified range of data onto any tier. The range is typically your oldest data. The dropped data resides in deep storage and can still be [queried from deep storage](../querying/query-from-deep-storage.md). 
 
 Interval drop rules have type `dropByInterval` and the following JSON structure: