Include statement attributes in EXPLAIN PLAN output

[abhishekrb19]

This PR adds attributes that contain metadata information about the query in the EXPLAIN PLAN output. The attributes currently contain two items:

• statementKind: denotes the overall statement type -- possible values SELECT, INSERT, REPLACE
• targetDataSource: provides the target datasource name for DML statements

A new method explainAttributes, is added to the SqlStatementHandler interface, and the Select, Insert, and Replace handlers implement the methods. The value is then obtained from the statement handler during query planning and set in the planner context.

For backward compatibility, these attributes are included as a third column to the output—updated docs to reflect the output changes. Also, it is added to both the legacy and native query plan outputs.

The EXPLAIN PLAN output for a SELECT query is in the README. Below is the output for an INSERT query:

[
  [
    {
      "query": {
        "queryType": "scan",
        "dataSource": {
          "type": "external",
          "inputSource": {
            "type": "s3",
            "uris": [
              "obfuscated-uri"
            ]
          },
          "inputFormat": {
            "type": "json",
            "keepNullColumns": false,
            "assumeNewlineDelimited": false,
            "useJsonNodeReader": false
          },
          "signature": [
            {
              "name": "time2",
              "type": "STRING"
            },
            {
              "name": "channel",
              "type": "STRING"
            },
            {
              "name": "countryName",
              "type": "STRING"
            }
          ]
        },
        "intervals": {
          "type": "intervals",
          "intervals": [
            "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
          ]
        },
        "virtualColumns": [
          {
            "type": "expression",
            "name": "v0",
            "expression": "timestamp_parse(\"time2\",null,'UTC')",
            "outputType": "LONG"
          },
          {
            "type": "expression",
            "name": "v1",
            "expression": "'Japan'",
            "outputType": "STRING"
          }
        ],
        "resultFormat": "compactedList",
        "orderBy": [
          {
            "columnName": "v1",
            "order": "ascending"
          },
          {
            "columnName": "channel",
            "order": "ascending"
          }
        ],
        "filter": {
          "type": "selector",
          "dimension": "countryName",
          "value": "Japan"
        },
        "columns": [
          "channel",
          "time2",
          "v0",
          "v1"
        ],
        "legacy": false,
        "context": {
          "finalizeAggregations": true,
          "groupByEnableMultiValueUnnesting": true,
          "maxNumTasks": 2,
          "queryId": "27058ca9-47be-4d7d-a4c6-580403209801",
          "scanSignature": "[{\"name\":\"channel\",\"type\":\"STRING\"},{\"name\":\"time2\",\"type\":\"STRING\"},{\"name\":\"v0\",\"type\":\"LONG\"},{\"name\":\"v1\",\"type\":\"STRING\"}]",
          "sqlInsertSegmentGranularity": "\"DAY\"",
          "sqlQueryId": "27058ca9-47be-4d7d-a4c6-580403209801",
          "useNativeQueryExplain": true
        },
        "granularity": {
          "type": "all"
        }
      },
      "signature": [
        {
          "name": "v0",
          "type": "LONG"
        },
        {
          "name": "time2",
          "type": "STRING"
        },
        {
          "name": "channel",
          "type": "STRING"
        },
        {
          "name": "v1",
          "type": "STRING"
        }
      ]
    }
  ],
  [
    {
      "name": "my_ds",
      "type": "DATASOURCE"
    },
    {
      "name": "EXTERNAL",
      "type": "EXTERNAL"
    }
  ],
  {
    "statementKind": "INSERT",
    "targetDataSource": "my_ds"
  }
]

This PR has:

• been self-reviewed.
  • using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
• added documentation for new or modified features or behaviors.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added or updated version, license, or notice information in licenses.yaml
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[zachjsh]

Nice test!

[zachjsh]

Seems like the explain plan output would look something like this:

[
  [
    {
      "query": {
        "queryType": "scan",
        "dataSource": {
          "type": "table",
          "name": "foo"
        },
        "intervals": {
          "type": "intervals",
          "intervals": [
            "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
          ]
        },
        "resultFormat": "compactedList",
        "columns": [
          "dim1"
        ],
        "legacy": false,
        "context": {
          "defaultTimeout": 300000,
          "maxScatterGatherBytes": 9223372036854776000,
          "sqlCurrentTimestamp": "2000-01-01T00:00:00Z",
          "sqlQueryId": "dummy",
          "vectorize": "false",
          "vectorizeVirtualColumns": "false"
        },
        "granularity": {
          "type": "all"
        }
      },
      "signature": [
        {
          "name": "dim1",
          "type": "STRING"
        }
      ]
    },
    {
      "query": {
        "queryType": "scan",
        "dataSource": {
          "type": "table",
          "name": "foo"
        },
        "intervals": {
          "type": "intervals",
          "intervals": [
            "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
          ]
        },
        "resultFormat": "compactedList",
        "filter": {
          "type": "selector",
          "dimension": "dim1",
          "value": "42"
        },
        "columns": [
          "dim1"
        ],
        "legacy": false,
        "context": {
          "defaultTimeout": 300000,
          "maxScatterGatherBytes": 9223372036854776000,
          "sqlCurrentTimestamp": "2000-01-01T00:00:00Z",
          "sqlQueryId": "dummy",
          "vectorize": "false",
          "vectorizeVirtualColumns": "false"
        },
        "granularity": {
          "type": "all"
        }
      },
      "signature": [
        {
          "name": "dim1",
          "type": "STRING"
        }
      ]
    },
    {
      "query": {
        "queryType": "scan",
        "dataSource": {
          "type": "table",
          "name": "foo"
        },
        "intervals": {
          "type": "intervals",
          "intervals": [
            "-146136543-09-08T08:23:32.096Z/146140482-04-24T15:36:27.903Z"
          ]
        },
        "resultFormat": "compactedList",
        "filter": {
          "type": "selector",
          "dimension": "dim1",
          "value": "44"
        },
        "columns": [
          "dim1"
        ],
        "legacy": false,
        "context": {
          "defaultTimeout": 300000,
          "maxScatterGatherBytes": 9223372036854776000,
          "sqlCurrentTimestamp": "2000-01-01T00:00:00Z",
          "sqlQueryId": "dummy",
          "vectorize": "false",
          "vectorizeVirtualColumns": "false"
        },
        "granularity": {
          "type": "all"
        }
      },
      "signature": [
        {
          "name": "dim1",
          "type": "STRING"
        }
      ]
    },
    {
      "statementKind": "SELECT"
    },
    {
      "targetDataSource": "null"
    }
  ],
  [
    {
      "name": "foo",
      "type": "DATASOURCE"
    }
  ]
]

I'm wondering if the targetDataSource and statementKind, should both go into some common metadata section, instead of being in separate objects by themselves within the returned plan

[gianm]

Mainly reviewed the API design; I just skimmed the code. For the API, a few comments:

• For best backwards compat with existing clients, we should add the new info as a third column in the EXPLAIN PLAN result rather than adding it to the queries array.
• We may want to add more statement-related attributes to EXPLAIN PLAN in the future (for example: partitioning for INSERT / REPLACE, or time chunks for REPLACE). So I suggest wrapping all of these into a model object that we can add more fields to over time.
• This is a change to a user-facing API, so it needs a documentation update. The relevant docs are querying/sql-translation.md and querying/sql.md.
• There's also some doc debt on querying/sql.md, as it doesn't mention the new EXPLAIN PLAN format, and has a warning box that seems only relevant to the legacy format. If you fix this in this PR, great, if not please raise a follow-up issue and tag it with Documentation.

[abhishekrb19]

@gianm @zachjsh thanks for the reviews. Can you please take another look when you get a chance?

[zachjsh]

LGTM

[kfaraz]

Overall LGTM, left some minor comments.