Schema Viewer Drawer

[emtwo]

Opening a new PR for #3291:

What type of PR is this? (check all applicable)

• Feature

Description

This PR is the first of a series of PRs for schema enhancements.

The way that schema updates work in this PR is through a periodic celery task that runs the queries to get column names and types etc. A separate periodic task runs to get samples of data for each table. The results are stored in the new schema tables. Whenever the schema is fetched from the UI, it just directly queries the data in these tables. If a refresh is forced by a user, the celery task that computes column names and types runs synchronously and a response is given after the tables have been updated.

Here's what the new schema response from query runner looks like. Basically we keep the columns field and add an optional additional metadata field):

{
  <table_name1>: {
    'columns': [<col1_name>, <col2_name> ...],
    'metadata': [{
      "name": <col1_name>,
      "type": <col1_type>
    }, ...]
  },
  <table_name2>: { ... },
  ...
}

Related Tickets & Documents

[1] Schema viewer drawer #3291 (this one)
[2] Schema admin configuration #3292
[3] Schema query samples #3293
[4] Data source descriptions #3401

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

Here are a couple of screenshots. (note the new "Show data samples" option in the data source page.)

[guidopetri]

@emtwo , thanks for the PR! We've updated a lot of things now that we're Community-driven so - if you're still interested in getting this merged - would you mind rebasing off master to re-run the CI, as well as updating merge conflicts?

We're trying to clean up our PR todo list, so if you're not interested, that's fine - we'll close the PR in about a week if we don't hear back. If you're interested in reopening the PR afterwards, we would also very much welcome that.

[justinclift]

Closing this, as there seems to be no interest in getting it merged. 😦