Skip to content

Conversation

emtwo
Copy link

@emtwo emtwo commented Mar 18, 2019

Opening a new PR for #3291:

What type of PR is this? (check all applicable)

  • Feature

Description

This PR is the first of a series of PRs for schema enhancements.

The way that schema updates work in this PR is through a periodic celery task that runs the queries to get column names and types etc. A separate periodic task runs to get samples of data for each table. The results are stored in the new schema tables. Whenever the schema is fetched from the UI, it just directly queries the data in these tables. If a refresh is forced by a user, the celery task that computes column names and types runs synchronously and a response is given after the tables have been updated.

Here's what the new schema response from query runner looks like. Basically we keep the columns field and add an optional additional metadata field):

{
  <table_name1>: {
    'columns': [<col1_name>, <col2_name> ...],
    'metadata': [{
      "name": <col1_name>,
      "type": <col1_type>
    }, ...]
  },
  <table_name2>: { ... },
  ...
}

Related Tickets & Documents

[1] Schema viewer drawer #3291 (this one)
[2] Schema admin configuration #3292
[3] Schema query samples #3293
[4] Data source descriptions #3401

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

Here are a couple of screenshots. (note the new "Show data samples" option in the data source page.)
screen shot 2019-02-27 at 1 51 01 pm
screen shot 2019-02-27 at 1 51 30 pm

@guidopetri
Copy link
Contributor

@emtwo , thanks for the PR! We've updated a lot of things now that we're Community-driven so - if you're still interested in getting this merged - would you mind rebasing off master to re-run the CI, as well as updating merge conflicts?

We're trying to clean up our PR todo list, so if you're not interested, that's fine - we'll close the PR in about a week if we don't hear back. If you're interested in reopening the PR afterwards, we would also very much welcome that.

@justinclift
Copy link
Member

Closing this, as there seems to be no interest in getting it merged. 😦

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants