Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Maps] Add mvt support for ES doc sources #75698

Merged
merged 57 commits into from
Sep 1, 2020
Merged

[Maps] Add mvt support for ES doc sources #75698

merged 57 commits into from
Sep 1, 2020

Conversation

thomasneirynck
Copy link
Contributor

WIP

Partially addresses #58519

Summary

This adds an .mvt vector tile option for Elasticsearch geo_shape documents.

This allows display of a large amount of documents. The display of the geometries is optimized for scale. When zoomed in, the full geometry is returned, when zoomed out, a simplified geometry is shown.

image
`

Introduce a new scaling-type "mvt tiles"

image

Extend ESSearchSource to implement

This is different from the original POC (#57248), which introduced a new source type altogether. There are some advantages to just extending the existing ESSearchSource.

  • less code overall
  • toggling scaling-type can re-use the same infrastructure which was in place for blended-layers, without having to accommodate "source-type-swapping"

Add new getTile route-backend

The URL encodes all relevant state. By doing so, the existing logic in TiledVectorLayer that determines when and how to swap mb-sources remains functional.

Introduce the concept of a too many features bounds

When the number of features exceeds the ES-limit, the tile contains a bounds of where the features are.

image

This is somewhat a "one-off"-addition, but imho it is relevant for the .mvt scaling type. This bounds indicates the user where the data is, but does not show the actual geometries. This is a similar approaches exist in other toolkits. The goal is to give users an indication of where to zoom-in.

e.g. consider building footprints for an entire US-state. There are too many to display at once (in the millions), and even if Maps could display them all at once, at zoomed out scales they would be much smaller than a pixel to see.

This needs design work. The current hash-pattern is somewhat ugly.

Future work

For feature completeness, following functionality needs to be added: (excluded from this PR, to keep changes to a minimum).

  • add support for geo_point
  • add support for grid/cluster layers as tiles

Todo

  • unit tests for the new contracts and UX
  • API integration tests for the new route
  • functional tests for round-trip verification
  • ts type check
  • Language
  • Known bugs
    • should only include necessary field-properties
    • time based data? allow/disallow?

@thomasneirynck
mvt init
bd4f39b
@thomasneirynck
cleanup
4da768c
@thomasneirynck
typing
aae411f 
typing
@thomasneirynck
update snapshot
a3fbf97
@thomasneirynck
Enable mvt only for geo_shape
828ce8a
@thomasneirynck
fix ts
02e7688
@thomasneirynck
Fix snapshot
058c116
@thomasneirynck
update functional tests
43efc7c
@thomasneirynck
Merge branch 'master' of github.com:elastic/kibana into maps/es_mvt
4c238c7
@thomasneirynck
feedback
76bdfcc 
feedback
@thomasneirynck
fix snapshot
2371e7b
@thomasneirynck
disable client-side metadata retrieval
9769159
@thomasneirynck
fix typos
3a5a642
@thomasneirynck
take into account url-state
aa3e6e1
@thomasneirynck
Merge branch 'master' of github.com:elastic/kibana into maps/es_mvt
318e2e8
@thomasneirynck
add unit test
d011799
@thomasneirynck
add test stub
4839327
@thomasneirynck
add functional test
d1cb30d
@thomasneirynck
add getTile unit test boilerplate
f611360
@thomasneirynck
fix import
abe3188
@thomasneirynck
exclude bbox-features from tooltips
c73faef
@thomasneirynck
only include required fields
919024a
@thomasneirynck
Merge branch 'master' of github.com:elastic/kibana into maps/es_mvt
0010eae
@thomasneirynck
simplify test
6e2455f
@thomasneirynck
import
38e3a54 
reuse utility

re-enable dynamic styling

update tile

ts fixes
@thomasneirynck
wording feedback
4df20b4
@thomasneirynck
update snap
b140223 
add unit tests

functional boiler plate

fine tune

remove cruft
@thomasneirynck thomasneirynck requested a review from a team as a code owner August 21, 2020 19:45
@thomasneirynck
Copy link
Contributor Author

This replaces #74319

@thomasneirynck thomasneirynck mentioned this pull request Aug 21, 2020
Closed
16 tasks
nreese
nreese approved these changes Aug 31, 2020
View reviewed changes
Copy link
Contributor

@nreese nreese left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great feature. Can't wait until vector tiles are enabled by default
LGTM
code review, tested in chrome

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

async chunks size

id value diff baseline
maps 3.3MB +9.2KB 3.3MB

page load bundle size

id value diff baseline
maps 297.8KB +853.0B 297.0KB

distributable file count

id value diff baseline
total 45519 +62 45457

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@thomasneirynck thomasneirynck merged commit d7869de into elastic:master Sep 1, 2020
This was referenced Sep 1, 2020
Closed
Merged
thomasneirynck added a commit to thomasneirynck/kibana that referenced this pull request Sep 1, 2020
@thomasneirynck
[Maps] Add mvt support for ES doc sources (elastic#75698)
59de0dd
thomasneirynck added a commit that referenced this pull request Sep 1, 2020
@thomasneirynck
[Maps] Add mvt support for ES doc sources (#75698) (#76424)
f8d88c8
gmmorris added a commit to gmmorris/kibana that referenced this pull request Sep 2, 2020
@gmmorris
Merge branch 'master' into actions/webhook-remove-header
c9deb85 
* master: (223 commits)
  skip flaky suite (elastic#75724)
  [Reporting] Add functional test for Reports in non-default spaces (elastic#76053)
  [Enterprise Search] Fix various icons in dark mode (elastic#76430)
  skip flaky suite (elastic#76245)
  Add `auto` interval to histogram AggConfig (elastic#76001)
  [Resolver] generator uses setup_node_env (elastic#76422)
  [Ingest Manager] Support both zip & tar archives from Registry (elastic#76197)
  [Ingest Manager] Improve agent vs kibana version checks (elastic#76238)
  Manually building `KueryNode` for Fleet's routes (elastic#75693)
  remove dupe tinymath section (elastic#76093)
  Create APM issue template (elastic#76362)
  Delete unused file. (elastic#76386)
  [SECURITY_SOLUTION][ENDPOINT] Trusted Apps Create API (elastic#76178)
  [Detections Engine] Add Alert actions to the Timeline (elastic#73228)
  [Dashboard First] Library Notification (elastic#76122)
  [Maps] Add mvt support for ES doc sources  (elastic#75698)
  Add setHeaderActionMenu API to AppMountParameters (elastic#75422)
  [ML] Remove "Are you sure" from data frame analytics jobs (elastic#76214)
  [yarn] remove typings-tester, use @ts-expect-error (elastic#76341)
  [Reporting/CSV] Do not fail the job if scroll ID can not be cleared (elastic#76014)
  ...
@florinvirdol florinvirdol mentioned this pull request Mar 23, 2021
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nreese nreese nreese approved these changes

@jsanz jsanz Awaiting requested review from jsanz

@kindsun kindsun Awaiting requested review from kindsun

Assignees
No one assigned
Labels
[Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation release_note:enhancement v7.10.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@thomasneirynck @elasticmachine @jsanz @nreese @kibanamachine