Skip to content

Custom Openmetadata connector to connect data from Dremio to Openmetadata.

License

Notifications You must be signed in to change notification settings

TIKI-Institut/openmetadata-dremio-connector

Repository files navigation

openmetadata-dremio-connector

Custom Openmetadata connector to connect data from Dremio to Openmetadata.

This connector uses the sqlalchemy-dremio package to establish a conncetion to Dremio over the arrow flight API.

Static Badge Static Badge Static Badge Static Badge Static Badge Static Badge Static Badge Static Badge Static Badge Static Badge

Configuration

This Connector is currently implemented as CustomDatabase connector and therefore the only way to configure any connection parameters is using the connectionOptions.

connectionOptions:
  hostPort: <host>:<arrowFlightPort>
  username: <username>
  password: <password>
  # optional
  UseEncryption: False
  disableCertificateVerification: True

Local Dev Stack

Requirements:

  • Access to a Dremio instance

Steps:

  1. Setup python venv and install requirements
  2. Start openmetadata local stack
    make local-openmetadata-stack 
  3. Create workflow.dremio.yaml as a copy from workflow.dremio.template.yaml
  4. Configure workflow.dremio.yaml. Mainly editing connection credentials to Dremio and the jwtToken for OpenMetadata (e.g. from the ingestion bot)
  5. Run / Debug ingestion
    python.exe -m metadata ingest -c ./workflow.dremio.yaml
    Or use the provided Run configuration for IntelliJ

About

Custom Openmetadata connector to connect data from Dremio to Openmetadata.

Resources

License

Stars

Watchers

Forks

Packages

No packages published