Skip to content

Commit

Permalink
docs: Refresh docs for SQL input source (apache#17031)
Browse files Browse the repository at this point in the history
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
  • Loading branch information
vtlim and techdocsmith authored Sep 16, 2024
1 parent 9696f0b commit 2e2f3cf
Show file tree
Hide file tree
Showing 5 changed files with 95 additions and 71 deletions.
4 changes: 2 additions & 2 deletions docs/development/extensions-core/druid-lookups.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,9 @@ This module can be used side to side with other lookup module like the global ca
To use this Apache Druid extension, [include](../../configuration/extensions.md#loading-extensions) `druid-lookups-cached-single` in the extensions load list.

:::info
If using JDBC, you will need to add your database's client JAR files to the extension's directory.
To use JDBC, you must add your database client JAR files to the extension's directory.
For Postgres, the connector JAR is already included.
See the MySQL extension documentation for instructions to obtain [MySQL](./mysql.md#installing-the-mysql-connector-library) or [MariaDB](./mysql.md#alternative-installing-the-mariadb-connector-library) connector libraries.
See the MySQL extension documentation for instructions to obtain [MySQL](./mysql.md#install-mysql-connectorj) or [MariaDB](./mysql.md#install-mariadb-connectorj) connector libraries.
Copy or symlink the downloaded file to `extensions/druid-lookups-cached-single` under the distribution root directory.
:::

Expand Down
73 changes: 47 additions & 26 deletions docs/development/extensions-core/mysql.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
id: mysql
title: "MySQL Metadata Store"
title: "MySQL metadata store"
---

<!--
Expand All @@ -25,41 +25,58 @@ title: "MySQL Metadata Store"

To use this Apache Druid extension, [include](../../configuration/extensions.md#loading-extensions) `mysql-metadata-storage` in the extensions load list.

:::info
The MySQL extension requires the MySQL Connector/J library or MariaDB Connector/J library, neither of which are included in the Druid distribution.
Refer to the following section for instructions on how to install this library.
:::
With the MySQL extension, you can use MySQL as a metadata store or ingest from a MySQL database.

## Installing the MySQL connector library
The extension requires a connector library that's not included with Druid.
See the [Prerequisites](#prerequisites) for installation instructions.

This extension can use Oracle's MySQL JDBC driver which is not included in the Druid distribution. You must
install it separately. There are a few ways to obtain this library:
## Prerequisites

- It can be downloaded from the MySQL site at: https://dev.mysql.com/downloads/connector/j/
- It can be fetched from Maven Central at: https://repo1.maven.org/maven2/com/mysql/mysql-connector-j/8.2.0/mysql-connector-j-8.2.0.jar
- It may be available through your package manager, e.g. as `libmysql-java` on APT for a Debian-based OS
To use the MySQL extension, you need to install one of the following libraries:
* [MySQL Connector/J](#install-mysql-connectorj)
* [MariaDB Connector/J](#install-mariadb-connectorj)

This fetches the MySQL connector JAR file with a name like `mysql-connector-j-8.2.0.jar`.
### Install MySQL Connector/J

Copy or symlink this file inside the folder `extensions/mysql-metadata-storage` under the distribution root directory.
The MySQL extension uses Oracle's MySQL JDBC driver.
The current version of Druid uses version 8.2.0.
Other versions may not work with this extension.

## Alternative: Installing the MariaDB connector library
You can download the library from one of the following sources:

This extension also supports using the MariaDB connector jar, though it is also not included in the Druid distribution, so you must install it separately.
- [MySQL website](https://dev.mysql.com/downloads/connector/j/)
Visit the archives page to access older product versions.
- [Maven Central (direct download)](https://repo1.maven.org/maven2/com/mysql/mysql-connector-j/8.2.0/mysql-connector-j-8.2.0.jar)
- Your package manager. For example, `libmysql-java` on APT for a Debian-based OS.

- Download from the MariaDB site: https://mariadb.com/downloads/connector
- Download from Maven Central: https://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/2.7.3/mariadb-java-client-2.7.3.jar
The download includes the MySQL connector JAR file with a name like `mysql-connector-j-8.2.0.jar`.
Copy or create a symbolic link to this file inside the `lib` folder in the distribution root directory.

This fetches the MariaDB connector JAR file with a name like `maria-java-client-2.7.3.jar`.
### Install MariaDB Connector/J

Copy or symlink this file to `extensions/mysql-metadata-storage` under the distribution root directory.
This extension also supports using the MariaDB connector jar.
The current version of Druid uses version 2.7.3.
Other versions may not work with this extension.

You can download the library from one of the following sources:

- [MariaDB website](https://mariadb.com/downloads/connectors/connectors-data-access/java8-connector)
Click **Show All Files** to access older product versions.
- [Maven Central (direct download)](https://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/2.7.3/mariadb-java-client-2.7.3.jar)

The download includes the MariaDB connector JAR file with a name like `maria-java-client-2.7.3.jar`.
Copy or create a symbolic link to this file inside the `lib` folder in the distribution root directory.

To configure the `mysql-metadata-storage` extension to use the MariaDB connector library instead of MySQL, set `druid.metadata.mysql.driver.driverClassName=org.mariadb.jdbc.Driver`.

Depending on the MariaDB client library version, the connector supports both `jdbc:mysql:` and `jdbc:mariadb:` connection URIs. However, the parameters to configure the connection vary between implementations, so be sure to [check the documentation](https://mariadb.com/kb/en/about-mariadb-connector-j/#connection-strings) for details.
The protocol of the connection string is `jdbc:mysql:` or `jdbc:mariadb:`,
depending on your specific version of the MariaDB client library.
For more information on the parameters to configure a connection,
[see the MariaDB documentation](https://mariadb.com/kb/en/about-mariadb-connector-j/#connection-strings)
for your connector version.


## Setting up MySQL
## Set up MySQL

To avoid issues with upgrades that require schema changes to a large metadata table, consider a MySQL version that supports instant ADD COLUMN semantics. For example, MySQL 8.

Expand Down Expand Up @@ -90,7 +107,7 @@ This extension also supports using MariaDB server, https://mariadb.org/download/
CREATE DATABASE druid DEFAULT CHARACTER SET utf8mb4;
-- create a druid user
CREATE USER 'druid'@'localhost' IDENTIFIED BY 'diurd';
CREATE USER 'druid'@'localhost' IDENTIFIED BY 'password';
-- grant the user all the permissions on the database we just created
GRANT ALL PRIVILEGES ON druid.* TO 'druid'@'localhost';
Expand All @@ -111,10 +128,11 @@ This extension also supports using MariaDB server, https://mariadb.org/download/

If using the MariaDB connector library, set `druid.metadata.mysql.driver.driverClassName=org.mariadb.jdbc.Driver`.

## Encrypting MySQL connections
This extension provides support for encrypting MySQL connections. To get more information about encrypting MySQL connections using TLS/SSL in general, please refer to this [guide](https://dev.mysql.com/doc/refman/5.7/en/using-encrypted-connections.html).
## Encrypt MySQL connections

## Configuration
This extension provides support for encrypting MySQL connections. To get more information about encrypting MySQL connections using TLS/SSL in general, please refer to this [guide](https://dev.mysql.com/doc/refman/5.7/en/using-encrypted-connections.html).

## Configuration properties

|Property|Description|Default|Required|
|--------|-----------|-------|--------|
Expand All @@ -129,7 +147,10 @@ If using the MariaDB connector library, set `druid.metadata.mysql.driver.driverC
|`druid.metadata.mysql.ssl.enabledSSLCipherSuites`|Overrides the existing cipher suites with these cipher suites.|none|no|
|`druid.metadata.mysql.ssl.enabledTLSProtocols`|Overrides the TLS protocols with these protocols.|none|no|

### MySQL InputSource
## MySQL input source

The MySQL extension provides an implementation of an SQL input source to ingest data into Druid from a MySQL database.
For more information on the input source parameters, see [SQL input source](../../ingestion/input-sources.md#sql-input-source).

```json
{
Expand Down
13 changes: 8 additions & 5 deletions docs/development/extensions-core/postgresql.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
id: postgresql
title: "PostgreSQL Metadata Store"
title: "PostgreSQL metadata store"
---

<!--
Expand All @@ -25,7 +25,9 @@ title: "PostgreSQL Metadata Store"

To use this Apache Druid extension, [include](../../configuration/extensions.md#loading-extensions) `postgresql-metadata-storage` in the extensions load list.

## Setting up PostgreSQL
With the PostgreSQL extension, you can use PostgreSQL as a metadata store or ingest from a PostgreSQL database.

## Set up PostgreSQL

To avoid issues with upgrades that require schema changes to a large metadata table, consider a PostgreSQL version that supports instant ADD COLUMN semantics.

Expand Down Expand Up @@ -69,7 +71,7 @@ To avoid issues with upgrades that require schema changes to a large metadata ta
druid.metadata.storage.connector.password=diurd
```

## Configuration
## Configuration properties

In most cases, the configuration options map directly to the [postgres JDBC connection options](https://jdbc.postgresql.org/documentation/use/#connecting-to-the-database).

Expand All @@ -87,9 +89,10 @@ In most cases, the configuration options map directly to the [postgres JDBC conn
| `druid.metadata.postgres.ssl.sslPasswordCallback` | The classname of the SSL password provider. | none | no |
| `druid.metadata.postgres.dbTableSchema` | druid meta table schema | `public` | no |

### PostgreSQL InputSource
## PostgreSQL input source

The PostgreSQL extension provides an implementation of an [SQL input source](../../ingestion/input-sources.md) which can be used to ingest data into Druid from a PostgreSQL database.
The PostgreSQL extension provides an implementation of an SQL input source to ingest data into Druid from a PostgreSQL database.
For more information on the input source parameters, see [SQL input source](../../ingestion/input-sources.md#sql-input-source).

```json
{
Expand Down
Loading

0 comments on commit 2e2f3cf

Please sign in to comment.