Dataproc Spark Connect Client

A wrapper of the Apache Spark Connect client with additional functionalities that allow applications to communicate with a remote Dataproc Spark cluster using the Spark Connect protocol without requiring additional steps.

Install

pip install dataproc_spark_connect

Uninstall

pip uninstall dataproc_spark_connect

Setup

This client requires permissions to manage Dataproc sessions and session templates. If you are running the client outside of Google Cloud, you must set following environment variables:

• GOOGLE_CLOUD_PROJECT - The Google Cloud project you use to run Spark workloads
• GOOGLE_CLOUD_REGION - The Compute Engine region where you run the Spark workload.
• GOOGLE_APPLICATION_CREDENTIALS - Your Application Credentials
• DATAPROC_SPARK_CONNECT_SESSION_DEFAULT_CONFIG (Optional) - The config location, such as tests/integration/resources/session.textproto

Usage

1. Install the latest version of Dataproc Python client and Dataproc Spark Connect modules:

pip install google_cloud_dataproc --force-reinstall
pip install dataproc_spark_connect --force-reinstall

2. Add the required import into your PySpark application or notebook:

from google.cloud.dataproc_spark_connect import DataprocSparkSession

3. There are two ways to create a spark session,

   1. Start a Spark session using properties defined in DATAPROC_SPARK_CONNECT_SESSION_DEFAULT_CONFIG:

      spark = DataprocSparkSession.builder.getOrCreate()

   2. Start a Spark session with the following code instead of using a config file:

      from google.cloud.dataproc_v1 import SparkConnectConfig
      from google.cloud.dataproc_v1 import Session
      dataproc_config = Session()
      dataproc_config.spark_connect_session = SparkConnectConfig()
      dataproc_config.environment_config.execution_config.subnetwork_uri = "<subnet>"
      dataproc_config.runtime_config.version = '3.0'
      spark = DataprocSparkSession.builder.dataprocConfig(dataproc_config).getOrCreate()

Billing

As this client runs the spark workload on Dataproc, your project will be billed as per Dataproc Serverless Pricing. This will happen even if you are running the client from a non-GCE instance.

Contributing

Building and Deploying SDK

1. Install the requirements in virtual environment.

pip install -r requirements.txt

2. Build the code.

python setup.py sdist bdist_wheel

2. Copy the generated .whl file to Cloud Storage. Use the version specified in the setup.py file.

VERSION=<version> gsutil cp dist/dataproc_spark_connect-${VERSION}-py2.py3-none-any.whl gs://<your_bucket_name>

3. Download the new SDK on Vertex, then uninstall the old version and install the new one.

%%bash
export VERSION=<version>
gsutil cp gs://<your_bucket_name>/dataproc_spark_connect-${VERSION}-py2.py3-none-any.whl .
yes | pip uninstall dataproc_spark_connect
pip install dataproc_spark_connect-${VERSION}-py2.py3-none-any.whl