Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use quotes for integer values within runtime_config.properties #4406

Merged
merged 2 commits into from
Nov 7, 2023

Conversation

dbeatty10
Copy link
Contributor

@dbeatty10 dbeatty10 commented Nov 7, 2023

Preview

What are you changing in this pull request and why?

properties within dataproc_v1.types.RuntimeConfig is a mapping of property names to values, which are used to configure workload execution. Its type is MutableMapping[str, str], but the value isn't currently guaranteed to have a string data type (str).

When dbt-labs/dbt-bigquery#1001 is resolved, it will cast all values to strings.

For dataproc config examples, we want to use quotes around integers values (like spark.executor.instances below) until dbt-labs/dbt-bigquery#1001 is resolved.

myproject:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: oauth
      project: "{{ env_var('GCP_PROJECT') }}"
      dataset: tmp
      threads: 4
      gcs_bucket: "my-bucket"
      dataproc_region: "europe-west1"
      submission_method: serverless
      dataproc_batch:
        runtime_config:
          properties:
            spark.executor.instances: "3"
            spark.driver.memory: 1g
        # ...

Checklist

Copy link

vercel bot commented Nov 7, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
docs-getdbt-com ✅ Ready (Inspect) Visit Preview 💬 Add feedback Nov 7, 2023 1:17am

@github-actions github-actions bot added content Improvements or additions to content size: x-small This change will take under 3 hours to fix. labels Nov 7, 2023
@dbeatty10 dbeatty10 marked this pull request as ready for review November 7, 2023 01:11
@dbeatty10 dbeatty10 requested a review from a team as a code owner November 7, 2023 01:11
Copy link
Contributor

@mirnawong1 mirnawong1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you for opening this up @dbeatty10 ! ✨

@dbeatty10 dbeatty10 merged commit bc7d4a2 into current Nov 7, 2023
7 checks passed
@dbeatty10 dbeatty10 deleted the dbeatty/runtime-config-properties-quotes branch November 7, 2023 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
content Improvements or additions to content size: x-small This change will take under 3 hours to fix.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants