Managed BigQueryIO #31486
Managed BigQueryIO #31486
Conversation
|
R: @chamikaramj |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control |
ahmedabu98
changed the title
ahmedabu98
changed the title
|
R: @chamikaramj |
…hods can instead be done in Dataflow service side
| @@ -36,6 +36,9 @@ dependencies { | |||
| permitUnusedDeclared project(":sdks:java:io:google-cloud-platform") // BEAM-11761 | |||
| implementation project(":sdks:java:extensions:schemaio-expansion-service") | |||
| permitUnusedDeclared project(":sdks:java:extensions:schemaio-expansion-service") // BEAM-11761 | |||
| implementation project(":sdks:java:managed") | |||
| permitUnusedDeclared project(":sdks:java:managed") // BEAM-11761 | |||
There was a problem hiding this comment.
Notes, no action required on this PR:
- This is a link to Jira, so probably there's a github issue it is migrated to
- This should be equivalent to
runtimeOnlybecause it is "implementation" but no static references to it. I would guess this works the same, or else the uberjar plugin might not treat it right. - Putting these deps into a docker container without making an uber jar would honestly be better in the case where it does end up in a container, so we keep the original jar metadata.
...pache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
Outdated
Show resolved
Hide resolved
.../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryFileLoadsWriteSchemaTransformProvider.java
Show resolved
Hide resolved
.../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryFileLoadsWriteSchemaTransformProvider.java
Show resolved
Hide resolved
.../java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryFileLoadsWriteSchemaTransformProvider.java
Outdated
Show resolved
Hide resolved
|
|
||
| if (!Strings.isNullOrEmpty(configuration.getCreateDisposition())) { | ||
| CreateDisposition createDisposition = | ||
| CreateDisposition.valueOf(configuration.getCreateDisposition().toUpperCase()); |
There was a problem hiding this comment.
As a larger point, I think we should do any transform overriding in job submission (BQ modes for batch/streaming etc.) so that we can just upgrade in the backend (at least in the first version).
There was a problem hiding this comment.
Do you mean making this switch in the SDK (ie. construction time)? I assumed we had settled on making it a runner side decision
Some decisions are actually dependent on the runner (e.g. at least one streaming mode in Dataflow)
There was a problem hiding this comment.
Do you mean making this switch in the SDK (ie. construction time)? I assumed we had settled on making it a runner side decision
Yeah. Added some comments to the relavent doc.
No description provided.