Skip to content

Uses Cloud Build to deploy a scalable batch ingestion pipeline consisting of GCS, Cloud Functions, Dataflow and BigQuery

License

Notifications You must be signed in to change notification settings

polleyg/gcp-batch-ingestion-bigquery

Repository files navigation

Uses Cloud Build to deploy a scalable batch ingestion pipeline consisting of GCS, Cloud Functions, Dataflow and BigQuery

See cloudbuild.yaml for an idea of what this CI/CD pipeline does. Essentially it runs a container for each step of the build using Cloud Build and deploys each component of the pipeline to GCP. It also uses Terraform to create the initial buckets for deploying all the binaries to. See the infra.tf file for all that nonsense.

Here's what it looks like:

screen shot 2018-09-04 at 5 33 17 pm

About

Uses Cloud Build to deploy a scalable batch ingestion pipeline consisting of GCS, Cloud Functions, Dataflow and BigQuery

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published