Skip to main content
Streaming pipelines can use Google Cloud Storage as a direct storage destination. This page describes the prerequisites, process, and other considerations of using Google Cloud Storage as a Streaming pipeline destination. You should have arrived here by first reading Create a Streaming pipeline.

​
Prerequisites

  • You need a Google Cloud account with a project that has the Cloud Storage API enabled.
  • You need permissions to create and manage Cloud Storage buckets in your Google Cloud project.
  • You need a Google Cloud service account with at least the Storage Object Admin role on the destination bucket, so that the can write objects to it.
  • If your bucket restricts access using VPC Service Controls or firewall rules, you need to allow the ‘s network to reach Cloud Storage.

​
Destination configuration

Refer to this section to complete the Destination configuration section of the Create a Streaming pipeline screen.
​
string
required
Enter the name of the Google Cloud Storage bucket you want to use as a destination. You can find your bucket name in the Google Cloud Console.
This must be the base directory of the Cloud Storage bucket, not a sub-directory.
​
string
required
Enter a prefix to represent the “folder” within the Cloud Storage bucket where all streaming data for this pipeline is saved. Use a unique prefix to avoid naming conflicts and allow the selected bucket to be reused as a destination for multiple pipelines.
This completes the destination configuration. The next groups of properties on this screen are for source setup and pipeline configuration.

​
Schema drift

Schema drift is supported for this destination. Read Schema drift to learn more.