The Load component lets you connect to your account and configure a query to load data from the API. You can select the data source, choose the fields to include in your query, and apply filters to refine your results. You can also specify a row limit to control the number of records returned.The results of your query can be loaded into your destination as a new table or can replace an existing table. You can also configure incremental load settings to load only new or updated records since the last load.Your destination can be your connected cloud data warehouse or a cloud storage location. You can also configure advanced settings to control the output encoding, key sanitization strategy, and other options.The Anaplan connector supports full data loads only. Each pipeline run exports the complete dataset from the selected Anaplan export action.If the component requires access to a cloud provider (AWS, Azure, or Google Cloud), it will use the cloud credentials associated with your environment to access resources.To stage data to Azure Blob Storage, the Azure credentials associated with your environment must be assigned the Storage Blob Data Contributor role. For more information, read User assigned with the Storage Blob Data Contributor role.
Your Anaplan password. This is stored as a secret definition. Read Secrets and secret definitions to learn how to store a password as a secret definition.
Login passwords have a 90-day lifespan in Anaplan, and must be reset manually.
An Anaplan workspace. You can find your Workspace ID in the URL of a given model. See the portion of the URL that reads /workspaces/[workspace-id]/ where [workspace-id] is an alphanumeric Workspace ID.
An Anaplan model. You can find your Model ID in the URL of a given model. See the portion of the URL that reads /models/[models-id]/ where [model-id] is an alphanumeric Model ID.
It’s recommended that you format your exports as “Tabular Single or Multi Column CSV” to allow exports to be stored in a cloud data warehouse as a standard table.
Select the destination for your data. This is either in Snowflake as a table or as files in cloud storage.
Snowflake: Load your data into a table in Snowflake. The data must first be staged via Snowflake or a cloud storage solution.
Cloud Storage: Load your data directly into files in your preferred cloud storage location. The format of these files can differ between source systems and will not have a file extension so we suggest inspecting the output to determine the format of the data.
The Snowflake warehouse used to run the queries. The special value [Environment Default] uses the warehouse defined in the environment. Read Overview of Warehouses to learn more.
The Snowflake database to access. The special value [Environment Default] uses the database defined in the environment. Read Databases, Tables and Views - Overview to learn more.
The Snowflake schema. The special value [Environment Default] uses the schema defined in the environment. Read Database, Schema, and Share DDL to learn more.
The name of the table to be created in your Snowflake database. You can use a Table Input component in a transformation pipeline to access and transform this data after it has been loaded.
Define what happens if the table name already exists in the specified Snowflake database and schema.
Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
Select one or more columns to use as the table’s primary key. If you select multiple columns, the component creates a composite primary key using the combination of the selected columns.Leave this blank to create the table without a primary key.
Select the stage access strategy. The strategies available depend on the cloud platform you select in Stage Platform.
Credentials: Connects to the external stage (AWS, Azure) using your configured cloud provider credentials. Not available for Google Cloud Storage.
Storage Integration: Use a Snowflake storage integration to grant access to Snowflake to read data from and write to a cloud storage location. This will reveal the Storage Integration property, through which you can select any of your existing Snowflake storage integrations.
Select the Snowflake internal stage type. Use the Snowflake links provided to learn more about each type of stage.
User: Each Snowflake user has a user stage allocated to them by default for file storage. You may find the user stage convenient if your files will only be accessed by a single user, but need to be copied into multiple tables.
Named: A named stage provides high flexibility for data loading. Users with the appropriate privileges on the stage can load data into any table. Furthermore, because the stage is a database object, any security or access rules that apply to all objects will apply to the named stage.
Named stages can be altered and dropped. User stages cannot.
Select a Databricks Unity Catalog. The special value [Environment Default] uses the catalog defined in the environment. Selecting a catalog will determine which databases are available in the next parameter.
The Databricks schema. The special value [Environment Default] uses the schema defined in the environment. Read Create and manage schemas to learn more.
The name of the table to be created in your Databricks schema. You can use a Table Input component in a transformation pipeline to access and transform this data after it has been loaded.
Define what happens if the table name already exists in the specified Databricks schema.
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
Select the Amazon Redshift schema that will contain your table. The special value [Environment Default] uses the schema defined in the environment. For information about using multiple schemas, read Schemas.
The name of the table to be created in your Amazon Redshift database. You can use a Table Input component in a transformation pipeline to access and transform this data after it has been loaded.
Define what happens if the table name already exists in the specified Amazon Redshift database and schema.
Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
An AWS S3 bucket to stage data into before it is loaded into your Amazon Redshift table. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.
Google BigQuery: Load your data into a table in Google BigQuery.
Cloud Storage: Load your data directly into files in your preferred cloud storage location. The format of these files can differ between source systems and will not have a file extension. Check the output to determine the format of the data.
The name of the table to be created in your Google BigQuery project. You can use a Table Input component in a transformation pipeline to access and transform this data after it has been loaded.
Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
Select one or more columns to use as the table’s primary key. If you select multiple columns, the component creates a composite primary key using the combination of the selected columns.Leave this blank to create the table without a primary key.
Converts common strings that represent null into a null value. This is case-sensitive and works with the following strings: "", “NULL”, “NUL”, “Null”, “null”. The default is No.
This parameter is only applicable when using Snowflake as your destination.
Deactivate soft delete for Azure blobs (Databricks)
If you intend to set your destination as Databricks and your stage platform as Azure Storage, you must turn off the “Enable soft delete for blobs” setting in your Azure account for your pipeline to run successfully. To do this:
In the Azure portal, navigate to your storage account.
In the menu, under Data management, click Data protection.
Clear the Enable soft delete for blobs checkbox. For more information, read Soft delete for blobs.
