Skip to main content
Production use of this feature is only available on specific Maia plans. Contact our Sales team for more information.
The Vector Search transformation component uses the Google BigQuery VECTOR_SEARCH function to find the rows of a base table that are nearest to each row of your input data, by comparing vector embeddings. The search runs inside Google BigQuery. The component takes a single input, with the embeddings to search for in one of its columns. The base table, which contains the data you’re searching, isn’t connected as an input—you select it in the component’s properties. Both embedding columns must contain ARRAY<FLOAT64> values. You can convert text into embeddings of this type using the Embed component, and use the output from that component as an input into Vector Search. For each input row, the component outputs up to as many of the nearest base-table rows as asked for in the Top K property, with the distance between each pair of embeddings. To learn more about vector search in Google BigQuery, read Search embeddings with vector search.

​
Use case

Typical use cases for a vector search include the following:
  • Performing a semantic text search to return the most contextually relevant documents, even if they don’t share exact keywords.
  • Personalizing content retrieval by matching users to relevant content based on their interests or behavior embeddings.
  • Powering support systems by finding the closest pre-written response or FAQ entry for a customer’s question.

​
Properties

​
string
required
A human-readable name for the component.
​
drop-down
required
The Google Cloud project that owns the BigQuery dataset. The special value [Environment Default] uses the Google Cloud project defined in the environment. For more information, read Creating projects.
​
drop-down
required
The Google BigQuery dataset containing the table to query. The special value [Environment Default] uses the dataset defined in the environment.
​
drop-down
required
The table that contains the data to be searched. The results for each input row are rows of this table.
​
drop-down
required
Select the column of the Base table that contains the embeddings you want to search. This column isn’t included in the output.
​
drop-down
required
Select the input column that contains the embeddings to search for. This column isn’t included in the output.
​
integer
required
The number of nearest base-table rows to return for each input row. The value must be 1 or more. The default is 5.
​
drop-down
The measure of distance between embeddings that the search uses. The following options are available:
  • Cosine
  • Euclidean
  • Dot Product
If you leave this blank and the search uses a vector index on the base table, the index’s distance type is used. Otherwise, Euclidean is used.
​
JSON
A JSON object of options for the search. You can set the fraction_lists_to_search and use_brute_force fields of the VECTOR_SEARCH options argument. The JSON must be well-formed.
  • fraction_lists_to_search: A number between 0 and 1 (exclusive). It’s only used when the search uses a vector index, and you can’t set it when use_brute_force is true.
  • use_brute_force: Set to true to skip the vector index and find the exact nearest neighbors.
For example:
​
string
The prefix added to the name of each base-table column in the output, to keep them apart from the input columns. The default is base_. Leave this blank to output the base-table columns under their own names.Each prefixed name must be different from the names of the input columns, ignoring case. If an input column and a base-table column have the same name, set a prefix.
​
string
required
The name of the column that holds the distance for each match. The default is distance.The name must be different from the names of the input columns and the prefixed base-table columns, ignoring case.

​
Explanation of output

The output contains one row for each match. Each input row returns up to Top K rows, one for each of its nearest base-table rows. The order of the output rows isn’t guaranteed. Each output row contains the input columns except Query embedding column, followed by the base-table columns except Base embedding column, each with Base column prefix added to its name. The last column, named by Distance column name, holds the distance that Google BigQuery computes between the input row’s embedding and the base-table row’s embedding.
The embeddings must have no NULL elements, and all embeddings in both columns must have the same number of dimensions. If even one input row’s embedding doesn’t meet the requirements of the VECTOR_SEARCH function, Google BigQuery fails the whole query, so the run fails and no rows are output, including those for valid input rows.