# Copyright 2026 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
Text + multimodal embedding generation and vector search in BigQuery with BigFrames#
Run in Colab
|
|
|
|
Share to:
Overview#
Objectives#
In this tutorial, you’ll build a semantic search utility for a fictional e-commerce pet store using BigQuery, BigQuery DataFrames (BigFrames), and Gemini Enterprise Agent Platform. Unlike simple keyword matching, semantic search uses vector embeddings to capture the conceptual meaning of both product descriptions and product images. This enables customers to find products based on intent and visual attributes, creating a much more intuitive shopping experience.
By the end of this tutorial, you will know how to:
Load data directly from Google Cloud Storage into a BigFrames DataFrame using
bpd.read_parquet.Generate text embeddings on DataFrame columns using
bigframes.bigquery.ai.embed.Perform text-to-text vector search with
bigframes.bigquery.vector_searchto find products with semantically similar descriptions.Work with unstructured image data using BigFrames’ ObjectRef functions (
bbq.obj.make_ref,bbq.obj.fetch_metadata,bbq.obj.get_access_url) and generate multimodal embeddings.Perform text-to-image vector search to find product images that visually match a natural language text query.
All seamlessly orchestrated using Python and BigFrames!
Services and Costs#
This tutorial uses the following billable components of Google Cloud:
You can use the Pricing Calculator to generate a cost estimate based on your projected usage.
Before you begin#
Set up your Google Cloud project#
The following steps are required, regardless of your notebook environment.
Select or create a Google Cloud project. When you first create an account, you get a $300 free credit towards your compute/storage costs.
Enable the BigQuery, BigQuery Connection, and Agent Platform APIs.
If you are running this notebook locally, you need to install the Cloud SDK.
Set your project ID#
PROJECT_ID = "" # @param {type:"string"}
# Set the project id
! gcloud config set project {PROJECT_ID}
Authenticate to your Google Cloud account#
Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below.
1. Colab Enterprise or BigQuery Studio Notebooks
Do nothing as you are already authenticated.
2. Colab, uncomment and run:
# from google.colab import auth
# auth.authenticate_user()
3. Local JupyterLab instance, uncomment and run:
# ! gcloud auth login
Create helper function to render images from Google Cloud Storage in this notebook#
This is a helpful utility function that you’ll use later in the tutorial. It takes the results of your search query (stored in a DataFrame) and displays the corresponding product images in a nice grid format, making it easy to see how well your search worked.
Import necessary libraries for BigFrames and the helper function#
# Imports for BigFrames, BigQuery, GCS, plotting, and image handling
import io
import math
import bigframes.bigquery as bbq
import bigframes.pandas as bpd
from google.cloud import bigquery
from google.cloud import storage
import matplotlib.pyplot as plt
from PIL import Image
Configure BigFrames and create a Google Cloud Storage client#
# Set variables
REGION = "US"
bucket = "sample-data-and-media"
# Configure BigFrames and create clients
bpd.options.bigquery.project = PROJECT_ID
bpd.options.bigquery.location = REGION
bq_client = bigquery.Client(project=PROJECT_ID)
client = storage.Client(project=PROJECT_ID)
bucket = client.bucket(bucket)
def display_product_images(df):
"""Displays product images from a DataFrame in a 3-column grid.
Simplified for clarity in a tutorial context.
"""
# Calculate grid size
num_images = len(df)
cols = 3
rows = math.ceil(num_images / cols)
fig, axs = plt.subplots(rows, cols, figsize=(10, rows * 3.5))
# Flatten the axes array for easy iteration, regardless of shape
axs = axs.flatten()
for i, row in df.reset_index(drop=True).iterrows():
# Extract the image path from the full GCS URI
image_path = row["uri"].split("/", 3)[-1]
# Get the image from GCS
blob = bucket.blob(image_path)
image = Image.open(io.BytesIO(blob.download_as_bytes()))
# Plot the image
axs[i].imshow(image)
axs[i].set_title(image_path.split("/")[-1], fontsize=10)
axs[i].axis("off")
# Hide any unused subplots
for j in range(num_images, len(axs)):
axs[j].axis("off")
plt.tight_layout()
plt.show()
Load and prepare the sample data#
Load the products DataFrame directly from Cloud Storage#
BigQuery DataFrames allows you to load data directly from Google Cloud Storage into a DataFrame without needing to manually create a BigQuery dataset or persistent table first.
This code uses bigframes.pandas.read_parquet with engine="bigquery" to read the pet products catalog from a public Parquet file in Google Cloud Storage (gs://sample-data-and-media/cymbal-pets/tables/products). BigQuery processes and executes transformations on this data directly in the cloud while providing a familiar pandas-like DataFrame API.
Read the data into a BigFrames DataFrame and inspect the first two rows to confirm the product catalog schema and contents:
products = bpd.read_parquet(
"gs://sample-data-and-media/cymbal-pets/tables/products",
engine="bigquery",
)
products.head(2)
Perform text-to-text semantic search#
With the product data prepared, it’s time to perform the first semantic search. You want to input a text query, like “kitten toy” and find the most similar products. To do this, you first need to convert the product names and descriptions into vector embeddings - numerical representations that capture their semantic meaning. Once the embeddings are generated, you can run a vector search.
Generate text embeddings for product descriptions#
To generate embeddings, you can use the bigframes.bigquery.ai.embed scalar function powered by Google’s pre-trained embedding models (gemini-embedding-001). bbq.ai.embed allows you to generate embeddings directly on DataFrame columns in a single line of Python.
Embeddings are dense numerical representations (an array of floating-point numbers of type ARRAY<FLOAT64>) that capture the semantic essence of each product’s text in high-dimensional vector space.
The following code concatenates product_name and description to give the model full context, calls bbq.ai.embed with the gemini-embedding-001 endpoint, and extracts the embedding vector (.struct.field("result")) directly into a new text_embedding column in the products DataFrame:
products["text_embedding"] = bbq.ai.embed(
products["product_name"] + " " + products["description"],
endpoint="gemini-embedding-001",
).struct.field("result")
Inspect the DataFrame to confirm that the text_embedding column is populated with arrays of float values representing each product’s name and description in vector space:
products[["product_id", "product_name", "description", "text_embedding"]].head(
2
)
Use vector_search to perform a text-to-text search#
Now, use bigframes.bigquery.vector_search to find products matching a user’s natural language search query. This operates in two key steps:
Embed the search query:
bbq.ai.embedconverts the search phrase ("kitten toy") into an embedding vector using the samegemini-embedding-001endpoint.Execute vector search: Because BigQuery’s vector search engine requires a BigQuery table reference for
base_table, callingproducts.to_gbq()materializes the DataFrame into a temporary session table on the fly.bbq.vector_searchthen compares the query vector against each product’stext_embeddingand returns thetop_k(in this case, 3) closest matches.
# Generate an embedding for the search query
query_embedding = bbq.ai.embed(
"kitten toy",
endpoint="gemini-embedding-001",
).struct.field("result")
text_matches = (
bbq.vector_search(
base_table=products.to_gbq(),
column_to_search="text_embedding",
query=query_embedding,
top_k=3,
)[[
"product_id",
"product_name",
"brand",
"category",
"description",
"average_rating",
"uri",
"distance",
]]
.sort_values("distance", ascending=True)
.reset_index(drop=True)
)
The results of the vector search are now stored in a BigFrames DataFrame called text_matches.
Let’s display the top rows.
text_matches.head()
View search results#
Now you can use the helper function to display the images of the top 3 products the vector search returned. As you can see, the results are semantically related to “kitten toy” even if the exact words don’t appear in the product names or descriptions.
display_product_images(text_matches)
Perform text-to-image (multimodal) semantic search#
The previous example performed a text-to-text search. Now we can look at multimodal search. This allows you to use a text query to find products based on their visual characteristics (not using any written descriptions). Note that multimodal search also extends to audio, video, and more.
Generate multimodal embeddings for product images#
To work with images, you need a multimodal embedding model. In this section, you’ll use multimodalembedding@001. Unlike text-only models, a multimodal model projects both text and visual concepts into the exact same shared embedding space.
This capability unlocks cross-modal retrieval: you can compare an embedding of a text query directly against embeddings of product photos to discover visually relevant products.
To generate embeddings from images stored in Google Cloud Storage, BigQuery uses ObjectRef values. You can construct and prepare these references on the fly by chaining three functions from bigframes.bigquery.obj:
bigframes.bigquery.obj.make_ref: Converts each Cloud Storage path from theuricolumn into a structuredObjectRef.bigframes.bigquery.obj.fetch_metadata: Inspects the object in Cloud Storage and populates its file metadata (such as content MIME type).bigframes.bigquery.obj.get_access_url: Generates read-access URL references that the multimodal embedding service can consume.
Passing these references to bbq.ai.embed with the multimodalembedding@001 endpoint computes each image’s embedding vector and assigns it directly to a new mm_embedding column in the DataFrame:
image_urls = bbq.obj.get_access_url(
bbq.obj.fetch_metadata(bbq.obj.make_ref(products["uri"])),
mode="r",
)
products["mm_embedding"] = bbq.ai.embed(
image_urls,
endpoint="multimodalembedding@001",
).struct.field("result")
Inspect the DataFrame to confirm that your new mm_embedding column is populated with multimodal embeddings derived from the product images:
products[["product_id", "product_name", "description", "mm_embedding"]].head(2)
Use vector_search to perform a text-to-image search#
Because the multimodal model maps both text and visual attributes into the same embedding space, you can embed a natural language text query (such as "kitten toy") and compare it directly against the product image embeddings.
This text-to-image search works in two steps:
Embed the text query with the multimodal model: Call
bbq.ai.embedon"kitten toy"withendpoint="multimodalembedding@001". Since this is the exact same model used for the product photos, the resulting text vector is directly comparable to the image vectors.Search against image embeddings: Pass the query embedding into
bbq.vector_search, settingcolumn_to_search="mm_embedding"to find the products that are visually most similar to the query.
mm_query_embedding = bbq.ai.embed(
"kitten toy",
endpoint="multimodalembedding@001",
).struct.field("result")
multimodal_matches = (
bbq.vector_search(
base_table=products.to_gbq(),
column_to_search="mm_embedding",
query=mm_query_embedding,
top_k=3,
)[[
"product_id",
"product_name",
"brand",
"category",
"description",
"average_rating",
"uri",
"distance",
]]
.sort_values("distance", ascending=True)
.reset_index(drop=True)
)
View search results#
Finally, let’s visualize the results of the text-to-image search. You should see images of products that are conceptually related to “kitten toy,” demonstrating the power of multimodal embeddings to find relevant items based on visual similarity to a text description.
display_product_images(multimodal_matches)
Clean up#
Because this tutorial executed all transformations and vector searches directly within BigFrames using ephemeral query jobs, without creating persistent BigQuery datasets, tables, or custom models, there are no permanent BigQuery datasets or models to delete.
To release any temporary BigQuery tables and session resources allocated during this notebook run, you can close the active BigFrames session using
bpd.close_session() in the below code cell.
If you created a dedicated Google Cloud project specifically for this tutorial and no longer need it, you can also delete the project to prevent any future charges.
bpd.close_session()
Run in Colab