{ "cells": [ { "cell_type": "code", "execution_count": null, "metadata": { "id": "cMQCs0oQf5Jo" }, "outputs": [], "source": [ "# Copyright 2026 Google LLC\n", "#\n", "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", "# you may not use this file except in compliance with the License.\n", "# You may obtain a copy of the License at\n", "#\n", "# https://www.apache.org/licenses/LICENSE-2.0\n", "#\n", "# Unless required by applicable law or agreed to in writing, software\n", "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", "# See the License for the specific language governing permissions and\n", "# limitations under the License." ] }, { "cell_type": "markdown", "metadata": { "id": "iR0jdheRGG89" }, "source": [ "# Text + multimodal embedding generation and vector search in BigQuery with BigFrames\n" ] }, { "cell_type": "markdown", "metadata": { "id": "td9kx9LVgSve" }, "source": [ "\n", " \n", " \n", " \n", " \n", "
\n", " \n", " \"Colab Run in Colab\n", " \n", " \n", " \n", " \"GitHub View on GitHub\n", " \n", " \n", " \n", " \"Colab Open in Colab Enterprise\n", " \n", " \n", " \n", " \"BQ Open in BQ Studio\n", " \n", "
\n", "\n", "
\n", "\n", "Share to:\n", "\n", "\n", " \"LinkedIn\n", "\n", "\n", "\n", " \"Bluesky\n", "\n", "\n", "\n", " \"X\n", "\n", "\n", "\n", " \"Reddit\n", "\n", "\n", "\n", " \"Facebook\n", "" ] }, { "cell_type": "markdown", "metadata": { "id": "intro_md" }, "source": [ "## Overview\n" ] }, { "cell_type": "markdown", "metadata": { "id": "SwQLISjqGTHd" }, "source": [ "### Objectives\n", "\n", "In this tutorial, you'll build a [**semantic search**](https://cloud.google.com/discover/what-is-semantic-search) utility for a fictional e-commerce pet store using [BigQuery](https://cloud.google.com/bigquery/docs), [BigQuery DataFrames (BigFrames)](https://cloud.google.com/bigquery/docs/use-bigquery-dataframes), and [Gemini Enterprise Agent Platform](https://docs.cloud.google.com/gemini-enterprise-agent-platform). Unlike simple keyword matching, semantic search uses [**vector embeddings**](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/embeddings) to capture the conceptual meaning of both product descriptions and product images. This enables customers to find products based on intent and visual attributes, creating a much more intuitive shopping experience.\n", "\n", "By the end of this tutorial, you will know how to:\n", "\n", "* **Load data directly from Google Cloud Storage into a BigFrames DataFrame** using `bpd.read_parquet`.\n", "* **Generate text embeddings** on DataFrame columns using `bigframes.bigquery.ai.embed`.\n", "* **Perform text-to-text vector search** with `bigframes.bigquery.vector_search` to find products with semantically similar descriptions.\n", "* **Work with unstructured image data** using BigFrames' ObjectRef functions (`bbq.obj.make_ref`, `bbq.obj.fetch_metadata`, `bbq.obj.get_access_url`) and generate multimodal embeddings.\n", "* **Perform text-to-image vector search** to find product images that visually match a natural language text query.\n", "\n", "All seamlessly orchestrated using Python and BigFrames!" ] }, { "cell_type": "markdown", "metadata": { "id": "costs_md" }, "source": [ "### Services and Costs\n", "\n", "This tutorial uses the following billable components of Google Cloud:\n", "\n", "* **BigQuery**: [Pricing](https://cloud.google.com/bigquery/pricing)\n", "* **Gemini Enterprise Agent Platform (Embeddings API)**: [Pricing](https://cloud.google.com/products/gemini-enterprise-agent-platform/pricing#generative_ai_pricing)\n", "\n", "You can use the [Pricing Calculator](https://cloud.google.com/products/calculator) to generate a cost estimate based on your projected usage." ] }, { "cell_type": "markdown", "metadata": { "id": "nkoCFoFVSPii" }, "source": [ "---\n", "\n", "## Before you begin" ] }, { "cell_type": "markdown", "metadata": { "id": "setup_md_1" }, "source": [ "### Set up your Google Cloud project\n", "**The following steps are required, regardless of your notebook environment.**\n", "\n", "1. [Select or create a Google Cloud project](https://console.cloud.google.com/cloud-resource-manager). When you first create an account, you get a $300 free credit towards your compute/storage costs.\n", "\n", "2. [Make sure that billing is enabled for your project](https://cloud.google.com/billing/docs/how-to/modify-project).\n", "\n", "3. [Enable the BigQuery, BigQuery Connection, and Agent Platform APIs](https://console.cloud.google.com/flows/enableapi?apiid=bigquery.googleapis.com,bigqueryconnection.googleapis.com,aiplatform.googleapis.com).\n", "\n", "4. If you are running this notebook locally, you need to install the [Cloud SDK](https://cloud.google.com/sdk)." ] }, { "cell_type": "markdown", "metadata": { "id": "YQ3g-h7uTaSf" }, "source": [ "### Set your project ID" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "set_project_id" }, "outputs": [], "source": [ "PROJECT_ID = \"\" # @param {type:\"string\"}\n", "\n", "# Set the project id\n", "! gcloud config set project {PROJECT_ID}" ] }, { "cell_type": "markdown", "metadata": { "id": "auth_md" }, "source": [ "### Authenticate to your Google Cloud account\n", "\n", "Depending on your Jupyter environment, you may have to manually authenticate. Follow the relevant instructions below." ] }, { "cell_type": "markdown", "metadata": { "id": "V6NjZRCXU5Ro" }, "source": [ "**1. Colab Enterprise or BigQuery Studio Notebooks**\n", "* Do nothing as you are already authenticated." ] }, { "cell_type": "markdown", "metadata": { "id": "l0dV1hvAU1ed" }, "source": [ "**2. Colab, uncomment and run:**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "auth_code" }, "outputs": [], "source": [ "# from google.colab import auth\n", "\n", "# auth.authenticate_user()" ] }, { "cell_type": "markdown", "metadata": { "id": "4Pp4LAJ3UyRP" }, "source": [ "**3. Local JupyterLab instance, uncomment and run:**\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "AzU7S3fMVDkW" }, "outputs": [], "source": [ "# ! gcloud auth login" ] }, { "cell_type": "markdown", "metadata": { "id": "BfaINa3YxYQ3" }, "source": [ "### Create helper function to render images from Google Cloud Storage in this notebook\n", "\n", "This is a helpful utility function that you'll use later in the tutorial. It takes the results of your search query (stored in a DataFrame) and displays the corresponding product images in a nice grid format, making it easy to see how well your search worked." ] }, { "cell_type": "markdown", "metadata": { "id": "4g4dsEJKusGN" }, "source": [ "#### Import necessary libraries for BigFrames and the helper function" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "jiepvKmLuyob" }, "outputs": [], "source": [ "# Imports for BigFrames, BigQuery, GCS, plotting, and image handling\n", "import io\n", "import math\n", "\n", "import bigframes.bigquery as bbq\n", "import bigframes.pandas as bpd\n", "from google.cloud import bigquery\n", "from google.cloud import storage\n", "import matplotlib.pyplot as plt\n", "from PIL import Image" ] }, { "cell_type": "markdown", "metadata": { "id": "rfm005SWxhxg" }, "source": [ "#### Configure BigFrames and create a Google Cloud Storage client\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "xUcaWbhWxkVv" }, "outputs": [], "source": [ "# Set variables\n", "REGION = \"US\"\n", "bucket = \"sample-data-and-media\"\n", "\n", "# Configure BigFrames and create clients\n", "bpd.options.bigquery.project = PROJECT_ID\n", "bpd.options.bigquery.location = REGION\n", "bq_client = bigquery.Client(project=PROJECT_ID)\n", "client = storage.Client(project=PROJECT_ID)\n", "bucket = client.bucket(bucket)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "s6YFp-2Axer1" }, "outputs": [], "source": [ "def display_product_images(df):\n", " \"\"\"Displays product images from a DataFrame in a 3-column grid.\n", "\n", " Simplified for clarity in a tutorial context.\n", " \"\"\"\n", " # Calculate grid size\n", " num_images = len(df)\n", " cols = 3\n", " rows = math.ceil(num_images / cols)\n", "\n", " fig, axs = plt.subplots(rows, cols, figsize=(10, rows * 3.5))\n", "\n", " # Flatten the axes array for easy iteration, regardless of shape\n", " axs = axs.flatten()\n", "\n", " for i, row in df.reset_index(drop=True).iterrows():\n", " # Extract the image path from the full GCS URI\n", " image_path = row[\"uri\"].split(\"/\", 3)[-1]\n", "\n", " # Get the image from GCS\n", " blob = bucket.blob(image_path)\n", " image = Image.open(io.BytesIO(blob.download_as_bytes()))\n", "\n", " # Plot the image\n", " axs[i].imshow(image)\n", " axs[i].set_title(image_path.split(\"/\")[-1], fontsize=10)\n", " axs[i].axis(\"off\")\n", "\n", " # Hide any unused subplots\n", " for j in range(num_images, len(axs)):\n", " axs[j].axis(\"off\")\n", "\n", " plt.tight_layout()\n", " plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "HozZFfxrg9d_" }, "source": [ "---\n" ] }, { "cell_type": "markdown", "metadata": { "id": "yla8q_2EFQl9" }, "source": [ "## Load and prepare the sample data" ] }, { "cell_type": "markdown", "metadata": { "id": "CFncMrqEGMAO" }, "source": [ "### Load the `products` DataFrame directly from Cloud Storage\n", "\n", "BigQuery DataFrames allows you to load data directly from Google Cloud Storage into a DataFrame without needing to manually create a BigQuery dataset or persistent table first.\n", "\n", "This code uses [`bigframes.pandas.read_parquet`](https://dataframes.bigquery.dev/reference/api/bigframes.pandas.read_parquet.html) with `engine=\"bigquery\"` to read the pet products catalog from a public Parquet file in Google Cloud Storage (`gs://sample-data-and-media/cymbal-pets/tables/products`). BigQuery processes and executes transformations on this data directly in the cloud while providing a familiar pandas-like DataFrame API." ] }, { "cell_type": "markdown", "metadata": { "id": "cgSJkD3Q93l7" }, "source": [ "Read the data into a BigFrames DataFrame and inspect the first two rows to confirm the product catalog schema and contents:" ] }, { "cell_type": "code", "source": [ "products = bpd.read_parquet(\n", " \"gs://sample-data-and-media/cymbal-pets/tables/products\",\n", " engine=\"bigquery\",\n", ")\n", "products.head(2)" ], "metadata": { "id": "s-qUnDtdmoYB" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "039VFiVBwzsF" }, "source": [ "---\n", "\n", "\n", "## Perform text-to-text semantic search\n", "\n", "With the product data prepared, it's time to perform the first [semantic search](https://cloud.google.com/discover/what-is-semantic-search). You want to input a text query, like \"kitten toy\" and find the most similar products. To do this, you first need to convert the product names and descriptions into [vector embeddings](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings) - numerical representations that capture their semantic meaning. Once the embeddings are generated, you can run a vector search.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "AgzCgXcUwcm4" }, "source": [ "### Generate text embeddings for product descriptions" ] }, { "cell_type": "markdown", "metadata": { "id": "dtgmxoax-pUT" }, "source": [ "To generate embeddings, you can use the [`bigframes.bigquery.ai.embed`](https://dataframes.bigquery.dev/reference/api/bigframes.bigquery.ai.embed.html) scalar function powered by Google's pre-trained embedding models ([`gemini-embedding-001`](https://ai.google.dev/gemini-api/docs/embeddings#model-versions)). `bbq.ai.embed` allows you to generate embeddings directly on DataFrame columns in a single line of Python.\n", "\n", "Embeddings are dense numerical representations (an array of floating-point numbers of type `ARRAY`) that capture the semantic essence of each product's text in high-dimensional vector space." ] }, { "cell_type": "markdown", "metadata": { "id": "kOo1l6rdOYk8" }, "source": [ "The following code concatenates `product_name` and `description` to give the model full context, calls `bbq.ai.embed` with the `gemini-embedding-001` endpoint, and extracts the embedding vector (`.struct.field(\"result\")`) directly into a new `text_embedding` column in the `products` DataFrame:" ] }, { "cell_type": "code", "source": [ "products[\"text_embedding\"] = bbq.ai.embed(\n", " products[\"product_name\"] + \" \" + products[\"description\"],\n", " endpoint=\"gemini-embedding-001\",\n", ").struct.field(\"result\")" ], "metadata": { "id": "byVthBfLxRqm" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "FylUH24h_kXN" }, "source": [ "Inspect the DataFrame to confirm that the `text_embedding` column is populated with arrays of float values representing each product's name and description in vector space:" ] }, { "cell_type": "code", "source": [ "products[[\"product_id\", \"product_name\", \"description\", \"text_embedding\"]].head(\n", " 2\n", ")" ], "metadata": { "id": "PyytEqB6y8Iw" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "EBBg2qw0_7Q-" }, "source": [ "### Use `vector_search` to perform a text-to-text search\n", "\n", "Now, use [`bigframes.bigquery.vector_search`](https://dataframes.bigquery.dev/reference/api/bigframes.bigquery.vector_search.html) to find products matching a user's natural language search query. This operates in two key steps:\n", "\n", "1. **Embed the search query**: `bbq.ai.embed` converts the search phrase (`\"kitten toy\"`) into an embedding vector using the same `gemini-embedding-001` endpoint.\n", "2. **Execute vector search**: Because BigQuery's vector search engine requires a BigQuery table reference for `base_table`, calling `products.to_gbq()` materializes the DataFrame into a temporary session table on the fly. `bbq.vector_search` then compares the query vector against each product's `text_embedding` and returns the `top_k` (in this case, 3) closest matches." ] }, { "cell_type": "code", "source": [ "# Generate an embedding for the search query\n", "query_embedding = bbq.ai.embed(\n", " \"kitten toy\",\n", " endpoint=\"gemini-embedding-001\",\n", ").struct.field(\"result\")\n", "\n", "text_matches = (\n", " bbq.vector_search(\n", " base_table=products.to_gbq(),\n", " column_to_search=\"text_embedding\",\n", " query=query_embedding,\n", " top_k=3,\n", " )[[\n", " \"product_id\",\n", " \"product_name\",\n", " \"brand\",\n", " \"category\",\n", " \"description\",\n", " \"average_rating\",\n", " \"uri\",\n", " \"distance\",\n", " ]]\n", " .sort_values(\"distance\", ascending=True)\n", " .reset_index(drop=True)\n", ")" ], "metadata": { "id": "p6sV3r0hzsH2" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "xzaZ3Gs2ASLu" }, "source": [ "The results of the vector search are now stored in a BigFrames DataFrame called `text_matches`.\n", "\n", "Let's display the top rows." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "a1SlsPNz79nk" }, "outputs": [], "source": [ "text_matches.head()" ] }, { "cell_type": "markdown", "metadata": { "id": "J-Ev_nYsAjVI" }, "source": [ "### View search results\n", "Now you can use the helper function to display the images of the top 3 products the vector search returned. As you can see, the results are semantically related to \"kitten toy\" even if the exact words don't appear in the product names or descriptions." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "c5096xH93K2T" }, "outputs": [], "source": [ "display_product_images(text_matches)" ] }, { "cell_type": "markdown", "metadata": { "id": "9ytmT_Q6-J2I" }, "source": [ "---\n", "\n", "\n", "## Perform text-to-image (multimodal) semantic search\n", "\n", "The previous example performed a text-to-text search. Now we can look at multimodal search. This allows you to use a text query to find products based on their visual characteristics (not using any written descriptions). Note that multimodal search also extends to audio, video, and more." ] }, { "cell_type": "markdown", "metadata": { "id": "86UojNj1HLGB" }, "source": [ "### Generate multimodal embeddings for product images" ] }, { "cell_type": "markdown", "metadata": { "id": "C603D8_wBLpv" }, "source": [ "To work with images, you need a multimodal embedding model. In this section, you'll use [`multimodalembedding@001`](https://docs.cloud.google.com/gemini-enterprise-agent-platform/models/embeddings/get-multimodal-embeddings?#supported-models). Unlike text-only models, a **multimodal** model projects both text and visual concepts into the exact same shared embedding space.\n", "\n", "This capability unlocks cross-modal retrieval: you can compare an embedding of a text query directly against embeddings of product photos to discover visually relevant products." ] }, { "cell_type": "markdown", "metadata": { "id": "uqON6RSYBW-y" }, "source": [ "To generate embeddings from images stored in Google Cloud Storage, BigQuery uses [`ObjectRef`](https://cloud.google.com/bigquery/docs/reference/standard-sql/objectref_functions) values. You can construct and prepare these references on the fly by chaining three functions from `bigframes.bigquery.obj`:\n", "\n", "* [`bigframes.bigquery.obj.make_ref`](https://dataframes.bigquery.dev/reference/api/bigframes.bigquery.obj.make_ref.html): Converts each Cloud Storage path from the `uri` column into a structured `ObjectRef`.\n", "* [`bigframes.bigquery.obj.fetch_metadata`](https://dataframes.bigquery.dev/reference/api/bigframes.bigquery.obj.fetch_metadata.html): Inspects the object in Cloud Storage and populates its file metadata (such as content MIME type).\n", "* [`bigframes.bigquery.obj.get_access_url`](https://dataframes.bigquery.dev/reference/api/bigframes.bigquery.obj.get_access_url.html): Generates read-access URL references that the multimodal embedding service can consume.\n", "\n", "Passing these references to `bbq.ai.embed` with the `multimodalembedding@001` endpoint computes each image's embedding vector and assigns it directly to a new `mm_embedding` column in the DataFrame:" ] }, { "cell_type": "code", "source": [ "image_urls = bbq.obj.get_access_url(\n", " bbq.obj.fetch_metadata(bbq.obj.make_ref(products[\"uri\"])),\n", " mode=\"r\",\n", ")\n", "products[\"mm_embedding\"] = bbq.ai.embed(\n", " image_urls,\n", " endpoint=\"multimodalembedding@001\",\n", ").struct.field(\"result\")" ], "metadata": { "id": "0i2Mt1z10PAK" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "YhJ9OzDmCANS" }, "source": [ "Inspect the DataFrame to confirm that your new `mm_embedding` column is populated with multimodal embeddings derived from the product images:" ] }, { "cell_type": "code", "source": [ "products[[\"product_id\", \"product_name\", \"description\", \"mm_embedding\"]].head(2)" ], "metadata": { "id": "zeVTux28BZBz" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "Yoy6iKuXCKI4" }, "source": [ "### Use `vector_search` to perform a text-to-image search\n", "\n", "Because the multimodal model maps both text and visual attributes into the same embedding space, you can embed a natural language text query (such as `\"kitten toy\"`) and compare it directly against the product image embeddings.\n", "\n", "This text-to-image search works in two steps:\n", "1. **Embed the text query with the multimodal model**: Call `bbq.ai.embed` on `\"kitten toy\"` with `endpoint=\"multimodalembedding@001\"`. Since this is the exact same model used for the product photos, the resulting text vector is directly comparable to the image vectors.\n", "2. **Search against image embeddings**: Pass the query embedding into `bbq.vector_search`, setting `column_to_search=\"mm_embedding\"` to find the products that are visually most similar to the query." ] }, { "cell_type": "code", "source": [ "mm_query_embedding = bbq.ai.embed(\n", " \"kitten toy\",\n", " endpoint=\"multimodalembedding@001\",\n", ").struct.field(\"result\")\n", "\n", "multimodal_matches = (\n", " bbq.vector_search(\n", " base_table=products.to_gbq(),\n", " column_to_search=\"mm_embedding\",\n", " query=mm_query_embedding,\n", " top_k=3,\n", " )[[\n", " \"product_id\",\n", " \"product_name\",\n", " \"brand\",\n", " \"category\",\n", " \"description\",\n", " \"average_rating\",\n", " \"uri\",\n", " \"distance\",\n", " ]]\n", " .sort_values(\"distance\", ascending=True)\n", " .reset_index(drop=True)\n", ")" ], "metadata": { "id": "D6yyqUTQBiyS" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "ma7A9jF0HbUg" }, "source": [ "### View search results" ] }, { "cell_type": "markdown", "metadata": { "id": "D1SRowjwCsda" }, "source": [ "Finally, let's visualize the results of the text-to-image search. You should see images of products that are conceptually related to \"kitten toy,\" demonstrating the power of multimodal embeddings to find relevant items based on visual similarity to a text description." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "BNOV5oYY3I74" }, "outputs": [], "source": [ "display_product_images(multimodal_matches)" ] }, { "cell_type": "markdown", "metadata": { "id": "07efC15wSPjB" }, "source": [ "---" ] }, { "cell_type": "markdown", "metadata": { "id": "clean_up_md" }, "source": [ "## Clean up\n", "\n", "Because this tutorial executed all transformations and vector searches directly within BigFrames using ephemeral query jobs, without creating persistent BigQuery datasets, tables, or custom models, there are no permanent BigQuery datasets or models to delete.\n", "\n", "To release any temporary BigQuery tables and session resources allocated during this notebook run, you can close the active BigFrames session using\n", "`bpd.close_session()` in the below code cell.\n", "\n", "If you created a dedicated Google Cloud project specifically for this tutorial and no longer need it, you can also [delete the project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) to prevent any future charges." ] }, { "cell_type": "code", "source": [ "bpd.close_session()" ], "metadata": { "id": "4prSZHoviS8V" }, "execution_count": null, "outputs": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.1" } }, "nbformat": 4, "nbformat_minor": 0 }