Describe product images with BigFrames multimodal DataFrames#
Based on notebook at https://github.com/googleapis/python-bigquery-dataframes/blob/main/notebooks/multimodal/multimodal_dataframe.ipynb
This notebook is introducing BigFrames Multimodal features:
Create Multimodal DataFrame
Combine unstructured data with structured data
Conduct image transformations
Use LLM models to ask questions and generate embeddings on images
PDF chunking function
Install the bigframes package and upgrade other packages that are already included in Kaggle but have versions incompatible with bigframes.
%pip install --upgrade bigframes google-cloud-automl google-cloud-translate google-ai-generativelanguage tensorflow
Important: restart the kernel by going to “Run -> Restart & clear cell outputs” before continuing.
Configure bigframes to use your GCP project. First, go to “Add-ons -> Google Cloud SDK” and click the “Attach” button. Then,
from kaggle_secrets import UserSecretsClient
user_secrets = UserSecretsClient()
user_credential = user_secrets.get_gcloud_credential()
user_secrets.set_tensorflow_credential(user_credential)
PROJECT = "bigframes-dev" # replace with your project.
# Refer to https://cloud.google.com/bigquery/docs/multimodal-data-dataframes-tutorial#required_roles for your required permissions
OUTPUT_BUCKET = "bigframes_blob_test" # replace with your GCS bucket.
# The connection (or bigframes-default-connection of the project) must have read/write permission to the bucket.
# Refer to https://cloud.google.com/bigquery/docs/multimodal-data-dataframes-tutorial#grant-permissions for setting up connection service account permissions.
# In this Notebook it uses bigframes-default-connection by default. You can also bring in your own connections in each method.
import bigframes
# Setup project
bigframes.options.bigquery.project = PROJECT
# Display options
bigframes.options.display.blob_display_width = 300
bigframes.options.display.progress_bar = None
import bigframes.pandas as bpd
# Create blob columns from wildcard path.
df_image = bpd.from_glob_path(
"gs://cloud-samples-data/bigquery/tutorials/cymbal-pets/images/*", name="image"
)
# Other ways are: from string uri column
# df = bpd.DataFrame({"uri": ["gs://<my_bucket>/<my_file_0>", "gs://<my_bucket>/<my_file_1>"]})
# df["blob_col"] = df["uri"].str.to_blob()
# From an existing object table
# df = bpd.read_gbq_object_table("<my_object_table>", name="blob_col")
/usr/local/lib/python3.11/dist-packages/bigframes/core/global_session.py:103: DefaultLocationWarning: No explicit location is set, so using location US for the session.
_global_session = bigframes.session.connect(
Please ensure you have selected a BigQuery account in the Notebook Add-ons menu.
# Take only the 5 images to deal with. Preview the content of the Mutimodal DataFrame
df_image = df_image.head(5)
df_image
| image | |
|---|---|
| 0 |  |
| 1 |  |
| 2 |  |
| 3 |  |
| 4 |  |
5 rows × 1 columns
2. Combine unstructured data with structured data#
Now you can put more information into the table to describe the files. Such as author info from inputs, or other metadata from the gcs object itself.
# Combine unstructured data with structured data
df_image["author"] = ["alice", "bob", "bob", "alice", "bob"] # type: ignore
df_image["content_type"] = df_image["image"].blob.content_type()
df_image["size"] = df_image["image"].blob.size()
df_image["updated"] = df_image["image"].blob.updated()
df_image
/usr/local/lib/python3.11/dist-packages/bigframes/bigquery/_operations/json.py:124: UserWarning: The `json_extract` is deprecated and will be removed in a future
version. Use `json_query` instead.
warnings.warn(bfe.format_message(msg), category=UserWarning)
/usr/local/lib/python3.11/dist-packages/bigframes/bigquery/_operations/json.py:124: UserWarning: The `json_extract` is deprecated and will be removed in a future
version. Use `json_query` instead.
warnings.warn(bfe.format_message(msg), category=UserWarning)
/usr/local/lib/python3.11/dist-packages/bigframes/bigquery/_operations/json.py:124: UserWarning: The `json_extract` is deprecated and will be removed in a future
version. Use `json_query` instead.
warnings.warn(bfe.format_message(msg), category=UserWarning)
| image | author | content_type | size | updated | |
|---|---|---|---|---|---|
| 0 |  |
alice | image/png | 1591240 | 2025-03-20 17:45:04+00:00 |
| 1 |  |
bob | image/png | 1182951 | 2025-03-20 17:45:02+00:00 |
| 2 |  |
bob | image/png | 1520884 | 2025-03-20 17:44:55+00:00 |
| 3 |  |
alice | image/png | 1235401 | 2025-03-20 17:45:19+00:00 |
| 4 |  |
bob | image/png | 1591923 | 2025-03-20 17:44:47+00:00 |
5 rows × 5 columns
Then you can filter the rows based on the structured data. And for different content types, you can display them respectively or together.
# filter images and display, you can also display audio and video types
df_image[df_image["author"] == "alice"]["image"].blob.display()
/usr/local/lib/python3.11/dist-packages/bigframes/bigquery/_operations/json.py:124: UserWarning: The `json_extract` is deprecated and will be removed in a future
version. Use `json_query` instead.
warnings.warn(bfe.format_message(msg), category=UserWarning)
3. Conduct image transformations#
BigFrames Multimodal DataFrame provides image(and other) transformation functions. Such as image_blur, image_resize and image_normalize. The output can be saved to GCS folders or to BQ as bytes.
df_image["blurred"] = df_image["image"].blob.image_blur(
(20, 20), dst=f"gs://{OUTPUT_BUCKET}/image_blur_transformed/", engine="opencv"
)
df_image["resized"] = df_image["image"].blob.image_resize(
(300, 200), dst=f"gs://{OUTPUT_BUCKET}/image_resize_transformed/", engine="opencv"
)
df_image["normalized"] = df_image["image"].blob.image_normalize(
alpha=50.0,
beta=150.0,
norm_type="minmax",
dst=f"gs://{OUTPUT_BUCKET}/image_normalize_transformed/",
engine="opencv",
)
/usr/local/lib/python3.11/dist-packages/bigframes/core/log_adapter.py:175: FunctionAxisOnePreviewWarning: Blob Functions use bigframes DataFrame Managed function with axis=1 senario, which is a preview feature.
return method(*args, **kwargs)
/usr/local/lib/python3.11/dist-packages/bigframes/core/log_adapter.py:175: FunctionAxisOnePreviewWarning: Blob Functions use bigframes DataFrame Managed function with axis=1 senario, which is a preview feature.
return method(*args, **kwargs)
/usr/local/lib/python3.11/dist-packages/bigframes/core/log_adapter.py:175: FunctionAxisOnePreviewWarning: Blob Functions use bigframes DataFrame Managed function with axis=1 senario, which is a preview feature.
return method(*args, **kwargs)
# You can also chain functions together
df_image["blur_resized"] = df_image["blurred"].blob.image_resize((300, 200), dst=f"gs://{OUTPUT_BUCKET}/image_blur_resize_transformed/", engine="opencv")
df_image
/usr/local/lib/python3.11/dist-packages/bigframes/core/log_adapter.py:175: FunctionAxisOnePreviewWarning: Blob Functions use bigframes DataFrame Managed function with axis=1 senario, which is a preview feature.
return method(*args, **kwargs)
| image | author | content_type | size | updated | blurred | resized | normalized | blur_resized | |
|---|---|---|---|---|---|---|---|---|---|
| 0 |  |
alice | image/png | 1591240 | 2025-03-20 17:45:04+00:00 |  |
 |
 |
 |
| 1 |  |
bob | image/png | 1182951 | 2025-03-20 17:45:02+00:00 |  |
 |
 |
 |
| 2 |  |
bob | image/png | 1520884 | 2025-03-20 17:44:55+00:00 |  |
 |
 |
 |
| 3 |  |
alice | image/png | 1235401 | 2025-03-20 17:45:19+00:00 |  |
 |
 |
 |
| 4 |  |
bob | image/png | 1591923 | 2025-03-20 17:44:47+00:00 |  |
 |
 |
 |
5 rows × 9 columns
4. Use LLM models to ask questions and generate embeddings on images#
from bigframes.ml import llm
gemini = llm.GeminiTextGenerator()
/usr/local/lib/python3.11/dist-packages/bigframes/core/log_adapter.py:175: FutureWarning: Since upgrading the default model can cause unintended breakages, the
default model will be removed in BigFrames 3.0. Please supply an
explicit model to avoid this message.
return method(*args, **kwargs)
# Ask the same question on the images
df_image = df_image.head(2)
answer = gemini.predict(df_image, prompt=["what item is it?", df_image["image"]])
answer[["ml_generate_text_llm_result", "image"]]
/usr/local/lib/python3.11/dist-packages/bigframes/core/array_value.py:108: PreviewWarning: JSON column interpretation as a custom PyArrow extention in
`db_dtypes` is a preview feature and subject to change.
warnings.warn(msg, bfe.PreviewWarning)
| ml_generate_text_llm_result | image | |
|---|---|---|
| 0 | The item is a tin of K9 Guard Dog Paw Balm. |  |
| 1 | The item is a bottle of K9 Guard Dog Hot Spot Spray. |  |
2 rows × 2 columns
# Ask different questions
df_image["question"] = ["what item is it?", "what color is the picture?"]
answer_alt = gemini.predict(df_image, prompt=[df_image["question"], df_image["image"]])
answer_alt[["ml_generate_text_llm_result", "image"]]
/usr/local/lib/python3.11/dist-packages/bigframes/core/array_value.py:108: PreviewWarning: JSON column interpretation as a custom PyArrow extention in
`db_dtypes` is a preview feature and subject to change.
warnings.warn(msg, bfe.PreviewWarning)
| ml_generate_text_llm_result | image | |
|---|---|---|
| 0 | The item is a tin of K9 Guard Dog Paw Balm. |  |
| 1 | The picture has colors such as white, gray, and a light blue (cyan). |  |
2 rows × 2 columns
# Generate embeddings.
embed_model = llm.MultimodalEmbeddingGenerator()
embeddings = embed_model.predict(df_image["image"])
embeddings
/usr/local/lib/python3.11/dist-packages/bigframes/core/log_adapter.py:175: FutureWarning: Since upgrading the default model can cause unintended breakages, the
default model will be removed in BigFrames 3.0. Please supply an
explicit model to avoid this message.
return method(*args, **kwargs)
/usr/local/lib/python3.11/dist-packages/bigframes/core/array_value.py:108: PreviewWarning: JSON column interpretation as a custom PyArrow extention in
`db_dtypes` is a preview feature and subject to change.
warnings.warn(msg, bfe.PreviewWarning)
| ml_generate_embedding_result | ml_generate_embedding_status | ml_generate_embedding_start_sec | ml_generate_embedding_end_sec | content | |
|---|---|---|---|---|---|
| 0 | [ 0.00638822 0.01666385 0.00451817 ... -0.02... | <NA> | <NA> | {"access_urls":{"expiry_time":"2025-08-19T02:3... | |
| 1 | [ 0.00973672 0.02148364 0.00244308 ... 0.00... | <NA> | <NA> | {"access_urls":{"expiry_time":"2025-08-19T02:3... |
2 rows × 5 columns