BigQuery DataFrames (bigframes) explained

BigQuery DataFrames is a pandas-style Python library that turns your code into SQL and runs it in BigQuery. The data stays in BigQuery until you call to_pandas, every step is billed like a normal query, and the default ordering mode can make a filter scan a whole table.

BigQuery DataFrames (the Python package is bigframes) is an open source library with a pandas-like API. bigframes.pandas doesn’t load your table into memory. Each operation is recorded, and when you ask for a result the library compiles everything into one SQL query and runs it as a normal BigQuery job. The data never leaves BigQuery unless you call to_pandas() or print the frame. The library is free; you pay for the BigQuery jobs it submits.

import bigframes.pandas as bpd

df = bpd.read_gbq("bigquery-public-data.ml_datasets.penguins")
avg = df.groupby("species")["body_mass_g"].mean().sort_values(ascending=False)
avg.to_pandas()   # this is the line that runs a query

We ran the examples in this article with bigframes 2.50.0. The code blocks below are Python, so they are not runnable on this site; the numbers come from real runs and dry runs.

What runs where

Google says the library implements more than 750 pandas and scikit-learn APIs this way. A subset of pandas is supported, not all of it. The type system follows BigQuery: INTERVAL and RANGE are not supported, and other types without a pandas equivalent show up as object.

You can look at the SQL for any frame with its .sql property. For the groupby above it is a nested SELECT over the table. One detail Google’s pages don’t mention: read_gbq reads the table as of a fixed time, and the query contains FOR SYSTEM_TIME AS OF '2026-10-06T22:18:55...'. That gives your session a consistent view if the table changes while you work. See time travel.

Nothing runs until you ask for a result

bigframes defers execution. These calls run a query: to_pandas(), to_pandas_batches(), peek(), head() when its result is shown, and print(df) or the notebook display of a frame (its repr, which calls head()). Setting bpd.options.display.repr_mode = "deferred" stops a bare df in a notebook from running a query and billing you for it.

Because queries are deferred, the same expensive step can run again in every later query that depends on it. df.cache() stores the result in a temporary table so later work reuses it. That table is billed as storage.

To see what a pull would process before you do it, ask for a dry run:

df.to_pandas(dry_run=True)   # returns stats including totalBytesProcessed, runs no query

In v2 the methods that return results to your machine enforce BigQuery’s maximum response size by default (allow_large_results=False). Set it to True per call or in bpd.options.bigquery if you pull large results, and expect a temporary table to be used then. See Query result limits.

Ordering mode decides what a filter costs

pandas has row order and a row index. BigQuery tables have neither, so bigframes has to make them up. The default, ordering_mode = "strict", builds a total order. The SQL bigframes generates for it hashes every column of every row with FARM_FINGERPRINT. Google says this can “cause a full data scan that ignores row and column filters”. We measured it on google_trends.top_terms, a 3.2 GB table partitioned by refresh_date:

All numbers are dry-run totalBytesProcessed. The table is refreshed daily, so the sizes will differ when you try it, but the shape holds. Strict mode reads every column of the rows it keeps, because the order is a hash of all of them. Partial mode reads two columns, and prunes the partition from a filter written after read_gbq.

Switch to partial mode before anything else:

bpd.options.bigquery.ordering_mode = "partial"

What partial mode takes away: iloc, the sequential index, drop_duplicates() and unique(), and implicit joins by index (use merge). head() fails with OrderRequiredError: Op head requires an ordering until you call sort_values, and a head() on an unordered frame can return different rows each time. Use peek(n) for a quick look: it returns n rows without ordering, and it is cheaper than head().

For a table with require_partition_filter there is one more rule. We tried read_gbq on a partitioned table that requires a filter, in both modes, without the filters= argument, and BigQuery rejected the call: “Cannot query over table … without a filter over column(s) ‘datehour’ that can be used for partition elimination”. bigframes runs a query when it reads the table, so a filter written afterwards comes too late. Pass filters= to read_gbq for those tables. See partitioning.

Every call is a job, and it is labelled

Everything bigframes does is a job in your project, with the usual billing: bytes on on-demand, slots on a reservation (see how queries are billed). We looked at the jobs from a short session and found two things worth knowing:

• Jobs carry labels such as bigframes-api: dataframe-cache and recent-bigframes-api-0: dataframe-__getitem__, naming the API call behind the job. They show up in INFORMATION_SCHEMA.JOBS, which makes it easy to see which Python calls cost money. See labels and billing export.
• Session setup runs small SELECT 1 jobs. They process nothing.

Sessions and temporary tables

A bigframes session holds the metadata for your frames and is tied to one location. Each frame belongs to a session, and mixing frames from two sessions fails with Cannot use combine sources from multiple sessions, even if both sessions have the same settings. bpd.options.bigquery configures a default global session; call bpd.close_session() to reset its project or location. The default location is the US multi-region, and reading a table sets it to the table’s location when you haven’t set one.

Intermediate data goes into the hidden dataset BigQuery creates for query results (its name starts with an underscore). In our run, cache() wrote a table named _<hash>_bqdf_<uuid> there, and the dataset’s default table and partition expiration was 1 day, with the cache table expiring after exactly 1 day. Google’s pricing note says these tables persist for seven days by default. We saw one day, so check the expiration on yours, and don’t assume the table is gone sooner. Either way, you pay storage for them until they expire. See temporary tables.

Machine learning

bigframes.ml has scikit-learn style classes (LinearRegression, KMeans, PCA, preprocessing transformers, pipelines). They are a wrapper over BigQuery ML. We fit a LinearRegression on the penguins table and looked at the jobs:

The model is a real BigQuery ML model, stored in the same hidden dataset. Model training prices and region availability are BigQuery ML’s, since that is what runs. See BigQuery ML.

bigframes.bigquery exposes SQL functions that pandas doesn’t have (array_agg, struct, unix_micros, sql_scalar for an arbitrary SQL expression, and the generative AI functions under bigframes.bigquery.ai). See generative AI functions.

Python functions you write yourself

pandas lets you apply any Python function. BigQuery can’t run arbitrary Python inline, so bigframes has two ways to deploy one:

• @bpd.remote_function creates a Cloud Run function, a BigQuery connection (by default bigframes-default-connection) and a BigQuery remote function that calls it. It needs several APIs enabled (Cloud Run Admin, Cloud Functions, Artifact Registry, Cloud Build, Compute Engine, BigQuery Connection, and more) and strong IAM roles. You pay for Cloud Run and the other services used. You must clean up the functions yourself: they aren’t removed when the session ends. See remote functions.
• @bpd.udf (Preview) creates a persistent BigQuery Python UDF routine. The function must be self-contained (no imports or variables from outside its body), must work with Python 3.11, and its code is visible to anyone with read access on the routine. Renaming a variable re-creates it.

Since version 2.0, deployed remote functions no longer use the Compute Engine service account by default, and their ingress is set to internal only. Supply cloud_function_service_account to control permissions.

Which to choose