# BigQuery DataFrames (bigframes) explained > BigQuery DataFrames is a pandas-style Python library that turns your code into SQL and runs it in BigQuery. The data stays in BigQuery until you call to_pandas, every step is billed like a normal query, and the default ordering mode can make a filter scan a whole table. Source: https://docs.querylab.io/bigquery/bigquery-dataframes/ Updated: 2026-10-07 BigQuery DataFrames (the Python package is `bigframes`) is an open source library with a pandas-like API. `bigframes.pandas` doesn't load your table into memory. Each operation is recorded, and when you ask for a result the library compiles everything into one SQL query and runs it as a normal BigQuery job. The data never leaves BigQuery unless you call `to_pandas()` or print the frame. The library is free; you pay for the BigQuery jobs it submits. ```python df = bpd.read_gbq("bigquery-public-data.ml_datasets.penguins") avg = df.groupby("species")["body_mass_g"].mean().sort_values(ascending=False) avg.to_pandas() # this is the line that runs a query ``` We ran the examples in this article with bigframes 2.50.0. The code blocks below are Python, so they are not runnable on this site; the numbers come from real runs and dry runs. ## What runs where | Where | What | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | BigQuery | Filters, joins, group-bys, window functions, `bigframes.bigquery` SQL functions, `bigframes.ml` model training and prediction | | Your machine | The `bigframes` Python session, and anything after `to_pandas()` | | Cloud Run functions | Your own Python functions, if you wrap them with `@remote_function` | | A Python UDF in BigQuery | Your own Python function, if you wrap it with `udf` (Preview) | Google says the library implements more than 750 pandas and scikit-learn APIs this way. A subset of pandas is supported, not all of it. The type system follows BigQuery: `INTERVAL` and `RANGE` are not supported, and other types without a pandas equivalent show up as `object`. You can look at the SQL for any frame with its `.sql` property. For the groupby above it is a nested `SELECT` over the table. One detail Google's pages don't mention: `read_gbq` reads the table as of a fixed time, and the query contains `FOR SYSTEM_TIME AS OF '2026-10-06T22:18:55...'`. That gives your session a consistent view if the table changes while you work. See [time travel](https://docs.querylab.io/bigquery/time-travel/). ## Nothing runs until you ask for a result bigframes defers execution. These calls run a query: `to_pandas()`, `to_pandas_batches()`, `peek()`, `head()` when its result is shown, and `print(df)` or the notebook display of a frame (its `repr`, which calls `head()`). Setting `bpd.options.display.repr_mode = "deferred"` stops a bare `df` in a notebook from running a query and billing you for it. Because queries are deferred, the same expensive step can run again in every later query that depends on it. `df.cache()` stores the result in a temporary table so later work reuses it. That table is billed as storage. To see what a pull would process before you do it, ask for a dry run: ```python df.to_pandas(dry_run=True) # returns stats including totalBytesProcessed, runs no query ``` In v2 the methods that return results to your machine enforce BigQuery's maximum response size by default (`allow_large_results=False`). Set it to `True` per call or in `bpd.options.bigquery` if you pull large results, and expect a temporary table to be used then. See [Query result limits](https://docs.querylab.io/bigquery/quotas-and-limits/). ## Ordering mode sets filter cost pandas has row order and a row index. BigQuery tables have neither, so bigframes has to make them up. The default, `ordering_mode = "strict"`, builds a total order. The SQL bigframes generates for it hashes every column of every row with `FARM_FINGERPRINT`. Google says this can "cause a full data scan that ignores row and column filters". We measured it on `google_trends.top_terms`, a 3.2 GB table partitioned by `refresh_date`: | Code | Strict (default) | Partial | | -------------------------------------------------------------------------- | ----------------------------------------------- | ---------------- | | `read_gbq(table)`, then filter on `refresh_date` and select 2 columns | 3,186,953,758 bytes (the whole table) | 48,123,180 bytes | | `read_gbq(table, filters=[("refresh_date", "==", day)])`, select 2 columns | 98,500,383 bytes (that partition, every column) | 48,123,180 bytes | All numbers are dry-run `totalBytesProcessed`. The table is refreshed daily, so the sizes will differ when you try it, but the shape holds. Strict mode reads every column of the rows it keeps, because the order is a hash of all of them. Partial mode reads two columns, and prunes the partition from a filter written after `read_gbq`. Switch to partial mode before anything else: ```python bpd.options.bigquery.ordering_mode = "partial" ``` What partial mode takes away: `iloc`, the sequential index, `drop_duplicates()` and `unique()`, and implicit joins by index (use `merge`). `head()` fails with `OrderRequiredError: Op head requires an ordering` until you call `sort_values`, and a `head()` on an unordered frame can return different rows each time. Use `peek(n)` for a quick look: it returns n rows without ordering, and it is cheaper than `head()`. For a table with `require_partition_filter` there is one more rule. We tried `read_gbq` on a partitioned table that requires a filter, in both modes, without the `filters=` argument, and BigQuery rejected the call: "Cannot query over table ... without a filter over column(s) 'datehour' that can be used for partition elimination". bigframes runs a query when it reads the table, so a filter written afterwards comes too late. Pass `filters=` to `read_gbq` for those tables. See [partitioning](https://docs.querylab.io/bigquery/partitioning/). ## Every call is a job, and it is labelled Everything bigframes does is a job in your project, with the usual billing: bytes on on-demand, slots on a reservation (see [how queries are billed](https://docs.querylab.io/bigquery/how-queries-are-billed/)). We looked at the jobs from a short session and found two things worth knowing: - Jobs carry labels such as `bigframes-api: dataframe-cache` and `recent-bigframes-api-0: dataframe-__getitem__`, naming the API call behind the job. They show up in `INFORMATION_SCHEMA.JOBS`, which makes it easy to see which Python calls cost money. See [labels and billing export](https://docs.querylab.io/bigquery/labels-and-billing-export/). - Session setup runs small `SELECT 1` jobs. They process nothing. ## Sessions and temporary tables A bigframes **session** holds the metadata for your frames and is tied to one location. Each frame belongs to a session, and mixing frames from two sessions fails with `Cannot use combine sources from multiple sessions`, even if both sessions have the same settings. `bpd.options.bigquery` configures a default global session; call `bpd.close_session()` to reset its project or location. The default location is the US multi-region, and reading a table sets it to the table's location when you haven't set one. Intermediate data goes into the hidden dataset BigQuery creates for query results (its name starts with an underscore). In our run, `cache()` wrote a table named `__bqdf_` there, and the dataset's default table and partition expiration was 1 day, with the cache table expiring after exactly 1 day. Google's pricing note says these tables persist for seven days by default. We saw one day, so check the expiration on yours, and don't assume the table is gone sooner. Either way, you pay storage for them until they expire. See [temporary tables](https://docs.querylab.io/bigquery/temporary-tables/). ## Machine learning `bigframes.ml` has scikit-learn style classes (`LinearRegression`, `KMeans`, `PCA`, preprocessing transformers, pipelines). They are a wrapper over BigQuery ML. We fit a `LinearRegression` on the penguins table and looked at the jobs: | Python call | What BigQuery ran | Billed | | ------------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------- | | `fit(X, y)` | a `SELECT` that prepares the training data, then `CREATE OR REPLACE MODEL ... model_type='LINEAR_REG'` | 10 MiB for each of the two jobs (the per-table minimum) | | `predict(X)` | `SELECT * FROM ML.PREDICT(MODEL ..., (...))` | 20 MiB | | `score(X, y)` | `SELECT * FROM ML.EVALUATE(MODEL ..., (...))` | 20 MiB | The model is a real BigQuery ML model, stored in the same hidden dataset. Model training prices and region availability are BigQuery ML's, since that is what runs. See [BigQuery ML](https://docs.querylab.io/bigquery/bigquery-ml/). `bigframes.bigquery` exposes SQL functions that pandas doesn't have (`array_agg`, `struct`, `unix_micros`, `sql_scalar` for an arbitrary SQL expression, and the generative AI functions under `bigframes.bigquery.ai`). See [generative AI functions](https://docs.querylab.io/bigquery/ai-functions/). ## Python functions you write yourself pandas lets you `apply` any Python function. BigQuery can't run arbitrary Python inline, so bigframes has two ways to deploy one: - **`@bpd.remote_function`** creates a Cloud Run function, a BigQuery connection (by default `bigframes-default-connection`) and a BigQuery remote function that calls it. It needs several APIs enabled (Cloud Run Admin, Cloud Functions, Artifact Registry, Cloud Build, Compute Engine, BigQuery Connection, and more) and strong IAM roles. You pay for Cloud Run and the other services used. You must clean up the functions yourself: they aren't removed when the session ends. See [remote functions](https://docs.querylab.io/bigquery/remote-functions/). - **`@bpd.udf`** (Preview) creates a persistent BigQuery Python UDF routine. The function must be self-contained (no imports or variables from outside its body), must work with Python 3.11, and its code is visible to anyone with read access on the routine. Renaming a variable re-creates it. Since version 2.0, deployed remote functions no longer use the Compute Engine service account by default, and their ingress is set to internal only. Supply `cloud_function_service_account` to control permissions. ## Which to choose | You have | Use | | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | | A large BigQuery table and a pandas habit | `bigframes.pandas`, in partial ordering mode | | A few thousand rows to analyse locally | `client.query(sql).to_dataframe()` or `to_pandas()` after a filter | | A big result that must land in a Python process | the [Storage Read API](https://docs.querylab.io/bigquery/storage-read-api/) (the BigQuery client library uses it for `to_dataframe`) | | SQL you already know | SQL, written directly | | A model trained on BigQuery data | `bigframes.ml`, or `CREATE MODEL` directly | Open the playground at https://app.querylab.io?playground=true&utm_source=docs&utm_medium=concepts to run it. It is the real app on public BigQuery datasets and needs no sign-in. ## Sources Based on the BigQuery documentation by Google, CC BY 4.0, rewritten with examples checked against BigQuery: - [Introduction to BigQuery DataFrames (Last updated 2026-10-05)](https://cloud.google.com/bigquery/docs/bigquery-dataframes-introduction) - [Optimize BigQuery DataFrames performance (Last updated 2026-10-05)](https://cloud.google.com/bigquery/docs/dataframes-performance) - [Manage BigQuery DataFrames sessions and I/O (Last updated 2026-10-05)](https://cloud.google.com/bigquery/docs/dataframes-sessions-io) - [BigQuery DataFrames ML and AI (Last updated 2026-10-05)](https://cloud.google.com/bigquery/docs/dataframes-ml-ai) - [Custom Python functions for BigQuery DataFrames (Last updated 2026-10-05)](https://cloud.google.com/bigquery/docs/dataframes-custom-python-functions) - [Migrate to BigQuery DataFrames version 2.0 (Last updated 2026-10-05)](https://cloud.google.com/bigquery/docs/migrate-dataframes)