BigQuery sharing (Analytics Hub) explained

BigQuery sharing, formerly Analytics Hub, lets you share a dataset with other projects and organizations without copying it. Subscribers get a read-only linked dataset, the publisher pays storage, and the subscriber pays for the queries they run.

BigQuery sharing is the product Google used to call Analytics Hub. A publisher puts a dataset in a listing, and a subscriber who clicks Subscribe gets a linked dataset in their own project: a read-only pointer to the publisher’s data. Nothing is copied. The subscriber queries it like any other dataset, the publisher pays for the storage, and the subscriber pays for the query. The API and IAM roles still say analyticshub.

The pieces

A shared dataset can hold tables, views, materialized views, table snapshots, external tables, routines (UDFs, table functions, SQL stored procedures), BigQuery ML models, authorized views and authorized datasets. A dataset that contains something outside that list can’t be shared. Row-level security and column-level security on the tables carry over to subscribers.

The same machinery shares Pub/Sub topics for streaming data. A subscriber then gets a linked Pub/Sub subscription. The rest of this article is about datasets.

What a subscriber sees

Subscribing creates a dataset in the subscriber’s project and region of choice. Tables and views in it can be queried with the usual three-part name:

SELECT * FROM `my-project`.`partner_demand_forecast`.`forecast_daily` LIMIT 10;

The data is live. A row the publisher inserts is visible to the next subscriber query, because there is only one copy. Compare snapshots and clones, which are copies of a point in time.

What a subscriber can’t do:

• Add, change or delete anything in a linked dataset. Only the description and labels can be edited.
• Set IAM on individual tables inside it, or attach IAM tags to them. Permissions apply to the whole linked dataset.
• Create a materialized view over a linked table, or take a snapshot of one.
• Use region-qualified INFORMATION_SCHEMA views to see the linked dataset’s metadata. The dataset-qualified form works: `partner_demand_forecast`.INFORMATION_SCHEMA.TABLES.

Google also warns that queries joining linked datasets can fail when they exceed 1 TB of physical storage, and says to contact support in that case.

If the publisher removes a subscription, the linked dataset stays in the subscriber’s project but unlinked, and can no longer be queried. Deleting a linked dataset is how a subscriber unsubscribes. It doesn’t touch the source, and subscribing again recreates it.

Who pays for what

Google states there is no extra cost for exchanges or listings. A query on a linked dataset is a job in the subscriber’s project, so it follows the subscriber’s pricing model, and the usual bytes-billed rules apply (see how queries are billed).

One edition rule catches people. If the publisher put row-level security or data masking on a listed table, subscribers must use an Enterprise or Enterprise Plus edition to run query jobs on the linked dataset. Plan for that before you put such a table in a listing.

Linked datasets can have replicas in other regions, chosen by the subscriber, and a listing can be offered in several regions if the publisher turned on cross-region replication for the shared dataset. Colocating the linked dataset with the subscriber’s own data is what makes joins possible, because a query can’t join across locations (see Locations). Google doesn’t state a separate charge for linked dataset replicas on its sharing pages. For Pub/Sub topics it says network egress applies, which is why a listing has a “data affinity” field.

Private and public

A private exchange or listing is visible only to the users and groups given a role on it. A public exchange gives the Viewer role to allAuthenticatedUsers, so every Google Cloud user can discover and subscribe to its listings.

Three rules from Google’s docs:

• Listings inherit the exchange’s setting, but a private exchange can contain public listings. A public exchange can’t contain private ones.
• A listing that is shared with allAuthenticatedUsers but not made publicly discoverable stays private. It doesn’t show in the public catalog; only people with its URL find it.
• To create a public listing, the project needs an organization and a billing account.

Public listings can be free datasets or commercial ones. A commercial listing is either “request access” (the provider contacts you and grants access offline) or sold through Cloud Marketplace, where the publisher sets price, free or trial terms and duration, and the exchange must be public.

Controlling what subscribers can take away

A linked dataset is read-only, but a subscriber can still query it and write the result somewhere. Data egress options close that gap. A publisher can turn on restrictions for the shared data, for query results, or both. With restrictions on:

• Copy, clone, export and snapshot are unavailable, in the console and the APIs.
• The BigQuery Data Transfer Service can’t read the dataset.
• CREATE TABLE AS SELECT, CREATE VIEW AS SELECT and writing to a destination table are blocked.

There are three levels in the console: no copy or export of the shared data (query results can still be exported), no copy or export of query results as well, and no table copy or export through APIs. Choose them when you create the listing.

Seeing who uses a listing

The publisher gets a usage dashboard in the console, and the same data in INFORMATION_SCHEMA.SHARED_DATASET_USAGE, one row per table touched by a subscriber job. Its columns include the job, table, rows and bytes processed, and the subscriber’s organization. We validated Google’s example query against real BigQuery with a dry run, which reads zero bytes:

SELECT
  subscriber_org_number,
  ANY_VALUE(subscriber_org_display_name) AS subscriber_org_display_name,
  SUM(total_bytes_processed) AS usage_bytes
FROM `region-us`.INFORMATION_SCHEMA.SHARED_DATASET_USAGE
GROUP BY 1;

The view identifies subscribers by organization and project. To log the user who ran each query (field job_principal_subject), the publisher must turn on subscriber email logging when creating the exchange or listing. It can’t be switched off afterwards; the only way back is to delete the exchange and create a new one.

Usage numbers have gaps. Only usage through BigQuery jobs is counted: Storage Read API reads, tabledata.list and BI Engine queries are not. Nothing is recorded for subscriptions made before 20 July 2023, and views only count for queries after 22 April 2024.

Limits

An exchange’s project and region can’t be changed after creation, and the shared dataset of a listing can’t be changed either. The dataset must be in the same region as the exchange.

Traps

• Views need access to the source. To query a plain view in a linked dataset, the subscriber needs explicit read permission on the source data. Share authorized views instead, so subscribers never need access to the base tables.
• A query plan shows the view’s SQL. Google says the plan reveals shared view and routine definitions, including project IDs and other datasets they use. Keep secrets out of shared view code.
• Schema updates reach subscribers at once, but search lags. With more than 100 subscribers or tables, the catalog index can take up to 18 hours to show changes.
• VPC Service Controls. Google recommends not sharing from or subscribing in a perimeter. If you must, add ingress and egress rules for publisher, exchange and subscriber projects.

See Access control for the IAM roles behind this: Analytics Hub Admin creates exchanges, Publisher creates listings, Subscriber subscribes, Viewer only browses.