Google BigQuery Data Transfer API

Name: BigQuery Data Transfer API
Brand: BigQuery Data Transfer API
Availability: InStock

✓ Official Vendor SpecAnalyticsData Pipelinesoauth214 EndpointsREST

For Agents

Schedule and monitor recurring data loads into BigQuery from SaaS apps, AWS, and Cloud Storage. Agents can create transferConfigs, trigger backfills, and read run history.

Install Jentic One Beta

Connect the BigQuery Data Transfer API to your agent

Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the BigQuery Data Transfer API, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.

Two steps, two machines. Install the instance in a safe environment, then register your agent from wherever it runs.

Step 1: Jentic One Host machine

# On the machine that will host your Jentic One instance:
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh

Step 2: Agent machine

# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh
jentic register       # connects your agent to your Jentic One instance

Jentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.

Capabilities

What an agent can do with BigQuery Data Transfer API.

Enroll a project into the available data sources catalog before creating transfers

Create transferConfigs that pull from Google Ads, YouTube, S3, or Cloud Storage on a schedule

Trigger on-demand backfills and ad-hoc runs for a transferConfig

GET STARTED

Start building with BigQuery Data Transfer API

Explore with Jentic

View OpenAPI Document

The BigQuery Data Transfer Service automates recurring data loads into BigQuery from SaaS applications (Google Ads, Campaign Manager, YouTube, Search Ads), other clouds (S3, Azure Blob), and Cloud Storage. The API exposes data sources, transfer configurations, runs, and transfer logs so you can enroll sources, schedule transfers, backfill historical windows, and monitor failures. Each transferConfig holds a schedule, destination dataset, and source-specific parameters. This is the right plane when the use case is 'land this source in BigQuery on a cadence' rather than ad-hoc loads.

Use Cases

Patterns agents use BigQuery Data Transfer API for, with concrete tasks.

★ Marketing Data Lake Population

Schedule daily transfers from Google Ads, Campaign Manager, and YouTube into a BigQuery marketing dataset so the warehouse always has yesterday's spend and impression data. transferConfigs are managed at /v1/{parent}/transferConfigs with schedule, destinationDatasetId, and a per-source params map. Failed runs surface in the runs and transferLogs collections for alerting.

Create a daily Google Ads transferConfig that lands account 123-456-7890 into dataset marketing_raw at 04:00 UTC.

Cross-Cloud Ingestion

Pull files from Amazon S3 or Azure Blob Storage into BigQuery on a schedule using the s3 and azure_blob_storage data sources. The transfer service handles auth, file pattern matching, and write disposition. This avoids running custom Cloud Functions for every cross-cloud pipeline.

Create an hourly S3 transferConfig that copies s3://logs/year=*/month=*/day=*/*.json into dataset raw_logs in project myproj.

Backfill and Replay

Trigger backfills via /v1/{parent}:scheduleRuns when source data is corrected or a transfer was paused. The runs collection records each execution with start/end time and state, and transferLogs explains why a run failed. Operators can replay arbitrary windows without manual SQL.

Schedule backfill runs for transferConfig projects/myproj/locations/US/transferConfigs/abc123 between 2026-06-01 and 2026-06-09.

Failure Monitoring

List runs filtered to states like FAILED and pull /v1/{name}/transferLogs for each, then forward summaries to a paging system. The transferConfig itself records lastModifiedTime and emailPreferences so on-call can be notified directly. This replaces hand-rolled monitoring of cron-driven loads.

List all transfer runs in project myproj location US in state FAILED in the last 24h and return their config name and error message.

Agent-Driven Pipeline Setup

Let an AI agent take a request like 'load our HubSpot contacts into BigQuery every morning' and create the matching transferConfig through Jentic. The agent picks the data source, configures parameters, and confirms credentials with checkValidCreds before scheduling. OAuth tokens stay in Jentic's vault.

From the request 'pull S3 logs into BigQuery hourly', create the transferConfig with the supplied bucket and credentials, run checkValidCreds, then schedule a backfill for the past 24h.

Key Endpoints

14 endpoints — the bigquery data transfer service automates recurring data loads into bigquery from saas applications (google ads, campaign manager, youtube, search ads), other clouds (s3, azure blob), and cloud storage.

METHOD

PATH

DESCRIPTION

POST

/v1/{+parent}/transferConfigs

Create a new transferConfig

GET

/v1/{+parent}/transferConfigs

List transferConfigs in a project and location

PATCH

/v1/{+name}

Update a transferConfig

DELETE

/v1/{+name}

Delete a transferConfig

POST

/v1/{+parent}:scheduleRuns

Schedule backfill runs for a transferConfig

POST

/v1/{+name}:checkValidCreds

Validate source credentials before saving

GET

/v1/{+parent}/runs

List runs of a transferConfig

GET

/v1/{+parent}/transferLogs

List transfer log lines for a run

POST

/v1/{+parent}/transferConfigs

Create a new transferConfig

GET

/v1/{+parent}/transferConfigs

List transferConfigs in a project and location

PATCH

/v1/{+name}

Update a transferConfig

DELETE

/v1/{+name}

Delete a transferConfig

POST

/v1/{+parent}:scheduleRuns

Schedule backfill runs for a transferConfig

Why Jentic?

Three things that make agents converge on Jentic-routed access.

Credential isolation

Google OAuth 2.0 client credentials and refresh tokens are stored encrypted in the Jentic vault (MAXsystem). Source credentials embedded in a transferConfig (S3 keys, Ads tokens) flow through Google's API and are not persisted by Jentic.

Intent-based discovery

Agents search Jentic with intents like 'schedule a BigQuery transfer from Google Ads' and Jentic returns the create transferConfig operation with the data source's specific parameter schema.

Time to first call

Direct integration: 1-2 days for OAuth, source enrollment, and per-source params. Through Jentic: under an hour — search, load schema, execute.

FAQs

Specific to using BigQuery Data Transfer API through Jentic.

What authentication does the BigQuery Data Transfer API use?

The API uses Google OAuth 2.0 with the cloud-platform and bigquery scopes. Source-side credentials (Ads accounts, S3 keys) are saved on each transferConfig. Through Jentic, the OAuth refresh token is stored encrypted; only short-lived access tokens reach the agent.

Can I create scheduled transfers from Google Ads with this API?

Yes. POST /v1/{parent}/transferConfigs with dataSourceId set to google_ads, the destinationDatasetId, schedule (e.g., 'every 24 hours'), and params containing the Ads customer_id creates a recurring load. The first run kicks off at the next scheduled slot.

What are the rate limits for the BigQuery Data Transfer API?

Quotas are project-level: typically a few thousand transferConfigs per project and modest write QPS on the same config. Run history is retained for 90 days. Check the Google Cloud quotas page for the values applied to your project.

How do I trigger a backfill through Jentic?

Search Jentic for 'backfill a BigQuery transfer', load the schema for POST /v1/{parent}:scheduleRuns, and execute with startTime and endTime in RFC3339. Jentic forwards the OAuth token, and the runs collection will show the new BACKFILL runs.

Is the BigQuery Data Transfer API free?

The API itself has no surcharge, but the data it loads is billed at the standard BigQuery storage and (for some sources) ingestion rate. Some SaaS connectors are also subject to per-source quotas. Storage in the destination dataset is the main ongoing cost.

Can I pause a transfer without deleting it?

Yes. PATCH /v1/{name} with disabled=true (and updateMask=disabled) pauses the schedule. The transferConfig stays in place; resume by patching back to disabled=false.

Google BigQuery Data Transfer API

Connect the BigQuery Data Transfer API to your agent

Capabilities

Start building with BigQuery Data Transfer API

Use Cases

Key Endpoints

Why Jentic?

Related APIs

FAQs