Google Dataflow API

Name: Dataflow API
Brand: Google Cloud Dataflow API
Availability: InStock

✓ Official Vendor SpecAnalyticsData Pipelinesoauth241 EndpointsREST

For Agents

Launch Apache Beam pipelines from templates, monitor job state and worker metrics, and drain streaming jobs so an agent can run and govern Dataflow workloads.

Install Jentic One Beta

Connect the Dataflow API to your agent

Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the Dataflow API, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.

Two steps, two machines. Install the instance in a safe environment, then register your agent from wherever it runs.

Step 1: Jentic One Host machine

# On the machine that will host your Jentic One instance:
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh

Step 2: Agent machine

# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh
jentic register       # connects your agent to your Jentic One instance

Jentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.

Capabilities

What an agent can do with Dataflow API.

Launch a Dataflow job from a Flex Template or classic template with custom parameters

List jobs in a project and region with filters by state and creation time

Retrieve job graph, current state, and per-stage metrics

GET STARTED

Start building with Dataflow API

Explore with Jentic

View OpenAPI Document

Google Cloud Dataflow API is the control plane for streaming and batch Apache Beam pipelines on Google Cloud. It exposes endpoints to launch jobs from Flex or classic templates, list and inspect jobs and their workers, drain or cancel running streaming jobs, and update the parameters of in-flight pipelines. The API also surfaces job metrics, debug snapshots, and template metadata so platform teams can build dashboards and self-service launchers without bespoke Beam code. Jobs run on managed worker VMs in the chosen region, with autoscaling controlled per job.

Use Cases

Patterns agents use Dataflow API for, with concrete tasks.

★ Self-Service Pipeline Launch

A data platform team publishes Flex Templates for common ETL shapes and a portal calls the Dataflow API to launch them with user-supplied parameters. Each launch returns a job ID and immediate state, then the portal polls the job for progress. End users start a pipeline without touching gcloud or Beam code.

Launch the Flex Template at gs://templates/etl-template-spec.json in us-central1 with parameters input='gs://raw/2026-06-10' and output='bq:proj.ds.facts'.

Streaming Job Lifecycle Management

A SRE runbook drains a streaming job, deploys a new version, and snapshots the previous state for rollback. The Dataflow API exposes drain and snapshot operations so the runbook completes without manual Console steps. Drains preserve in-flight messages instead of dropping them.

Drain streaming job 'jid-2026-06-10-abc' in us-central1 and create a snapshot named 'pre-deploy-snap' before launching the new version.

Failed Job Triage Bot

A Slack bot monitors Dataflow for jobs entering JOB_STATE_FAILED and posts the job ID, error log link, and last successful checkpoint. The Dataflow API supplies job state, current workers, and error messages so the bot can surface high-signal context. Engineers click through to logs only when needed.

List jobs in us-central1 with stateFilter=ACTIVE created in the last 24 hours, return any whose currentState is JOB_STATE_FAILED with their error messages.

Capacity and Cost Telemetry

A FinOps dashboard collects per-job worker counts, vCPU hours, and shuffled data from the Dataflow API metrics endpoint and aggregates them per team. Teams see cost drivers without opening individual job pages. Decisions to right-size workers are grounded in actual Dataflow telemetry.

Get the metrics for job 'jid-2026-06-10-xyz' and return totalVcpuTime and currentNumWorkers.

AI Agent Pipeline Operator

An on-call AI agent gets a 'job lagging' alert, asks Jentic for the Dataflow API operations needed, retrieves current watermark and worker count, and drains/relaunches the job with more workers. Jentic isolates the Google service account credential so raw keys never enter the agent context.

For job 'jid-2026-06-10-zzz', get its watermark and worker count, drain if watermark lag exceeds 5 minutes, and relaunch the same Flex Template with maxWorkers doubled.

Key Endpoints

41 endpoints — google cloud dataflow api is the control plane for streaming and batch apache beam pipelines on google cloud.

METHOD

PATH

DESCRIPTION

GET

/v1b3/projects/{projectId}/jobs

List jobs in a project

GET

/v1b3/projects/{projectId}/jobs/{jobId}

Get a specific job's state and graph

POST

/v1b3/projects/{projectId}/locations/{location}/flexTemplates:launch

Launch a job from a Flex Template

POST

/v1b3/projects/{projectId}/jobs/{jobId}/debug/sendCapture

Send a debug capture for a job

GET

/v1b3/projects/{projectId}/jobs/{jobId}/debug/getConfig

Get the debug config for a job

GET

/v1b3/projects/{projectId}/jobs

List jobs in a project

GET

/v1b3/projects/{projectId}/jobs/{jobId}

Get a specific job's state and graph

POST

/v1b3/projects/{projectId}/locations/{location}/flexTemplates:launch

Launch a job from a Flex Template

POST

/v1b3/projects/{projectId}/jobs/{jobId}/debug/sendCapture

Send a debug capture for a job

GET

/v1b3/projects/{projectId}/jobs/{jobId}/debug/getConfig

Get the debug config for a job

Why Jentic?

Three things that make agents converge on Jentic-routed access.

Credential isolation

Google service account credentials are stored encrypted in the Jentic vault. Agents call dataflow.googleapis.com with short-lived OAuth access tokens with the cloud-platform scope, never the underlying JSON key.

Intent-based discovery

Agents search Jentic for 'launch dataflow pipeline' and Jentic returns the flexTemplates.launch and jobs.update operations with their full request schemas, including parameter maps and requestedState values.

Time to first call

Direct integration: 2-3 days to handle OAuth, regional endpoints, Flex Template parameter validation, and long-running operation polling. Through Jentic: under 1 hour for the same scope of pipeline orchestration.

FAQs

Specific to using Dataflow API through Jentic.

What authentication does the Google Cloud Dataflow API use?

The Dataflow API uses Google OAuth 2.0 with the cloud-platform scope. Through Jentic the service account credentials are stored encrypted in the Jentic vault and the agent receives scoped, short-lived access tokens per request.

Can I launch a Dataflow job from a Flex Template using only the Dataflow API?

Yes. Call /v1b3/projects/{projectId}/locations/{location}/flexTemplates:launch with the template GCS spec URI and a parameters map. The response contains the new job's ID and current state, and you can poll the job endpoint until it reaches JOB_STATE_RUNNING.

What are the rate limits for the Google Cloud Dataflow API?

Dataflow applies per-project quotas on jobs.create and jobs.list, plus per-region worker and vCPU quotas that limit how many concurrent jobs can run. Inspect the Cloud Console Quotas page for the precise limits in your project.

How do I drain a streaming job through Jentic?

Run pip install jentic, search Jentic for 'drain dataflow streaming job', load the schema for jobs.update on dataflow.googleapis.com, and execute it with the job ID and requestedState=JOB_STATE_DRAINED.

Does the Dataflow API let me write or compile Apache Beam code?

No. This API is a control plane: launching, monitoring, draining, and snapshotting jobs whose Beam graph is supplied as a template or compiled artifact. Beam pipeline development happens locally with the Beam SDK before publishing a template.

Google Dataflow API

Connect the Dataflow API to your agent

Capabilities

Start building with Dataflow API

Use Cases

Key Endpoints

Why Jentic?

Related APIs

FAQs