Google Cloud Dataproc API

Name: Cloud Dataproc API
Brand: Cloud Dataproc API
Availability: InStock

✓ Official Vendor SpecAnalyticsData Pipelinesoauth265 EndpointsREST

For Agents

Provision and operate Hadoop and Spark clusters, submit jobs, and run serverless Spark batches in Google Cloud. Lets agents drive analytics workloads without managing infrastructure by hand.

Install Jentic One Beta

Connect the Cloud Dataproc API to your agent

Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the Cloud Dataproc API, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.

Two steps, two machines. Install the instance in a safe environment, then register your agent from wherever it runs.

Step 1: Jentic One Host machine

# On the machine that will host your Jentic One instance:
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh

Step 2: Agent machine

# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh
jentic register       # connects your agent to your Jentic One instance

Jentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.

Capabilities

What an agent can do with Cloud Dataproc API.

Provision Dataproc clusters with custom machine types, autoscaling, and initialization actions

Submit Spark, PySpark, Hive, Pig, Presto, and SparkR jobs to a cluster

Run serverless Spark batches without standing up a cluster

GET STARTED

Start building with Cloud Dataproc API

Explore with Jentic

View OpenAPI Document

Use Cases

Patterns agents use Cloud Dataproc API for, with concrete tasks.

★ On-Demand Spark Cluster Workloads

Spin up a Dataproc cluster, submit Spark or PySpark jobs against Cloud Storage data, and tear the cluster down once the job finishes. The API exposes clusters create, jobs submit, and clusters delete in a tight loop suitable for ephemeral analytics. Cluster startup is typically 90 seconds; deleting the cluster after the job completes keeps cost bounded to job runtime.

Create cluster 'etl-2026-06-10', submit a PySpark job with mainPythonFileUri gs://acme/jobs/etl.py, wait for state DONE, and delete the cluster

Serverless Spark Batches

Run Spark workloads without managing cluster infrastructure using Dataproc Batches. POST /v1/{+parent}/batches accepts a Spark, PySpark, SparkR, or SparkSQL payload plus runtime config and a Cloud Storage staging bucket. The service provisions transient infrastructure for the run and bills only for the batch's runtime.

POST /v1/{+parent}/batches with a PySpark batch pointing at gs://acme/jobs/score.py and runtimeConfig version 2.2, then poll the batch resource until state is SUCCEEDED

Multi-Step Workflow Templates

Define a Dataproc workflow template that creates a managed cluster, runs a sequence of jobs (with dependencies), and deletes the cluster on completion. Workflow templates support parameterised execution so the same template runs against different inputs. Use the instantiate or instantiateInline endpoints to start a run from CI or an agent.

Instantiate workflow template 'nightly-etl' with parameter input_date=2026-06-09 and watch the resulting Operation until it reaches DONE

AI Agent Analytics Workload Operator

An AI agent can run Spark workloads on demand through Jentic without operator-written cluster code. Jentic search returns the matching cluster, job, batch, or workflow operation, the agent loads the schema, and Jentic executes against dataproc.googleapis.com using vault-stored credentials. This collapses the multi-day setup of Dataproc OAuth and operation polling into a single agent run.

Use Jentic to search 'submit a pyspark job to dataproc', load the submit schema, and execute it against the named cluster with the provided main file URI

Key Endpoints

65 endpoints — cloud dataproc is google's managed hadoop and spark service for running batch, streaming, and interactive analytics workloads.

METHOD

PATH

DESCRIPTION

POST

/v1/projects/{projectId}/regions/{region}/clusters

Create a Dataproc cluster

POST

/v1/projects/{projectId}/regions/{region}/clusters/{clusterName}:diagnose

Diagnose a cluster and produce a diagnostic tarball

POST

/v1/projects/{projectId}/regions/{region}/jobs

Submit a job to a cluster

POST

/v1/{+parent}/batches

Run a serverless Spark batch

POST

/v1/{+parent}/workflowTemplates

Create a workflow template

POST

/v1/projects/{projectId}/regions/{region}/clusters/{clusterName}:stop

Stop a running cluster

POST

/v1/projects/{projectId}/regions/{region}/clusters

Create a Dataproc cluster

POST

/v1/projects/{projectId}/regions/{region}/clusters/{clusterName}:diagnose

Diagnose a cluster and produce a diagnostic tarball

POST

/v1/projects/{projectId}/regions/{region}/jobs

Submit a job to a cluster

POST

/v1/{+parent}/batches

Run a serverless Spark batch

POST

/v1/{+parent}/workflowTemplates

Create a workflow template

Why Jentic?

Three things that make agents converge on Jentic-routed access.

Credential isolation

Google OAuth client secrets and refresh tokens are stored encrypted in the Jentic vault. Agents receive scoped, short-lived access tokens for dataproc.googleapis.com; raw credentials never enter the agent context.

Intent-based discovery

Agents search Jentic by intent (e.g. 'submit a pyspark job to dataproc') and Jentic returns the matching operation with its input schema, so the agent calls the right endpoint without browsing the discovery doc.

Time to first call

Direct Dataproc integration: 2-5 days for OAuth, cluster-config schema work, and operation polling. Through Jentic: under 1 hour.

FAQs

Specific to using Cloud Dataproc API through Jentic.

What authentication does the Cloud Dataproc API use?

The Cloud Dataproc API uses OAuth 2.0 with the cloud-platform scope. Through Jentic the OAuth client and refresh tokens are stored in the Jentic vault and the agent receives short-lived scoped access tokens, so raw Google credentials never enter the agent context.

Can I run serverless Spark with the Cloud Dataproc API?

Yes. POST /v1/{+parent}/batches submits a PySpark, Spark, SparkR, or SparkSQL workload with runtime and environment configuration; Dataproc provisions transient infrastructure for the duration of the run and tears it down automatically.

What are the rate limits for the Cloud Dataproc API?

Google enforces standard Cloud quotas on dataproc.googleapis.com: per-project rate limits on read/write calls plus quotas on concurrent clusters, jobs, and batches per region. Quotas are visible in the Cloud Console under IAM and admin, quotas, filtered to dataproc.googleapis.com.

How do I submit a PySpark job through Jentic?

Search Jentic for 'submit a pyspark job to dataproc', load the schema for POST /v1/projects/{projectId}/regions/{region}/jobs, and execute with placement.clusterName and pysparkJob.mainPythonFileUri set. Jentic returns the Job resource with its job ID for status polling.

Is the Cloud Dataproc API free?

API calls are free; clusters, batches, and sessions are billed by underlying Compute Engine vCPU and memory plus a per-vCPU-hour Dataproc premium. Batches and Sessions are billed per runtime second only, with no charge while idle.

How do I diagnose a failing cluster?

Call POST /v1/projects/{projectId}/regions/{region}/clusters/{clusterName}:diagnose to produce a diagnostic tarball in Cloud Storage. The response contains the gs:// path of the tarball, which holds master and worker logs plus a YARN dump for offline analysis.

Google Cloud Dataproc API

Connect the Cloud Dataproc API to your agent

Capabilities

Start building with Cloud Dataproc API

Use Cases

Key Endpoints

Why Jentic?

Related APIs

FAQs