GCLOUD_DATAPROC_JOBS_SUBMIT_PIG(1)

NAME

gcloud dataproc jobs submit pig - submit a Pig job to a cluster

SYNOPSIS

gcloud dataproc jobs submit pig --cluster=CLUSTER (--execute=QUERY, -e QUERY | --file=FILE, -f FILE) [--async] [--bucket=BUCKET] [--continue-on-failure] [--driver-log-levels=[PACKAGE=LEVEL,...]] [--jars=[JAR,...]] [--labels=[KEY=VALUE,...]] [--params=[PARAM=VALUE,...]] [--properties=[PROPERTY=VALUE,...]] [GLOBAL-FLAG ...]

DESCRIPTION
Submit a Pig job to a cluster.

REQUIRED FLAGS

    --cluster=CLUSTER
        The Dataproc cluster to submit the job to.

Exactly one of these must be specified:

    --execute=QUERY, -e QUERY
        A Pig query to execute as part of the job.

    --file=FILE, -f FILE
        HCFS URI of file containing Pig script to execute as the job.

OPTIONAL FLAGS

    --async
        Does not wait for the job to run.

    --bucket=BUCKET
        The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.

    --continue-on-failure
        Whether to continue if a single query fails.

    --driver-log-levels=[PACKAGE=LEVEL,...]
        A list of package to log4j log level pairs to configure driver logging. For example: root=FATAL,com.example=INFO

    --jars=[JAR,...]
        Comma separated list of jar files to be provided to Pig and MR. May contain UDFs.

    --labels=[KEY=VALUE,...]
        A list of label KEY=VALUE pairs to add.

    --params=[PARAM=VALUE,...]
        A list of key value pairs to set variables in the Pig queries.

    --properties=[PROPERTY=VALUE,...]
        A list of key value pairs to configure Pig.

GLOBAL FLAGS
Run $ gcloud help for a description of flags available to all commands.

EXAMPLES
To submit a Pig job with a local script, run:
    $ gcloud dataproc jobs submit pig --cluster my_cluster \
        --file my_queries.pig
To submit a Pig job with inline queries, run:
    $ gcloud dataproc jobs submit pig --cluster my_cluster \
        -e "LNS = LOAD 'gs://my_bucket/my_file.txt' AS (line)" \
        -e \
        "WORDS = FOREACH LNS GENERATE FLATTEN(TOKENIZE(line)) AS word" \
        -e "GROUPS = GROUP WORDS BY word" \
        -e "WORD_COUNTS = FOREACH GROUPS GENERATE group, COUNT(WORDS)" \
        -e "DUMP WORD_COUNTS"