GCLOUD_DATAPROC_JOBS_SUBMIT_PIG(1)

NAME

gcloud dataproc jobs submit pig - submit a Pig job to a cluster

SYNOPSIS

gcloud dataproc jobs submit pig --cluster CLUSTER [--async] [--bucket BUCKET] [--continue-on-failure] [--driver-log-levels [PACKAGE=LEVEL,...]] [--execute QUERY, -e QUERY] [--file FILE, -f FILE] [--jars [JAR,...]] [--params [PARAM=VALUE,...]] [--properties [PROPERTY=VALUE,...]] [GLOBAL-FLAG ...]

DESCRIPTION
Submit a Pig job to a cluster.

REQUIRED FLAGS
--cluster CLUSTER
    The Dataproc cluster to submit the job to.

OPTIONAL FLAGS
--async
    Does not wait for the job to run.
--bucket BUCKET
    The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.
--continue-on-failure
    Whether to continue if a single query fails.
--driver-log-levels [PACKAGE=LEVEL,...]
    A list of package to log4j log level pairs to configure driver logging. For example: root=FATAL,com.example=INFO
--execute QUERY, -e QUERY
    A Pig query to execute as part of the job.
--file FILE, -f FILE
    HCFS URI of file containing Pig script to execute as the job.
--jars [JAR,...]
    Comma separated list of jar files to be provided to Pig and MR. May contain UDFs.
--params [PARAM=VALUE,...]
    A list of key value pairs to set variables in the Pig queries.
--properties [PROPERTY=VALUE,...]
    A list of key value pairs to configure Pig.

GLOBAL FLAGS
Run $ gcloud help for a description of flags available to all commands.

EXAMPLES
To submit a Pig job with a local script, run:
    $ gcloud dataproc jobs submit pig --cluster my_cluster \
        --file my_queries.pig
To submit a Pig job with inline queries, run:
    $ gcloud dataproc jobs submit pig --cluster my_cluster \
        -e "LNS = LOAD 'gs://my_bucket/my_file.txt' AS (line)" \
        -e \
        "WORDS = FOREACH LNS GENERATE FLATTEN(TOKENIZE(line)) AS word" \
        -e "GROUPS = GROUP WORDS BY word" \
        -e "WORD_COUNTS = FOREACH GROUPS GENERATE group, COUNT(WORDS)" \
        -e "DUMP WORD_COUNTS"