GCLOUD_DATAPROC_JOBS_SUBMIT_PYSPARK(1)

NAME

gcloud dataproc jobs submit pyspark - submit a PySpark job to a cluster

SYNOPSIS

gcloud dataproc jobs submit pyspark PY_FILE --cluster CLUSTER [--archives [ARCHIVE,...]] [--async] [--bucket BUCKET] [--driver-log-levels [PACKAGE=LEVEL,...]] [--files [FILE,...]] [--properties [PROPERTY=VALUE,...]] [--py-files [PY_FILE,...]] [GLOBAL-FLAG ...] [JOB_ARGS ...]

DESCRIPTION
Submit a PySpark job to a cluster.

POSITIONAL ARGUMENTS
PY_FILE
    The main .py file to run as the driver.
[JOB_ARGS ...]
    The arguments to pass to the driver.

REQUIRED FLAGS
--cluster CLUSTER
    The Dataproc cluster to submit the job to.

OPTIONAL FLAGS
--archives [ARCHIVE,...]
    Comma separated list of archives to be provided to the job. must be one of the following file formats: .zip, .tar, .tar.gz, or .tgz.
--async
    Does not wait for the job to run.
--bucket BUCKET
    The Cloud Storage bucket to stage files in. Defaults to the cluster's configured bucket.
--driver-log-levels [PACKAGE=LEVEL,...]
    A list of package to log4j log level pairs to configure driver logging. For example: root=FATAL,com.example=INFO
--files [FILE,...]
    Comma separated list of files to be provided to the job.
--properties [PROPERTY=VALUE,...]
    A list of key value pairs to configure PySpark.
--py-files [PY_FILE,...]
    Comma separated list of Python files to be provided to the job.Must be one of the following file formats" .py, ,.zip, or .egg

GLOBAL FLAGS
Run $ gcloud help for a description of flags available to all commands.

EXAMPLES
To submit a PySpark job with a local script, run:
    $ gcloud dataproc jobs submit pyspark --cluster my_cluster \
        my_script.py
To submit a Spark job that runs a script that is already on the cluster, run:
    $ gcloud dataproc jobs submit pyspark --cluster my_cluster \
        file:///usr/lib/spark/examples/src/main/python/pi.py 100