Support Questions
Find answers, ask questions, and share your expertise

unable to run a job with spark-submit pyspark

unable to run a job with spark-submit pyspark

Expert Contributor

I am new to pyspark and trying to run a job which gets a count in a table.

Say, select count(*) from students;

I have enabled hivecontext in sqlcontext.

When I submit the job using below

spark-submit --master yarn --deploy-mode cluster --driver-memory 1g --files /etc/spark/hive-site.xml

I am getting the below error

AM Host : N/A
Aggregate Resource Allocation : 0 MB-seconds, 0 vcore-seconds
Log Aggregation Status : NOT_START

 Application is Activated, waiting for resources to be assigned for AM.  Details : AM Partition = <DEFAULT_PARTITION> ; Partition Resource = <memory:215040, vCores:114> ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 100.0 % ; Queue's Absolute max capacity = 100.0 % ;
        Unmanaged Application : false
        Application Node Label Expression : <Not set>
        AM container Node Label Expression : <DEFAULT_PARTITION>

Could you please help.