Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

not able to launch PYSPARK CDH 5.5.0 and spark version 1.5.0

Solved Go to solution

not able to launch PYSPARK CDH 5.5.0 and spark version 1.5.0

Explorer

Hi All,

 

I am not able to launch PYSPARK in CDH 5.5.0 and spark version 1.5.0 when i type pyspark its returning

/22 00:43:33 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:34 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:35 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:36 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:37 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:38 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:39 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:40 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:41 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:42 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:43 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:44 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:45 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:46 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:47 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:48 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:49 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:50 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:51 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:52 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:53 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:54 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:55 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:56 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:57 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)
16/01/22 00:43:58 INFO Client: Application report for application_1453446402525_0004 (state: ACCEPTED)

----

 

Its not ending ..Its not launching the application

 

Can you please suggest

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: not able to launch PYSPARK CDH 5.5.0 and spark version 1.5.0

Contributor

The attached log indicates that application is accepted by cluster manager (YARN) but unable to execute due to resource crunch. Please make sure there are enough resources available in your cluster while submitting the job.

 

Do check following and configure based on hosts:

yarn.nodemanager.resource.memory-mb

yarn.nodemanager.resource.cpu-vcores

yarn.scheduler.maximum-allocation-mb
yarn.scheduler.minimum-allocation-mb

 

 yarn-scheduler.maximum-allocation-vcores

yarn.scheduler.minimum-allocation-vcores

 

 

3 REPLIES 3

Re: not able to launch PYSPARK CDH 5.5.0 and spark version 1.5.0

Explorer

We are facing the same problem. Have you been able to resolve the issue? Please can you share your findings?

Highlighted

Re: not able to launch PYSPARK CDH 5.5.0 and spark version 1.5.0

Contributor

The attached log indicates that application is accepted by cluster manager (YARN) but unable to execute due to resource crunch. Please make sure there are enough resources available in your cluster while submitting the job.

 

Do check following and configure based on hosts:

yarn.nodemanager.resource.memory-mb

yarn.nodemanager.resource.cpu-vcores

yarn.scheduler.maximum-allocation-mb
yarn.scheduler.minimum-allocation-mb

 

 yarn-scheduler.maximum-allocation-vcores

yarn.scheduler.minimum-allocation-vcores

 

 

Re: not able to launch PYSPARK CDH 5.5.0 and spark version 1.5.0

Explorer

Your tip was spot on Umesh! We realised that as a part of installing CDH 5.5 and Spark, the default configuration for the parameters you mentioned, were changed (not sure how). We restored them to the default values and were able to run Pyspark and Spark-shell.

Many thanks for your help.