Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Started testing CDM on AWS ec2 single node but getting Errors

Started testing CDM on AWS ec2 single node but getting Errors

New Contributor

Hello,

 

I set up a single Node ec2 instance to run Cloudera Manger. The installation seemed to complete fine except for this on HDFS:

 

Bad : 718 under replicated blocks in the cluster. 720 total blocks in the cluster. Percentage under replicated blocks: 99.72%. Critical threshold: 40.00%.

This is the error message I get when I run pyspark:

ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Required executor memory (1024+384     
MB) is above the max threshold (1024 MB) of this cluster! Please check 
the values of 'yarn.scheduler.maximum-allocation-mb' and/or 
'yarn.nodemanager.resource.memory-mb'.

I'm not sure where to start looking into, but if anybody can direct me inthe right direction I would appareciate it.

 

sparke_error.png

 

This is my dashboard:

Screen Shot 2017-09-20 at 9.18.46 AM.png

I based my settings on this tutorial by Hue.

Thanks!

 

 

 

1 REPLY 1
Highlighted

Re: Started testing CDM on AWS ec2 single node but getting Errors

Champion

@Haduugen

 

1. You can ignore the under replicated blocks error. Because by default, it will expect 3 replication but it is not possible with your single node cluster. (or) you can change replication to 1 and restart the cluster. It may help you

 

2. Go to Yarn -> configuration -> increase the "yarn.nodemanager.resource.memory-mb " , restart yarn and try again, it may help you