Member since
05-03-2017
37
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2689 | 08-09-2017 12:05 AM |
08-10-2017
12:01 AM
Hello, Is this article: http://blog.cloudera.com/blog/2012/09/automating-your-cluster-with-cloudera-manager-api/ In the step3 ("configuring HDFS") I just get errors and errors. My conditions matched ok with this new configuration. Any question let me know
... View more
08-09-2017
12:05 AM
I got a solution. for group in dn_groups: group.update_config({'dfs_data_dir_list': '/dfs'})
... View more
08-08-2017
08:24 AM
There is any way to specify or skip the services dependencies? Name of the ZooKeeper service that this YARN service instance depends on Name of the HDFS service that this YARN service instance depends on When I am starting a cluster JUST using API, this is one error that I've found on YARN Service, and I can't format the NameNode just because of this. Any ideas? Thanks
... View more
Labels:
- Labels:
-
Cloudera Manager
08-08-2017
03:44 AM
hi, I am working with python and cm_api. I got the following response when I execute my script. I copied and paste the code from the cloudera blog This is the code: hdfs_service_config = {
'dfs_replication': 2,
}
nn_config = {
'dfs_name_dir_list': '/dfs/nn',
'dfs_namenode_handler_count': 30,
}
snn_config = {
'fs_checkpoint_dir_list': '/dfs/snn',
}
dn_config = {
'dfs_data_dir_list': '/dfs/dn1,/dfs/dn2,/dfs/dn3',
'dfs_datanode_failed_volumes_tolerated': 1,
}
hdfs.update_config(
svc_config=hdfs_service_config,
NAMENODE=nn_config,
SECONDARYNAMENODE=snn_config,
DATANODE=dn_config) cm-api (14.0.0) Any ideas?
... View more
Labels:
- Labels:
-
Cloudera Manager
-
HDFS
08-07-2017
02:58 AM
After adding the "classpath" when I execute, hadoop classpath, it doesn't return the path that I have added. Any ideas?
... View more
08-02-2017
11:55 PM
good spot! This is the output: jar -tf /opt/spark/yarn/spark-2.1.0-yarn-shuffle.jar | grep -i 'YarnShuffleService' org/apache/spark/network/yarn/YarnShuffleService$AppId.class org/apache/spark/network/yarn/YarnShuffleService.class
... View more
08-02-2017
07:08 AM
After add <property>
<name>yarn.nodemanager.aux-services</name>
<value>spark_shuffle,mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property> I am getting the following error: 2017-08-02 13:52:06,207 FATAL org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Failed to initialize spark_shuffle
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2234)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:236)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:318)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:562)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:609)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2202)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2226)
... 10 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2108)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2200) I have in my CLASSPATH the library need it <property>
<name>yarn.application.classpath</name>
<value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*,/opt/spark/yarn/spark-2.1.0-yarn-shuffle.jar</value>
</property>
<property> Any ideas?
... View more
Labels:
- Labels:
-
Apache Spark
-
Apache YARN
07-17-2017
12:26 AM
Is not possible to upgrade to hadoop 2.7/2.8 form 2.6? 2.6 is from November 2014! Regards
... View more
Labels:
- Labels:
-
Apache Spark