Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hadoop issue in installation and configuration

hadoop issue in installation and configuration

New Contributor

After installing Hadoop when I am trying to start start-dfs.sh it is showing following error message. I have searched a lot and found that WARN is because I am using UBUNTU 64bit OS and Hadoop is compiled against 32bit. So its not an issue to work on. But the Incorrect configuration is something I am worried about. And also not able to start the primary and secondary namenodes.

sameer@sameer-Compaq-610:~$ start-dfs.sh
15/07/27 07:47:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
localhost: ssh: connect to host localhost port 22: Connection refused
localhost: ssh: connect to host localhost port 22: Connection refused
Starting secondary namenodes [0.0.0.0]
0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused**
15/07/27 07:47:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

My current configuration: hdfs-site.xml

<configuration>

 <property>
   <name>dfs.replication</name>
   <value>1</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/sameer/mydata/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/sameer/mydata/hdfs/datanode</value>
 </property>

</configuration>

core-site.xml
<configuration>
   <property>
      <name>fs.default.name </name>
      <value> hdfs://localhost:9000 </value> 
   </property>
</configuration>

yarn-site.xml
<configuration>

<!-- Site specific YARN configuration properties -->
   <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value> 
   </property>
   <property>
      <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
      <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>
</configuration>

mapred-site.xml

<configuration>
   <property> 
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
   </property>
</configuration>

Please find what I am doing wrong in configuration or somewhere else.?

Thanks,

Nicolewells

1 REPLY 1
Highlighted

Re: hadoop issue in installation and configuration

New Contributor

Try these configs.

hdfs-site.xml

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

-- <value>file:/home/sameer/mydata/hdfs/namenode</value> -> Wrong

<value>/data/hdfs/namenode</value> // Do not use "/home" directory path for hdfs service. That because /home/sameer directory permission is 700, and it'll cause permission issues. // And normally many services which are deploying in Linux or Ubuntu are using local file option with "file:///SOME_PATH"

</property>

<property>

<name>dfs.datanode.data.dir</name>

-- <value>file:/home/sameer/mydata/hdfs/datanode</value> -> Wrong

<value>/data/hdfs/datanode</value>

</property>

</configuration>

core-site.xml

<configuration>

<property>

<name>fs.default.name </name>

-- <value> hdfs://localhost:9000 </value> -> Wrong

<value> hdfs://NAMENODE_SERVER_FQDN:9000 </value> // fs.default.name is deprecated since that hadoop v2 as I know. // And I'm not recommend using localhost value for hostname. Just use FQDN even if it's a single node.

</property>

</configuration>

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property> <- Add

<name>yarn.web-proxy.address</name>

<value>YARN_SERVER_FQDN:8089</value> // Installed resourcemanager's fqdn in server.

</property>

<property> <- Add <name>yarn.resourcemanager.address</name>

<value>YARN_SERVER_FQDN:8032</value> // Installed resourcemanager's fqdn in server.

</property>

</configuration>

mapred-site.xml

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name> <- Add this config for jobhistoryserver

<value>NAMENODE_SERVER_FQDN:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name> <- Add this config for jobhistoryserver

<value>NAMENODE_SERVER_FQDN:19888</value>

</property>

<property>

<name>mapreduce.jobhistory.intermediate-done-dir</name> <- Add this config for jobhistoryserver

<value>/mr-history/tmp</value> // It is temporarily directory in HDFS for MR Jobs.

</property>

<property>

<name>mapreduce.jobhistory.done-dir</name> <- Add this config for jobhistoryserver

<value>/mr-history/done</value> // It is finished MR Jobs directory in HDFS.

</property>

</configuration>

Don't have an account?
Coming from Hortonworks? Activate your account here