hadoop issue in installation and configuration

[#1]

After installing Hadoop when I am trying to start start-dfs.sh it is showing following error message. I have searched a lot and found that WARN is because I am using UBUNTU 64bit OS and Hadoop is compiled against 32bit. So its not an issue to work on. But the Incorrect configuration is something I am worried about. And also not able to start the primary and secondary namenodes.

sameer@sameer-Compaq-610:~$ start-dfs.sh
15/07/27 07:47:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Starting namenodes on []
localhost: ssh: connect to host localhost port 22: Connection refused
localhost: ssh: connect to host localhost port 22: Connection refused
Starting secondary namenodes [0.0.0.0]
0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused**
15/07/27 07:47:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

My current configuration: hdfs-site.xml

<configuration>

 <property>
   <name>dfs.replication</name>
   <value>1</value>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/home/sameer/mydata/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/home/sameer/mydata/hdfs/datanode</value>
 </property>

</configuration>

core-site.xml
<configuration>
   <property>
      <name>fs.default.name </name>
      <value> hdfs://localhost:9000 </value> 
   </property>
</configuration>

yarn-site.xml
<configuration>

<!-- Site specific YARN configuration properties -->
   <property>
      <name>yarn.nodemanager.aux-services</name>
      <value>mapreduce_shuffle</value> 
   </property>
   <property>
      <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
      <value>org.apache.hadoop.mapred.ShuffleHandler</value>
   </property>
</configuration>

mapred-site.xml

<configuration>
   <property> 
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
   </property>
</configuration>

Please find what I am doing wrong in configuration or somewhere else.?

Thanks,

Nicolewells

[#2]

Try these configs.

hdfs-site.xml

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

<property>

<name>dfs.namenode.name.dir</name>

-- <value>file:/home/sameer/mydata/hdfs/namenode</value> -> Wrong

<value>/data/hdfs/namenode</value> // Do not use "/home" directory path for hdfs service. That because /home/sameer directory permission is 700, and it'll cause permission issues. // And normally many services which are deploying in Linux or Ubuntu are using local file option with "file:///SOME_PATH"

</property>

<property>

<name>dfs.datanode.data.dir</name>

-- <value>file:/home/sameer/mydata/hdfs/datanode</value> -> Wrong

<value>/data/hdfs/datanode</value>

</property>

</configuration>

core-site.xml

<configuration>

<property>

<name>fs.default.name </name>

-- <value> hdfs://localhost:9000 </value> -> Wrong

<value> hdfs://NAMENODE_SERVER_FQDN:9000 </value> // fs.default.name is deprecated since that hadoop v2 as I know. // And I'm not recommend using localhost value for hostname. Just use FQDN even if it's a single node.

</property>

</configuration>

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<property>

<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property> <- Add

<name>yarn.web-proxy.address</name>

<value>YARN_SERVER_FQDN:8089</value> // Installed resourcemanager's fqdn in server.

</property>

<property> <- Add <name>yarn.resourcemanager.address</name>

<value>YARN_SERVER_FQDN:8032</value> // Installed resourcemanager's fqdn in server.

</property>

</configuration>

mapred-site.xml

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

<property>

<name>mapreduce.jobhistory.address</name> <- Add this config for jobhistoryserver

<value>NAMENODE_SERVER_FQDN:10020</value>

</property>

<property>

<name>mapreduce.jobhistory.webapp.address</name> <- Add this config for jobhistoryserver

<value>NAMENODE_SERVER_FQDN:19888</value>

</property>

<property>

<name>mapreduce.jobhistory.intermediate-done-dir</name> <- Add this config for jobhistoryserver

<value>/mr-history/tmp</value> // It is temporarily directory in HDFS for MR Jobs.

</property>

<property>

<name>mapreduce.jobhistory.done-dir</name> <- Add this config for jobhistoryserver

<value>/mr-history/done</value> // It is finished MR Jobs directory in HDFS.

</property>

</configuration>