Created 06-23-2016 07:37 AM
I installed a three node hadoop cluster. The master and slave node starts separately but the datanode isn't shown in namenode webUI. The log file for datanode shows the following error :
2016-06-18 21:23:53,980 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode/192.168.1.100:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2016-06-18 21:23:55,029 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode/192.168.1.100:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2016-06-18 21:23:56,030 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode/192.168.1.100:9000. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2016-06-18 21:23:57,031 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode/192.168.1.100:9000. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2016-06-18 21:23:58,032 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: namenode/192.168.1.100:9000. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
namenode machine's infomation:
cat /etc/hosts
#127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 #::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.1.100 namenode 192.168.1.101 datanode1 192.168.1.102 datanode2
cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0 IPV6INIT=yes BOOTPROTO=dhcp UUID=61fe61d3-fcda-4fed-ba81-bfa767e0270a ONBOOT=yes TYPE=Ethernet DEFROUTE=yes PEERDNS=yes PEERROUTES=yes IPV4_FAILURE_FATAL=no IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no NAME="System eth0" BOOTPROTO="static" ONBOOT="yes" IPADDR=192.168.1.100 GATEWAY=192.168.1.1 NETMASK=255.255.255.0 DNS1=192.168.1.1
cat /etc/hostname
namenode
cat core-site.xml
<configuration> <property> <name>hadoop.tmp.dir</name> <value>/home/hadoop/tmp</value> <description>Abase for other temporary directories.</description> </property> <property> <name>fs.defaultFS</name> <value>hdfs://namenode:9000</value> </property> <property> <name>io.file.buffer.size</name> <value>4096</value> </property>
cat hdfs-site.xml
<configuration> <property> <name>dfs.nameservices</name> <value>hadoop-cluster1</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>namenode:50090</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:///home/hadoop/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:///home/hadoop/dfs/data</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property>
cat mapred-site.xml
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobtracker.http.address</name> <value>namenode:50030</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>namenode:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>namenode:19888</value> </property>
cat yarn-site.xml
<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>namenode:8032</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>namenode:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>namenode:8031</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>namenode:8033</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>namenode:8088</value> </property>
cat slaves
datanode1 datanode2
Created 06-23-2016 08:29 AM
First please check if namenode is up and running. If yes, then from datanode host run below command and see if it able to connect to the NN port.
From DN:
telnet 192.168.1.100 9000
If telnet is not responding then you might have firewall rules blocking the connection. Try disabling firewall and see if that make any difference.
Created 06-23-2016 08:29 AM
First please check if namenode is up and running. If yes, then from datanode host run below command and see if it able to connect to the NN port.
From DN:
telnet 192.168.1.100 9000
If telnet is not responding then you might have firewall rules blocking the connection. Try disabling firewall and see if that make any difference.
Created 06-23-2016 01:18 PM
you are right,the firewall is running