question Re: hadoop 3 nodes configuration issues: Datanode dont appear in slaves and Nodemanager is appearing in salves but also in master in Archives of Support Questions (Read Only)

hadoop 3 nodes configuration issues: Datanode dont appear in slaves and Nodemanager is appearing in salves but also in master

joncodin — Fri, 29 Apr 2016 06:10:42 GMT

Im installing hadoop 2.7.1 on 3 nodes and Im having some difficulties in the configuration process.

I want to have:

node1 (master) - as the namenode and resource manager

node2 (slave) - as the datanode and nodemanager

node3(slave) - as the datanode and nodemanager

Im doing the configurations like below to achieve the goal:

etc/hosts file:

127.0.0.1 localhost
192.168.1.60 NameNode
192.168.1.61 Slave1
192.168.1.62 Slave2

core-site.xml:

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://NameNode:9000</value>
</property>
</configuration>

hdfs-site.xml:

<configuration>
<name>dfs.replication</name>
<value>3</value>

In the slaves files i enter the hostnames of the slaves machines:

Slave1

Slave2

I created a masters file and entered the hostname of the master machine:

NameNode

Note: I didnt configure the yarn-site.xml and mapred-site.xml files. Its needed?

Problem:

With my configuration above Im having two issues when start all deamons and check with jpscommand:

1) the node manager appears in the master and not only in the slaves machines

2) the datanode dont appear in the slaves machines

jps in the master machine:

ResourceManager
NameNode
NodeManager
SecondaryNameNode

jps command in slave machines:

NodeManager

Re: hadoop 3 nodes configuration issues: Datanode dont appear in slaves and Nodemanager is appearing in salves but also in master

ravi1 — Fri, 29 Apr 2016 06:28:43 GMT

Check if you find nodemanager and datanode logs on slave nodes where they didn't start. They should tell what went wrong. Most likely they failed with errors. You may also need yarn-site.xml to configure yarn.nodemanager.log-dirs and yarn.nodemanager.local-dirs params.

Try following these instructions for start and stop services since this is the supported way. Also, always try to configure FQDN instead of just hostnames.

Re: hadoop 3 nodes configuration issues: Datanode dont appear in slaves and Nodemanager is appearing in salves but also in master

hduraiswamy — Fri, 29 Apr 2016 10:31:46 GMT

I am assuming you are trying to install manually from apache. Is there a reason you are not using HDP w/ Ambari?

Re: hadoop 3 nodes configuration issues: Datanode dont appear in slaves and Nodemanager is appearing in salves but also in master

joncodin — Fri, 29 Apr 2016 18:11:39 GMT

Thanks, Where I cant find that logs? Im trying to find but without success. And when I execute the echo $HOSTANAME command the fully hostname that I get is what I put in the question.

Re: hadoop 3 nodes configuration issues: Datanode dont appear in slaves and Nodemanager is appearing in salves but also in master

joncodin — Fri, 29 Apr 2016 18:12:58 GMT

Yes Im trying to install manually, because I think its better to learn the process how to get hadoop running.

Re: hadoop 3 nodes configuration issues: Datanode dont appear in slaves and Nodemanager is appearing in salves but also in master

ravi1 — Fri, 29 Apr 2016 20:01:52 GMT

/var/log/hadoop/hdfs/hadoop-hdfs-datanode-<hostname>.log has datanode log and /var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-<hostname>.log has nodemanager on each node. You can also look at .out files with same name in the same directories.