I have two computers, the one I work on (CENTOS installed) and a second computer (also CENTOS (server), to act as the datanode), both not in a VM environment. I want to create a multi-node cluster with these computers. I have directly connected the computers together to test for possible network issues (ports etc.) and found that not to be the issue. I also used the guide by
https://tecadmin.net/set-up-hadoop-multi-node-cluster-on-centos-redhat/# and
https://dwbi.org/etl/bigdata/183-setup-hadoop-cluster. I have created a 'hadoop' user on both machines, with permissions, and established a password-less SSH access between both.
The hostname for the computers are:
1. NameNode (main computer): master
2. DataNode (the server): datanode1
The /etc/hosts file I have as(showing 'computerIP' in place of the actual IP's):
computerIP master
computerIP datanode1
My .xml file configurations on the NameNode are:
1. core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020/</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
2. hdfs-site.xml:
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/opt/volume/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/opt/volume/datanode</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:/opt/volume/namesecondary</value>
</property>
<property>
<name>dfs.replication</name>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:8020/</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
</configuration>
3. mapred-site.xml:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user/app</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Djava.security.egd=file:/dev/../dev/urandom</value>
</property>
</configuration>
4. yarn-site.xml:
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
<property>
<name>yarn.resourcemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.nodemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>file:/opt/volume/local</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>file:/opt/volume/yarn/log</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>hdfs://master:8020/var/log/hadoop-yarn/apps</value>
</property>
</configuration>
5. JAVA_HOME (Where java is located):
# The java implementation to use.
export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
6. Slaves file:
datanode1
7. Masters file:
master
My .bashrc file is as follows:
export JAVA_HOME=/usr/lib/java-1.8.0
export PATH=$PATH:$JAVA_HOME/bin
export HADOOP_HOME=/opt/hadoop/hadoop-2.8.3
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export CLASSPATH=$CLASSPATH:/usr/local/hadoop/lib/*:.
export HADOOP_OPTS="$HADOOP_OPTS -Djava.security.egd=file:/dev/../dev/urandom"
The permissions are as follows on both machines (from terminal):
[hadoop@master hadoop]$ ls -al /opt
total 0
drwxr-xr-x. 5 hadoop hadoop 44 Feb 15 16:05 .
dr-xr-xr-x. 17 root root 242 Feb 21 11:38 ..
drwxr-xr-x. 3 hadoop hadoop 53 Feb 15 16:00 hadoop
drwxr-xr-x. 2 hadoop hadoop 6 Sep 7 01:11 rh
drwxr-xr-x. 7 hadoop hadoop 84 Feb 20 11:27 volume
For the DataNode:
[hadoop@datanode1 ~]$ ls -al /opt
total 0
drwxrwxrwx. 4 hadoop hadoop 34 Feb 20 11:06 .
dr-xr-xr-x. 17 root root 242 Feb 19 16:13 ..
drwxr-xr-x. 3 hadoop hadoop 53 Feb 20 11:07 hadoop
drwxrwxrwx. 5 hadoop hadoop 59 Feb 21 09:53 volume
So when I go to format the namenode: hdfs namenode -format, I get that the NameNode is formatted on the 'master'.
And then I go start the system, $HADOOP_HOME/sbin/start-dfs.sh and get the following output:
[hadoop@master hadoop]$ $HADOOP_HOME/sbin/start-dfs.sh
Starting namenodes on [master]
master: starting namenode, logging to /opt/hadoop/hadoop-2.8.3/logs/hadoop-hadoop-namenode-master.out
datanode1: starting datanode, logging to /opt/hadoop/hadoop-2.8.3/logs/hadoop-hadoop-datanode-datanode1.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/hadoop-2.8.3/logs/hadoop-hadoop-secondarynamenode-master.out
Showing that the datanode is started, yet i go to the 50070 terminal to find that the datanode storage is not configured. I then stop the whole process, $HADOOP_HOME/sbin/stop-dfs.sh only to find that indeed the datanode didnt even start in the first place.
[hadoop@master hadoop]$ $HADOOP_HOME/sbin/stop-dfs.sh
Stopping namenodes on [master]
master: stopping namenode
datanode1: no datanode to stop
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
This is even when the computers are directly connected together. I have no idea why the datanode is not starting, and I hope someone could help. Need this for my masters thesis.
Thanks!
