Datanodes start and then stop ... only one datanode process working at a time

[#1]

hi

The Env details:

I installed a hadoop 2.7.2 (not HW but pure Hadoop) multinode cluster on AWS (1 Namenode/1 2nd NN/ 3 datanodes - ubuntu 14.04).

The cluster was based on the following tutorial(http://mfaizmzaki.com/2015/12/17/how-to-install-hadoop-2-7-1-multi-node-cluster-on-amazon-aws-ec2-instance-improved-part-1/) --> this means the first install (master) is copied and tuned across

The Issue:

The 3 data nodes individually work correctly if I configure the cluster with 1 Datanode (I specifically excluded the 2 others).

As soon as I add another data node the data node booting first log a FATAL error (see extract of the log file hereafter and snapshot of the VERSION file) and stop. The data node booting second work then fine...

1. Any idea-recommendation ?
2. Am I doing something wrong cloning the AMI of the master on other machine?

Thanks Folks!

Log File

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Unsuccessfully sent block report 0x1858458671b, containing 1 storage report(s), of which we sent 0. The reports had 0 total blocks and used 0 RPC(s). This took 5 msec to generate and 35 msecs for RPC and NN processing. Got back no commands.

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1251070591-172.Y.Y.Y-1454167071207 (Datanode Uuid 54bc8b80-b84f-4893-8b96-36568acc5d4b) service to master/172.Y.Y.Y:9000 is shutting down org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.UnregisteredNodeException): Data node DatanodeRegistration(172.X.X.X:50010, datanodeUuid=54bc8b80-b84f-4893-8b96-36568acc5d4b, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-56;cid=CID-8e09ff25-80fb-4834-878b-f23b3deb62d0;nsid=278157295;c=0) is attempting to report storage ID 54bc8b80-b84f-4893-8b96-36568acc5d4b. Node 172.Z.Z.Z:50010 is expected to serve this storage.

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-1251070591-172.31.34.94-1454167071207 (Datanode Uuid 54bc8b80-b84f-4893-8b96-36568acc5d4b) service to master/172.Y.Y.Y:9000

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool BP-1251070591-172.Y.Y.Y-1454167071207 (Datanode Uuid 54bc8b80-b84f-4893-8b96-36568acc5d4b) INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Removing block pool BP-1251070591-172.31.34.94-1454167071207

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at HNDATA2/172.X.X.x ************************************************************/

[#2]

I did it. Hadoop 2.7.2 is installed, configure and it runs on my cluster! thanks guys for your great recommendation

[#3]

@luc tiber Wonderful!!! Very nice! now time to play with ambari 🙂

[#4]

Well that was my "next" question... "what next"? I mean I installed Hadoop but what should come next. Ambari, Zookeeper, HBASE, HCat...(assuming I just want to try - so no need for me to have a ferrari to learn to drive)

Since I guess there are as much perspective as there are people, feel free to fire up guys!

[#5]

@luc tiber Yes

Make http://docs.hortonworks.com/index.html as your main landing point

Ambari 2.2 or HDP 2.3

Once you click HDP 2.3 then you will see the following

Click anyone of the link based on your requirement

Tutorials http://hortonworks.com/products/hortonworks-sandbox/#tutorial_gallery