In this case the source host (Appmaster) was unable to received any response from destination as this is a faulty routing issue and packet was just lost. I checked the source code for the package "org.apache.hadoop.ipc.Client" and found that it uses PING utility to check response from destination host and keep trying until it received any responce. This is clearly mentioned in the this link http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.action/0.2.0/org/apache/hadoop/ipc/Client.java "This class sends a ping to the remote side when timeout on reading. If no failure is detected, it retries until at least a byte is read." So, due to routing issue and it keep trying until the job was killed. thanks to grepcode.com for facilitating easy way to read source code. ... View more

I found the solution. There was incorrect hostname entry on /etc/hosts file on Resource manager node and as result nodemanager registration failed as resource manager does not accept request from a unauthorized host. Thanks Khireswar ... View more

@Lester Martin @William Gonzalez ... View more

I got it working on Ambari 2.2.1 1.Create mount points: #mkdir /hadoop/hdfs/data1 /hadoop/hdfs/data2 /hadoop/hdfs/data3 #chown hdfs:hadoop /hadoop/hdfs/data1 /hadoop/hdfs/data2 /hadoop/hdfs/data3 (**We are using the configuration for test purpose only, so no disks are mounted.) 2.Login to Ambari > HDFS>setting 3.Add datanode directories as shown below: Datanode>datanode directories: [DISK]/hadoop/hdfs/data,[SSD]/hadoop/hdfs/data1,[RAMDISK]/hadoop/hdfs/data2,[ARCHIVE]/hadoop/hdfs/data3 Restart hdfs hdfs service. Restart all other afftected services. Create a directory /cold # su hdfs [hdfs@hdp-qa2-n1 ~]$ hadoop fs -mkdir /cold Set COLD storage policy on /cold [hdfs@hdp-qa2-n1 ~]$ hdfs storagepolicies -setStoragePolicy -path /cold -policy COLD Set storage policy COLD on /cold 5. Run get storage policy: [hdfs@hdp-qa2-n1 ~]$ hdfs storagepolicies -getStoragePolicy -path /cold The storage policy of /cold: BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]} ... View more

Following steps working for me: 1.Create mount points: #mkdir /hadoop/hdfs/data1 /hadoop/hdfs/data2 /hadoop/hdfs/data3 #chown hdfs:hadoop /hadoop/hdfs/data1 /hadoop/hdfs/data2 /hadoop/hdfs/data3 (**We are using the configuration for test purpose only, so no disks are mounted.) 2.Login to Ambari > HDFS>setting 3.Add datanode directories as shown below: Datanode>datanode directories: [DISK]/hadoop/hdfs/data,[SSD]/hadoop/hdfs/data1,[RAMDISK]/hadoop/hdfs/data2,[ARCHIVE]/hadoop/hdfs/data3 Restart hdfs hdfs service. Restart all other afftected services. Create a directory /cold # su hdfs [hdfs@hdp-qa2-n1 ~]$ hadoop fs -mkdir /cold Set COLD storage policy on /cold [hdfs@hdp-qa2-n1 ~]$ hdfs storagepolicies -setStoragePolicy -path /cold -policy COLD Set storage policy COLD on /cold 5. Run get storage policy: [hdfs@hdp-qa2-n1 ~]$ hdfs storagepolicies -getStoragePolicy -path /cold The storage policy of /cold: BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]} ... View more