Below are the steps to troubleshoot distcp:-  1. it is not problem with the hdfs or Kerberos or distcp but a MapReduce.  2. We tried to run a sample MR job to test, then it failed with the following exception Error: Java.io.IOException: initialization of all the collectors failed. Error in last collector was:java.io.IOException: Invalid “mapreduce.task.io.sort.mb”:3276. (The total amount of buffer memory to use while sorting files, in MB). It was expecting less than 2048. Changing this property able to run the distcp smooth.  I want to take a moment and say thanks to Shelton for responding it on time.  ... View more

@Manoj690  It's always a good idea to share the HDP and Zk version plus the zk logs in /var/log/* having said that can you share your zoo.cfg ? If you really need enable all four letter word commands by default, you can use the asterisk option so you don't have to include every command one by one in the list.See below 4lw.commands.whitelist=* As you have not shared your logs that's a starting point, then restart your zookeeper and let me know! ... View more

Hi @Shelton  Finally I've solved the problem and hope my experience could help someone else who is banging his/her head on this scenario.   At time of writing I was accessing the GCP cluster via public IP addresses. I had a /etc/hosts file resolving hostnames to their public IP Then customer's NOC granted me access from internal network over VPN and next attempt went fine.   Still can't understand why, maybe some networking related config issue, but being on the internal side of the network solved my problem   Simple as that! Thanks for your precious support, and patience Stefano ... View more

@mike_bronson7    When your cluster is in HA it uses a namespace that acts as a load balancer to facilitate the switch from active to standby and vice versa. The dfs-site-xml holds these values  filter using  dfs.nameservices  the nameservice-id  should be your namespace or in HA look for dfs.ha.namenodes.[nameservice ID]   dfs.ha.namenodes.[nameservice ID] e.g dfs.ha.namenodes.mycluster   And that's the value to set  eg    hdfs://mycluster_namespace/user/ams/hbase   The refresh the stale configs , now HBase should sending the metrics  to that directory  HTH   ... View more