@seth The OP's problem does not seem like a Hue problem. ... View more

Sorry I am not able to see your uploaded CM figure. From the stacktrace it looks like the NameNode 10.0.0.157 is done. Would you please check? ... View more

I am not sure about the CM's warning, but in principal, you should only add an add number of Zookeeper instances, e.g. 3, 5, or even 7. The RetryInvocationHandler warning should be unrelated to the zookeeper issue though. Instead, it's probably that the first namenode is the standby NN. If you manually fail over, I think you wouldn't see the warning again. You might also want to enable command line debug logs with the following command: export HADOOP_ROOT_LOGGER=DEBUG,console ... View more

I reproduced the error by intentionally corrupt the _index file.   If you meant "restore" by unarchiving the har file with hdfs dfs -cp command, I find it returns the same AIOOBE, so you won't be able to unarchive it.   Your best bet is to download the _index file, manually repair it, replace the _index file, and see how it goes.   Meanwhile, I filed an Apache jira HADOOP-14950 to handle the AIOOBE better, but it wouldn't help fix your corrupt _index file. ... View more

If you still have the source file, try to archive it again and see if it still produces the same error. If so, I'd be interested in knowing what's inside that _index file. ... View more

It looks like your har file is mal-formed. Inside the har file there is an index file called _index. The index file is expected to be in the format of <filename> <dir> pair in each line, and the later part of the line seems to get lost. ... View more

Hi @BellRizz thanks offering the workaround. What is the version of CDH that you have? From the description, it sounds like a race condition between reader and writer, and I suspect that is caused by HDFS-11056 (Concurrent append and read operations lead to checksum error) or HDFS-11160 (VolumeScanner reports write-in-progress replicas as corrupt incorrectly). While the summary of HDFS-11160 seems to suggest differently, I have seen customers hitting this issue with concurrent reads and writes. ... View more

Interesting story. The decomm process would not complete until all blocks have at least 1 good replica on other DNs. (good replica = replicas that are not stale and on a DataNode that is not being decommissioned or already decommissioned)   DirectoryScanner in a DataNode scans the entire directory, reconciling inconsistency between in-memory block map and on-disk replica, so it would eventually pick up the added replica, just a matter of time. ... View more

Have you tried restart the DN where you copied the blocks to? Also, try force a full block report: hdfs dfsadmin -triggerBlockReport <datanode_host:ipc_port> ... View more

The following is the HDFS NameNode configurations that can be updated to alleviate the issue:   <property> <name>ha.health-monitor.rpc-timeout.ms</name> <value>45000</value> <description> Timeout for the actual monitorHealth() calls. </description> </property>  I suggest bump ha.health-monitor.rpc-timeout.ms from 45000 (milliseconds) to 90000 and see if it helps. ... View more