I'm just curious to know what would be the main cause of corrupted blocks 🙂 As it would always have at least three copies, I thought this would be hard to happen (but happens) If HDFS can't create one copy or detects corruption, wouldn't it try to recover by copying a good one into another DataNode? Or once a file was properly created in HDFS, does it never check if file is corrupted or not until HDFS is restarted? ... View more

I'm looking at https://hadoop.apache.org/docs/current/hadoop-proj... and I don't find an easy way to copy one folder. Do I have to get the list of the content of the folder and download one by one? ... View more

I asked a question regarding OOM in NodeManager log and it turned out I need to increase the NM's heap size. So that I was wondering if there is any document which advises me how much memory I should assign? I'm hoping something similar to HDFS NameNode heap size ( http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-... ) Or as it would be hard to estimate, I should just try twice higher and see how it goes? ... View more

Hive's default value for "hive.exec.reducers.bytes.per.reducer" is 256MB (which used to be 1GB) Looks like this was introduced in AMBARI-8092 and AMBARI-8836. I was wondering if there is any specific reason Ambari sets 64MB? In general, is it good thing to have many reducers? ... View more

As long as reading http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Sys_Admin_Guides/content/ref-263ee41f-a0a9-4dea-ad4a-b3c257b8e188.1.html , looks like implying same realm. Even Cloudera mentions Distinct Realms... http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_admin_distcp_data_cluster_migrate.html#concept_hcs_srr_sr_unique_1 ... View more