@epowell   The issue might be related to the below jira which is opened a long back still in open status   https://issues.apache.org/jira/browse/HDFS-3447   as an alternate way to connect to hdfs, go to hdfs-site.xml and get dfs.nameservices and try to connect to hdfs using namespace as follows, it may help you   hdfs://<ClusterName>-ns/<hdfs_path>    Note: I didn't get a chance to explore this... also not sure how it will respond in old cdh version   ... View more

@gaurav796   The difference is     insertInto: To overwrite any existing data   Mode comes with additional options, like    mode("append"): Append contents of this DataFrame to existing data mode("overwrite:): Overwrite existing data.   Note: I didn't get a chance to explore this before reply ... View more

@ganeshkumarj   a. mapred.map.tasks - The default number of map tasks per job is 2. Ignored when mapred.job.tracker is "local". You can modify using set mapred.map.tasks = <value> b. mapred.reduce.tasks - The default number of reduce tasks per job is 1. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Ignored when mapred.job.tracker is "local". you can modify using set mapred.reduce.tasks = <value>   https://hadoop.apache.org/docs/r1.0.4/mapred-default.html ... View more

@cdhhadoop   as mentioned, you will get warning if b > a  ... View more

@cdhhadoop   Get the value of a. CM -> HDFS -> Configuration -> DataNode Block Count Thresholds b. CM -> HDFS -> WebUI -> Namenode Web UI -> Click on datanode menu -> Get the block count of your node   if b > a then you will get block count warning   also cloudera advice says "presence of many small files" also create this warning   action:  1. if it is not disturbing anything then you can ignore this warning but just keep an on eye on block pool usage percentage from 'b' 2. you can increase block count thresholds in 'a' 3. you can cleanup unwanted data, but if your trash folder maintains old data (for ex: 24 hrs) then you will see the result after 24 hours 4. add additional data nodes and apply rebalance etc     ... View more

@desind   To add on to your point, the cluster setup is applicable to all the mapreduce job, so it may impact other non-mapreduce jobs.    In fact I am not against setup higher value in cluster itself, but you can do that based on how many jobs requires higher values and performance, etc     ... View more