Hi , I am using HDP 2.6 with Spark 2.1 ( also Spark 1.6) with Yarn as resource manager . I am trying out TeraSort benchmarking jobs on a experimental cluster. I want to run 'hdfs dfs -du' or 'hdfs fs -du' command every time before starting a Spark job to analyse available disk space in data nodes. From the following question I understand that running these commands is expensive on cluster https://community.hortonworks.com/questions/92214/can-hdfs-dfsadmin-and-hdfs-dsfs-du-be-taxing-on-my.html So I wanted to know whether hadoop automatically runs dfs -du command in the background, whenever a new Spark job is started. Or do I need to run manually ? Thanks, Steev ... View more

Hi, We have a 3 node HDP cluster with Ambari 2.4 . We run Tera Sort jobs for bench marking. I would like to know how to hot swap a Data node hard disk (failed disk) without stopping the cluster services and without stopping on going Tera sort job ? Thanks, Steevan ... View more

Hi I am running a Terasort application on 1G data using HDP 2.5 with 3 node cluster and spark 1.6 . Sorting completes fine . But the output data is not compressed. Output data size written HDFS file system is also 1G in size. I tried following spark configuration options while submitting spark job. --conf spark.hadoop.mapred.output.compress=true --conf spark.hadoop.mapred.output.compression.codec=true --conf spark.hadoop.mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec --conf spark.hadoop.mapred.output.compression.type=BLOCK This did not help. Also tried following and it did not help. --conf spark.hadoop.mapreduce.output.fileoutputformat.compress=true --conf spark.hadoop.mapreduce.map.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec --conf spark.hadoop.mapred.output.fileoutputformat.compress.type=BLOCK How to get output data written HDFS in compressed format ? Thanks Steevan ... View more