we configured 4 disk! , it is not the first time that we configured , and this is the same on all our lab cluster , please look on this , you can see clearly 4 disks ! ... View more

the last one is : http://<active namenode host>:50070/dfshealth.html#tab-datanode-volume-failures ... View more

this is what we get from http://xxx.xxx.xxx.xxx:50070/dfshealth.html#tab-datanode ... View more

this is what we get from "http://<active namenode host>:50070/dfshealth.html#tab-overview" ... View more

Logging Levels The valid logging levels are log4j’s Levels (from most specific to least): OFF (most specific, no logging) FATAL (most specific, little data) ERROR WARN INFO DEBUG TRACE (least specific, a lot of data) ALL (least specific, all data) ... View more

for example I change all to ALL ( latest ) , to get the most details in spark logs and then restart the spark , but I not see that logs gives more data # Set everything to be logged to the console log4j.rootCategory=ALL, console log4j.appender.console=org.apache.log4j.ConsoleAppender log4j.appender.console.target=System.err log4j.appender.console.layout=org.apache.log4j.PatternLayout log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n # Settings to quiet third party logs that are too verbose log4j.logger.org.eclipse.jetty=ALL log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ALL log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=ALL log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=ALL log4j.logger.org.apache.spark.metrics.MetricsConfig=ALL log4j.logger.org.apache.spark.deploy.yarn.Client=ALL ... View more

this is example what we have on the datanode machine , you can see that used are around 47-61% , and each disk is 20G size ... View more

@Jay , can you refer to Karthik Palanisamy quastion ? ... View more

yes dfs.datanode.data.dir are - /data/sdb,/data/sdc,/data/sdd,/data/sde , and hdfs-site.xml is also with all right configuration ... View more