Spark job is writing  event log to  hdfs://namenode:8021/spark-history , but our job creates many events within  10  to 12 minutes. As per the document spark.eventLog.buffer.kb  =100kb Does it mean when event writes to /spark-history/application_xxxxxx_xx/xx  ,  whenever buffer gets 100kb?  It means every time it is going to call fsync?   spark.eventLog.buffer.kb 100k Buffer size to use when writing to output streams, in KiB unless otherwise specified. ... View more

I  have seen  some of my jobs are using only  one reducer in the end.  Does it mean we have one output file?  Or it depends? ... View more

Our  user is running a job and which is a hive query and  number of  mapper is always  6 and not changing even the data size change.  It is  a  insert query.   How do  I change number of mappers ?  Which parameter determine number of mappers? ... View more

I can do query through   spark using spark.sql, but when I tried via beeline ranger is blocking the access because of the policy.  How does it work via spark query?  Doesn't it check HDFS level permission?  I ... View more