Find sample spark-default.conf as below. Replace the correct value for below markers: <hadoop-client-native> : Dir path to hadoop native dir , In hdp clusters , it is generally /usr/hdp/current/hadoop-client/lib/native <spark-history-dir> : HDFS dir on cluster where spark history server event log should be stored. Make sure this dir is owned by spark:hadoop with 777 permission spark.driver.extraLibraryPath <hadoop-client-native>:<hadoop-client-native>/Linux-amd64-64 spark.eventLog.dir hdfs:///<spark-history-dir> spark.eventLog.enabled true spark.executor.extraLibraryPath <hadoop-client-native>:<hadoop-client-native>/Linux-amd64-64 spark.history.fs.logDirectory hdfs:///<spark-history-dir> spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider spark.history.ui.port 18080 spark.yarn.containerLauncherMaxThreads 20 spark.yarn.driver.memoryOverhead 384 spark.yarn.executor.memoryOverhead 384 spark.yarn.historyServer.address xxx:18080 spark.yarn.preserve.staging.files false spark.yarn.queue default spark.yarn.scheduler.heartbeat.interval-ms 5000 spark.yarn.submit.file.replication 3 Find sample spark-env.sh as below. Please update the paths as per your environment. export SPARK_CONF_DIR=/etc/spark/conf export SPARK_LOG_DIR=/var/log/spark export SPARK_PID_DIR=/var/run/spark export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/current/hadoop-client} export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/usr/hdp/current/hadoop-client/conf} # The java implementation to use. export JAVA_HOME=<jdk path> Hope this helps. ... View more

@Thangarajan Pannerselvam, 1) Python: You can actually use panda to get a data frame from a query result. Read below blog for details on how to run a query on a database and get data frame from a result. https://www.dataquest.io/blog/python-pandas-databases/ 2) Scala: You can also use jdbc driver to run a query and save the result of query in data frame. ( use spark.read.jdbc api). Details can be found in below link. https://sparkour.urizone.net/recipes/using-jdbc/ ... View more

@Devender Yadav, It seems that TimeStampType do not support nanoseconds yet. https://issues.apache.org/jira/browse/SPARK-17914 is opened to track this issue. You might want to use some other data type like String to store this data if you require nanoseconds data. ... View more

@Cheng Xu, can you please try adding cobertura jar to spark.driver.extraClassPath? ... View more

@Abhijeet Rajput, Found an article which compares performance of RDD/ Dataframe and SQL . It will help you make informed decision. https://community.hortonworks.com/articles/42027/rdd-vs-dataframe-vs-sparksql.html In summary, You mainly need to analyze your use case ( like what type of queries will you be running , how big is data set etc). Depending on your use case, you can choose to go with either SQL or Dataframe API. For example: If your use case involves lot of groupby, orderby like queries, you should go with sparkSQL instead data frame api. ( because sparkSQL executes faster than data frame api for such use case) ... View more