@Manikandan Jeyabal The problem perhaps could be at the project level then. Could you check your pom file and make sure you have all necessary spark depenedencies. I tried this in zeppelin ui and is working fine: Also make sure you clean/build and perhaps exit eclipse just in case there is something wrong with eclipse. Finally here is a link on how to setup depenedecies eclipse: https://community.hortonworks.com/articles/147787/how-to-setup-hortonworks-repository-for-spark-on-e.html HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@Quan Pham If cluster is not secured / kerberized and you have not configured any alternative authentication method like ldap then this is exactly what you will experience. You should consider securing your cluster by using kerberos authentication. You can enable kerberos using ambari. HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@Guozhen Li In yarn client mode the client machine - Windows machine needs to have network access to any of the cluster worker nodes (on any of the executors and AM could potentially run) and vise versa, the executors should be able to connect to the driver running on the windows client machine - I think you are right that this may be due firewall or network problem. HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@Sudharsan Ganeshkumar If the above has helped, please take a moment to login and click the "accept" link on the answer. ... View more

@Sundar Gampa If the above helped please remember to login and click the "accept" link on the answer. ... View more

@Sundar Gampa That path looks like the spark container working directory. Am I correct? This is taken from yarn configuration property yarn.nodemanager.local-dirs Out of the box spark provides ways to copy data to this directory by using --files --jar --archive arguments when running the spark-submit command. You can read more about those here: https://spark.apache.org/docs/latest/running-on-yarn.html Having that said if you like to add the directory resdata you simply need to zip the files you like to be part of the directory and add the zip file as spark-submit ... --files resdata.zip#resdata ... HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@Xiong Duan Based on the error stack I think it could be due missing configuration. It would be helpful if you share your workflow file and properties file. Have you tried using shell action instead ? ... View more

@Sudharsan Ganeshkumar AFAIK the csv format is not compatible between spark sql and hive serde and hence the error you are getting. A solution to this problem would be to: 1. create an external table pointing to the path where you will save the csv file 2. save the csv file instead of using saveAsTable function spark.sql("CREATE EXTERNAL TABLE Student_Spark2(col1 int,col2 string) STORED AS TEXTFILE LOCATION '/path/in/hdfs'") //later save rddstudent.write.format("csv").save("/path/in/hdfs/student_spark2") HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more