@Nikil Katturi So you are using scopt to parse. The key is to pass the arguments after the application jar, which I think you are doing but perhaps there is a problem with the SPARKLE_JARS which is right before. You can review this for more information: https://stackoverflow.com/questions/40535304/how-to-pass-parameters-properties-to-spark-jobs-with-spark-submit I tested with spark pi example like this: /usr/hdp/current/spark2-client/bin/spark-submit --master local --class org.apache.spark.examples.SparkPi --driver-memory 512m --executor-memory 512m --num-executors 2 --executor-cores 1 examples/jars/spark-examples*.jar --job="Hello" And I get an error indicating the actual argument was passed correctly to the main function as expected. Which means spark-submit understood this was an argument to the applications and did not parse it. So in your case I suggest you check the order of the different parameters for spark-submit, use the above link example and make sure the application jar is last. Then in the application itself try to print the arguments in the stdout to troubleshoot in case the error is coming from your custom code. HTH ... View more

@Sriram Did it work? Please keep me posted 🙂 ... View more

@Nanda Kumar Yes, both are compatible and supported! You can install both in the same cluster and use both of them. If you are satisfied with the answer please remember to login and mark the answer as "Accepted" Thanks! ... View more

@Nikil Katturi There is no --job option for spark 2. I imagine this is set for the spark application you created, right? And is suppose to pass it as an argument to this application. I see you have set the --verbose to the spark submit command so I was expecting to see more output hopefully before the ERROR. Perhaps since you are redirecting the output to log file you can attach this to the post? ... View more

@Sriram Actually for non kerberos here are the instructions to setup user impersonation: https://community.hortonworks.com/articles/113228/how-to-enable-user-impersonation-for-jdbc-interpre-1.html HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@Sriram So based on the zeppelin configuration for jdbc interpreter I see you are using hive user (this explains why you can see all databases and have full access). Please review the documentation on how to configure zeppelin jdbc interpreter for impersonation as I mentioned above, you can also check this documentation: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_zeppelin-component-guide/content/config-hive-access.html HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more