There's two main differences: 1) Livy will run the job in one of its SparkSessions, whereas by submitting to Yarn, you are creating a stand-alone Spark-Job. The former will be slightly quicker to come on-line, but you may not be able to use all the features of the ressource manager, with regards to using all available ressources. 2) As far as I can tell Livy cannot yet do what Knox does, with regards to bridging authentication realms. Knox allows you to authenticate against the Cluster's security context using username and password, and then internally use cryptographic authentication (i.e. Kerberos). With Livy, LDAP authentication is still an open Pull Request, but I expect you could use Kerberos/SPNEGO authentication. For this, your workstation/CI-server would have to be in the same Kerberos Realm as your cluster, which often isn't the case. Livy is nicer, if you have small jobs that you want to deploy with low latency and high modularity, Knox+Yarn is convenient, when you're outside the cluster, and have long running jobs. You can combine both: https://community.hortonworks.com/articles/70499/adding-livy-server-as-service-to-apache-knox.html ... View more

The default installation with Spark2 is providing a TGZ instead of a jar in a similarly named folder/location in hdfs - would that work as well as value for __spark__.jar, or is the jar still a requirement even with Spark2? ... View more

@Atul Kulkarni There's little point in using dfs.permissions without security. I suspect you can setup a user-name/password authentication filter for YARN-RM and simply add the username/password to the curl-request using -u username and enter the password interactively. If you use such a simple filter, you should also probably use SSL, otherwise the password can be easily intercepted. I still stand by my original comment, that without security enabled, dfs.permissions are basically ineffective, except in annoying cases like this. Much better to enable Kerberization and SPNEGO, and have security that works transparently and without passwords. ... View more

The first method does not work for me on HDP 2.6.2. What happens, is that the SOLR UI accessed through Knox (Or Knox itself does?) forwards to the actual address of the SOLR UI, and obviously that access doesn't work (due to firewall, differing Kerberos realms - whichever of the reasons you use Knox for). Are you aware of any supplemental configuration to achieve a working Knox-proxy with the HDP-Search-SOLR? Adding /solr to the url helps marginally, because it avoids the redirect, but the URLs and content are not working correctly. ... View more

Could you add more detail to your note? In particular, I'm not aware of any certificate extention "usr_cert". do you mean "-ext EKU=clientAuth"? ... View more