There's two main differences: 1) Livy will run the job in one of its SparkSessions, whereas by submitting to Yarn, you are creating a stand-alone Spark-Job. The former will be slightly quicker to come on-line, but you may not be able to use all the features of the ressource manager, with regards to using all available ressources. 2) As far as I can tell Livy cannot yet do what Knox does, with regards to bridging authentication realms. Knox allows you to authenticate against the Cluster's security context using username and password, and then internally use cryptographic authentication (i.e. Kerberos). With Livy, LDAP authentication is still an open Pull Request, but I expect you could use Kerberos/SPNEGO authentication. For this, your workstation/CI-server would have to be in the same Kerberos Realm as your cluster, which often isn't the case. Livy is nicer, if you have small jobs that you want to deploy with low latency and high modularity, Knox+Yarn is convenient, when you're outside the cluster, and have long running jobs. You can combine both: https://community.hortonworks.com/articles/70499/adding-livy-server-as-service-to-apache-knox.html ... View more

The default installation with Spark2 is providing a TGZ instead of a jar in a similarly named folder/location in hdfs - would that work as well as value for __spark__.jar, or is the jar still a requirement even with Spark2? ... View more

@Atul Kulkarni There's little point in using dfs.permissions without security. I suspect you can setup a user-name/password authentication filter for YARN-RM and simply add the username/password to the curl-request using -u username and enter the password interactively. If you use such a simple filter, you should also probably use SSL, otherwise the password can be easily intercepted. I still stand by my original comment, that without security enabled, dfs.permissions are basically ineffective, except in annoying cases like this. Much better to enable Kerberization and SPNEGO, and have security that works transparently and without passwords. ... View more

The options presented here are pretty discouraging. My use case would be to export client configuration via a Jenkins job, to allow continuous integration from dynamically scaling nodes that aren't Ambari-managed. Dynamically scaling edge-nodes should be a pretty standard use-case, but Ambari is anything but dynamic. On the other hand, it's convenient enough to manage the master/worker nodes. Ideally, I'd integrate Ambari with Git, and perform a simple checkout of the current client-configuration via Jenkins. Accessing the database directly is a big hack, and using Ambari's Rest-API is an insane overhead for something that conceptually is so simple. I also can't believe that Ambari is cooking its own version management, when there are dozen of highly evolved VCS which could be re-used. It's not as though the performance of the database backed Ambari VCS was a compelling reason to stick with it. I wish Knox was a better piece of software, since technically it's mostly designed to abstract client config away, but documentation and scalability, as well as sheer functionality are still in their infancy. And yes, I know, that a "simple" shell script could do this, but once you want this to work reliably, simplicity goes right out the window. VCS-Integration is probably worth filing a JIRA feature request for, and could help obsoleting/replacing/extending blueprints, which are too cumbersome for most of the applications they should support. ... View more

The first method does not work for me on HDP 2.6.2. What happens, is that the SOLR UI accessed through Knox (Or Knox itself does?) forwards to the actual address of the SOLR UI, and obviously that access doesn't work (due to firewall, differing Kerberos realms - whichever of the reasons you use Knox for). Are you aware of any supplemental configuration to achieve a working Knox-proxy with the HDP-Search-SOLR? Adding /solr to the url helps marginally, because it avoids the redirect, but the URLs and content are not working correctly. ... View more

Could you add more detail to your note? In particular, I'm not aware of any certificate extention "usr_cert". do you mean "-ext EKU=clientAuth"? ... View more

I think there's a bug though - if you revert to a previous config (as you usually would, when undoing a change), the HS2 interactive is not removed from the previously selected host. ... View more

Were you able to use the full R interpreter functionality? A quick test showed me that all of the more interesting features are broken, when using livy as an intermediary. Am I missing some step (well, I AM missing some R libraries potentially) or is the vis/rendering aspect broken? ... View more