@Eon kitex Hortonworks does a lot of integration testing before it releases any new version of HDP. It means that Hortons makes sure that all components being part of HDP have been tested to work together so that they are supportable and enterprise ready. You cannot upgrade individual components. If you want to just test spark 2.0, you can download it separately on a different machine and evaluate it. But you cannot upgrade just spark HDP 2.3.2. When Hortonworks starts supporting Spark 2.0, it will be included in newer versions of HDP. ... View more

@Vipin Rathor I figured this one out. Here is the thing. spark doesn't allow to submit keytab and principal with proxy-user. The reason is you need keytab and principal when you are running long running jobs. In this case, for long running jobs, the keytab will be copied to application master staging area and this keytab and principal will be used to renew delegation tokens required for HDFS. This enables the application to continue working with any security issue. Remember this feature is explicitly for long running applications. See details here under YARN mode, second paragraph. Imagine if all application users logging into my applications can proxy to my keytab user since the yarn application will be submitted as the proxy-user but it will copy the keytab. This means those users can read the contents of that keytab. This is a huge security flaw. And that's why what I was trying above is not allowed. I have to do what Hive does to run "spark-submit". Basically kinit before submitting my application and then provide a proxy-user. So here is how I solved it. kinit <principal>@<REALM> -k -t <mykeytab file> spark-submit <all my options include --proxy-user <my proxy user>> This way my proxy-user cannot read keytab contents and is only used as a proxy-user. My application is not long running (like more than 7 days which is the usual lifetime of a TGT), so I am fine. ... View more

@Saurabh Kumar You might be following a different set of instructions. The one i am looking at uses solr-yarn.jar and here is the format hadoop jar solr-yarn.jar org.apache.solr.cloud.yarn.SolrClient -nodes=2 -zkHost=localhost:2181 -solr=hdfs://localhost:9000/solr/solr.tgz -jar=hdfs://localhost:9000/solr/solr-yarn.jar -memory 512 -hdfs_home=hdfs://localhost:9000/solr/index_data This is where I think you should be able to add "-queue=<queue name>" did you try following the instructions on this link? the class name which supports "queue" is not in the command you are running. Is it possible for you to try the following? https://lucidworks.com/blog/2014/12/01/solr-yarn/ ... View more

@kishore sanchina nofile is the number of open files a linux user can open in a session. Check this link for more details. There is a soft and hard limit for this. nproc is the number of processes a linux user may open in a session. This link may also help you understand how this works. ... View more