Let us assume there is a workflow which has three shell actions .First Shell action is a python utility used for creation of Spark context object. I would like to pass the spark context created in first shell action to second and third shell actions(python scripts) Is this even possible? Any other suggestions for above usecase ... View more

I have enabled namespacing property in hbase-site.xml , but I am unable to create schema using Java Phoenix connectivity. Error message shows that namespacing property should be enabled ( which is already the case) . Any way to specifically mention/override the hbase properties in the java program ? ... View more

I have a program for bulk loading of data to HBase using creation of HFile , for which I have used configureIncrementalLoad () of HFileOutputFormat2 . But , this HFile is created using 'hbase' user, I want this user to be actual user , who is running the java program. Any suggestions on it ? I have researched about hadoop.proxyusers.hbase related properties, but little confused regarding the same. ... View more

Initially we had cluster name as ClusterHW_1 in ambari and corresponding ranger default policies as ClusterHW_1_hadoop,Cluster HW_1_hbase,ClusterHW_1_hive . After couple of days , we renamed the cluster ClusterHW_2 and restarted most of the services due to change in Hadoop.proxy.hosts value to *. We observed that earlier ranger policies were not working and ambari has created new ranger policies as ClusterHW_2_hadoop,Cluster HW_2_hbase,ClusterHW_2_hive . Can someone explain this behaviour and how is it working in detailed way and how to prevent this , if possible? ... View more

1) Can we import unix groups (group 1, group 2 etc) in to Ranger as groups direclty 2) Can we directly have yarn queues as per unix group ? Like group1 queue , group 2 queue 3) If user 1 is part of two yarn queues(queue1 and queue2) at the same time, which queue will user1 be allocated to? ... View more

I can see lot of pyspark jobs with Progress (10%) with fixed allocated memory. How do I enable dynamic memory deallocation ? Developers are running pyspark jobs inside Zeppelin interpreter and spark shell . Any suggestion on above issue ? ... View more

Hi, I have created user úser1' in zeppelin. I am running some shell scripts to create directories in my hdfs directory, but its throwing access denied exception Error , user=zeppelin, Access denied May I know the way to execute shell script as user1 from zeppelin interpreter rather than 'zeppelin' ... View more

I am able to create File/Tez/Hive view using Ambari api. How do i create smartsense view using rest api? ... View more

I have 5 worker nodes in Hortonworks cluster with 15 cores per worker node for Yarn and 1 TB Yarn memory in total with default spark config values as of now i.e 1 G executor memory etc. 1. Which parameters should I tune for maximum utilization of spark job for each user ? 2. What should be the recommended values for max number of executors, executor memory and executor per cores for above Hardware configuration Note : Dynamic resource allocation is already enabled. Any tips on tuning spark jobs are always welcomed. @Guilherme Braccialli @Andrew Watson ... View more