@rmr1989 , Seems logging is not working. Try to see if you can restart services under maintenance. And Check if you open track with Hortonworks support and get them to look into this. Regards, Fahim ... View more

@Fady Haikal , It will automatically done. Make sure you shut down VM and restarted it. Regards, Fahim ... View more

Hi @Tanmoy , When you start sandbox environment , what is IP you see when it is completed. Try to copy and paste that address to chrome and see if it works. Regards, Fahim ... View more

@rmr1989 , seems log process is hanging and not writing logs to files. Do you see any alerts in ambari ? Regards, Fahim ... View more

@Fady Haikal , If you have VMware image. When you open VMware player , select HDP image - down right side it will have two options - one option to launch it and another option for setting -- Go to setting and select memory and change it according to your requirement. Regards, Fahim ... View more

@Bas Madala , It is better to run ambari it under separate non-root account. Non root account should have capabilities mentioned which is required to run Ambari. Regards, Fahim ... View more

@sharad vishe , Below link wold also help with spark-submit. https://community.hortonworks.com/articles/28070/starting-spark-jobs-directly-via-yarn-rest-api.html ... View more

@sharad vishe , I donot think spark-submit is supported with apache knox. Only Spark web UI support is there. Please check below link for reference. http://knox.apache.org/books/knox-0-12-0/user-guide.html#Quick+Start Regards, Fahim ... View more

@ASIF Khan , Please check below presentation. It explains deployment mode for HBase for multitenancy and provide useful information. https://www.slideshare.net/Hadoop_Summit/multitenant-multicluster-and-multicontainer-apache-hbase-deployments Regards, Fahim ... View more

@Chiranjeevi Nimmala, What is group permission on HDFS location for Hive where this hive table is stored ? 'hdpmasters' group should have permission of HDFS file system where hive table is stored. See if below property is set. webhcat.proxyuser.root.groups * webhcat.proxyuser.root.hosts * Regards, Fahim ... View more

@sparkifyed , see if you can login to decommission node and see if ambari-agent is stopped on that server. If not stop all hadoop services on decommission node and see if you are able to view "hosts" in Ambari. It may be possible that Amabri-server is trying to connect to decommission server every time and hangs in between. Open a case with Hortonworks , if it is impacting your production system. Regards, Fahim ... View more

@Prasad T You can check queue property for Administration ACLs. It may help you to achieve our objective. You can check queue property yarn.scheduler.capacity.root.acl_administer_queue -- admin1,admin2 ... View more

@Narasimha K , 401 error seems to be related to authentication. Is ResourceManager Web UI requires authentication ? Can you try to check from log who is trying to access ResourceManger web UI ? Regards, Fahim ... View more

@HEMANTH KUMAR RATAKONDA Can you please let us know how you investigated that table is corrupted and need repair ? It will help. Regards, Fahim ... View more

@ANSARI FAHEEM AHMED , Can you try to see memory assigned to Spark History Server ? See if you can increase it. Second thought is please check no of applications is being shown in History server and see what is limit assigned to it. Hope this may help. Regards, Fahim ... View more

@Hoang Le , Please check this thread as well. It may be of help. https://issues.apache.org/jira/browse/SPARK-18584 ... View more

Hi @Hoang Le , One thing I can think of is , you can try to create two different user group for each of thrift server Ex. Group 1 and Group 2 and then create two queues SparkThrift1 , SparkThrift2 . Assign group1 to SparkThrift1 and group2 to SparkThrift2 queue and see if it is working as you expected. Regards, Fahim ... View more

@sadanjan mallireddy , ZKFC keep monitoring Name node server and keep pinging it after every X seconds interval and check health satus and if it does not find it in good state it works with zookeeper and try to convert another stand by name node to active name node and transition happens. In this case you should check whether due to any network issue or memory issue main name node is going in hang mode. Regards, Fahim ... View more

@Elnaz Ghada , You can find out list of application using command. "yarn application -list -appStates ALL" You can replace "ALL" with "Pending" state. Run this command for every 5 or x minutes and keep appending with time stamp to one file.Once you have data , write a script to find a unique application id and time stamp against it. And find last time stamp against same application id. ( Last time stamp - first time stamp ) per application id will give you pending time for application id. You can do different type of reporting against this data based on your requirement like "jobs running longer than > 60 minutes." Regards, Fahim ... View more

@Suhel , You can try to get report from Ambari Metrics. or another way is to schedule a job to collect memory data on every machine after every 60 seconds. You can use "memfree" or "sar" command to collect memory usage with timestamp and keep appending output to a file and generate report top on that in excel or any other reporting tool to find out memory usage per node. Regards, Fahim ... View more

@Mohammed Ashik , You can find out list of application using command. "yarn application -list -appStates ALL" Run this command for every 5 or x minutes and keep appending with time stamp to one file.Once you have data , write a script to find a unique application id and time stamp against it. And find last time stamp against same application id. ( Last time stamp - first time stamp ) per application id will give you running time for application. You can do different type of reporting against this data based on your requirement like "jobs running longer than > 60 minutes." Regards, Fahim ... View more

@Mukund Tripathi I am not sure about above error. But if you want to check size of your table you can execute "hdfs dfs -du -h /hbase/data/default/TEST12_2277594" . This should give you size of HBase table. Regards, Fahim ... View more

@Dino Bukvic In Ranger , Setting -> Permissions can help to setup group which can have access to Ranger components. Regards, Fahim ... View more

Hi @James Ballantine Below is link on creating Hadoop cluster on Virtual Box. It may help to answer your question. https://hadoopmagic.wordpress.com/2013/07/13/how-to-create-hadoop-cluster-on-virtual-machines-running-on-same-laptopdesktop/ Regards, Fahim ... View more

@Aishwarya Dixit I think below link describes how to avoid storing password in plain. Check AD , LDAP authentication section. https://zeppelin.apache.org/docs/0.6.2/security/shiroauthentication.html A link to one of password related issue : https://issues.apache.org/jira/browse/ZEPPELIN-530 Regards, Fahim ... View more

Hi Hemanth , What I assuming is , you have already created hive table and try to read it from Spark. What I am suggesting is from Spark also you can create a hive table using Spark SQL. Try to create a small hive table using Spark and try to read also. This will prove that your Spark functionality is working correctly with Hive. And issue is with specific table you posted in comment. Regards, Fahim ... View more

@Arpan Rajani You can try to create a group for Spark user and give it read permission on HDFS location of hive table using ACL. This should let Spark user read data from Spark-shell. Regards, Fahim ... View more

Hi , BI clients are connected with Spark Thrift server. Hive LLAP integration is also increasing. - Major difference would be around processing. Spark will do in memory processing and will need higher amount of memory then Hive LLAP. Thus more costly as infrastructure wise. - Hive is well connected with Ranger PlugIn for security and Ambari. Regards, Fahim ... View more

Hi, If you run "ls -ltr" command on each of this file or table names , it will return modified date , you can use this in reporting.Hope this helps. Regards, Fahim ... View more