Member since
05-13-2017
42
Posts
0
Kudos Received
0
Solutions
11-23-2017
12:24 PM
Hi Community members, Does hive provide any metadata tables in metastore for audits and table stat collection which can be directly queried for audit purpose and query optimization ? Regards, Fahim
... View more
Labels:
- Labels:
-
Apache Hive
09-26-2017
10:53 AM
@rmr1989 , Seems logging is not working. Try to see if you can restart services under maintenance. And Check if you open track with Hortonworks support and get them to look into this. Regards, Fahim
... View more
09-26-2017
10:50 AM
@Fady Haikal , It will automatically done. Make sure you shut down VM and restarted it. Regards, Fahim
... View more
09-26-2017
07:59 AM
Hi @Tanmoy , When you start sandbox environment , what is IP you see when it is completed. Try to copy and paste that address to chrome and see if it works. Regards, Fahim
... View more
09-26-2017
07:56 AM
@rmr1989 , seems log process is hanging and not writing logs to files. Do you see any alerts in ambari ? Regards, Fahim
... View more
09-26-2017
07:54 AM
@Fady Haikal , If you have VMware image. When you open VMware player , select HDP image - down right side it will have two options - one option to launch it and another option for setting -- Go to setting and select memory and change it according to your requirement. Regards, Fahim
... View more
09-07-2017
05:56 PM
@Bas Madala , It is better to run ambari it under separate non-root account. Non root account should have capabilities mentioned which is required to run Ambari. Regards, Fahim
... View more
08-14-2017
01:00 PM
@sharad vishe , Below link wold also help with spark-submit. https://community.hortonworks.com/articles/28070/starting-spark-jobs-directly-via-yarn-rest-api.html
... View more
08-14-2017
12:56 PM
@sharad vishe , I donot think spark-submit is supported with apache knox. Only Spark web UI support is there. Please check below link for reference. http://knox.apache.org/books/knox-0-12-0/user-guide.html#Quick+Start Regards, Fahim
... View more
08-13-2017
08:46 AM
@ASIF Khan , Please check below presentation. It explains deployment mode for HBase for multitenancy and provide useful information. https://www.slideshare.net/Hadoop_Summit/multitenant-multicluster-and-multicontainer-apache-hbase-deployments Regards, Fahim
... View more
08-11-2017
11:09 AM
@Chiranjeevi Nimmala, What is group permission on HDFS location for Hive where this hive table is stored ? 'hdpmasters' group should have permission of HDFS file system where hive table is stored. See if below property is set. webhcat.proxyuser.root.groups * webhcat.proxyuser.root.hosts * Regards, Fahim
... View more
08-11-2017
11:00 AM
@sparkifyed , see if you can login to decommission node and see if ambari-agent is stopped on that server. If not stop all hadoop services on decommission node and see if you are able to view "hosts" in Ambari. It may be possible that Amabri-server is trying to connect to decommission server every time and hangs in between. Open a case with Hortonworks , if it is impacting your production system. Regards, Fahim
... View more
08-11-2017
07:41 AM
@Prasad T You can check queue property for Administration ACLs. It may help you to achieve our objective. You can check queue property yarn.scheduler.capacity.root.acl_administer_queue -- admin1,admin2
... View more
08-11-2017
07:14 AM
@Narasimha K , 401 error seems to be related to authentication. Is ResourceManager Web UI requires authentication ? Can you try to check from log who is trying to access ResourceManger web UI ? Regards, Fahim
... View more
08-10-2017
06:16 PM
@HEMANTH KUMAR RATAKONDA Can you please let us know how you investigated that table is corrupted and need repair ? It will help. Regards, Fahim
... View more
08-10-2017
01:33 PM
@ANSARI FAHEEM AHMED , Can you try to see memory assigned to Spark History Server ? See if you can increase it. Second thought is please check no of applications is being shown in History server and see what is limit assigned to it. Hope this may help. Regards, Fahim
... View more
08-10-2017
01:24 PM
@Hoang Le , Please check this thread as well. It may be of help. https://issues.apache.org/jira/browse/SPARK-18584
... View more
08-10-2017
01:21 PM
Hi @Hoang Le , One thing I can think of is , you can try to create two different user group for each of thrift server Ex. Group 1 and Group 2 and then create two queues SparkThrift1 , SparkThrift2 . Assign group1 to SparkThrift1 and group2 to SparkThrift2 queue and see if it is working as you expected. Regards, Fahim
... View more
08-10-2017
11:50 AM
@sadanjan mallireddy , ZKFC keep monitoring Name node server and keep pinging it after every X seconds interval and check health satus and if it does not find it in good state it works with zookeeper and try to convert another stand by name node to active name node and transition happens. In this case you should check whether due to any network issue or memory issue main name node is going in hang mode. Regards, Fahim
... View more
08-10-2017
11:34 AM
@Elnaz Ghada , You can find out list of application using command. "yarn application -list -appStates ALL" You can replace "ALL" with "Pending" state. Run this command for every 5 or x minutes and keep appending with time stamp to one file.Once you have data , write a script to find a unique application id and time stamp against it. And find last time stamp against same application id. ( Last time stamp - first time stamp ) per application id will give you pending time for application id. You can do different type of reporting against this data based on your requirement like "jobs running longer than > 60 minutes." Regards, Fahim
... View more
08-10-2017
11:32 AM
@Suhel , You can try to get report from Ambari Metrics. or another way is to schedule a job to collect memory data on every machine after every 60 seconds. You can use "memfree" or "sar" command to collect memory usage with timestamp and keep appending output to a file and generate report top on that in excel or any other reporting tool to find out memory usage per node. Regards, Fahim
... View more
08-10-2017
11:25 AM
@Mohammed Ashik , You can find out list of application using command. "yarn application -list -appStates ALL" Run this command for every 5 or x minutes and keep appending with time stamp to one file.Once you have data , write a script to find a unique application id and time stamp against it. And find last time stamp against same application id. ( Last time stamp - first time stamp ) per application id will give you running time for application. You can do different type of reporting against this data based on your requirement like "jobs running longer than > 60 minutes." Regards, Fahim
... View more
08-10-2017
10:44 AM
@Mukund Tripathi I am not sure about above error. But if you want to check size of your table you can execute "hdfs dfs -du -h /hbase/data/default/TEST12_2277594" . This should give you size of HBase table. Regards, Fahim
... View more
08-10-2017
09:47 AM
@Dino Bukvic In Ranger , Setting -> Permissions can help to setup group which can have access to Ranger components. Regards, Fahim
... View more
08-10-2017
07:59 AM
Hi @James Ballantine Below is link on creating Hadoop cluster on Virtual Box. It may help to answer your question. https://hadoopmagic.wordpress.com/2013/07/13/how-to-create-hadoop-cluster-on-virtual-machines-running-on-same-laptopdesktop/ Regards, Fahim
... View more
08-10-2017
07:37 AM
@Aishwarya Dixit I think below link describes how to avoid storing password in plain. Check AD , LDAP authentication section. https://zeppelin.apache.org/docs/0.6.2/security/shiroauthentication.html A link to one of password related issue : https://issues.apache.org/jira/browse/ZEPPELIN-530 Regards, Fahim
... View more
08-10-2017
07:31 AM
Hi Hemanth , What I assuming is , you have already created hive table and try to read it from Spark. What I am suggesting is from Spark also you can create a hive table using Spark SQL. Try to create a small hive table using Spark and try to read also. This will prove that your Spark functionality is working correctly with Hive. And issue is with specific table you posted in comment. Regards, Fahim
... View more
08-09-2017
12:36 PM
@Arpan Rajani You can try to create a group for Spark user and give it read permission on HDFS location of hive table using ACL. This should let Spark user read data from Spark-shell. Regards, Fahim
... View more
08-09-2017
12:33 PM
Hi , BI clients are connected with Spark Thrift server. Hive LLAP integration is also increasing. - Major difference would be around processing. Spark will do in memory processing and will need higher amount of memory then Hive LLAP. Thus more costly as infrastructure wise. - Hive is well connected with Ranger PlugIn for security and Ambari. Regards, Fahim
... View more
08-09-2017
12:23 PM
Hi, If you run "ls -ltr" command on each of this file or table names , it will return modified date , you can use this in reporting.Hope this helps. Regards, Fahim
... View more