Member since
07-15-2016
43
Posts
3
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2364 | 09-21-2017 05:43 PM | |
1873 | 04-12-2017 03:30 PM |
10-16-2017
06:20 PM
I create a table and tried to insert the data but got the below exception due to case sensitivity. INSERT INTO TABLE X(A,B,C) select A,B,C FROM Y; FAILED: SemanticException 1:27 '[A, B, C]' in insert schema specification are not found among regular columns of default.X nor dynamic partition columns.. Error encountered near token 'C' Whereas the below successful: INSERT INTO TABLE X(a,b,c) select A,B,C FROM Y; After some investigation, I found Hive metastore store table info in lowercase and is case-sensitive. Is there any way I can make sure JDBC drive can handle this scenario? We are using a tool to push data from RDBMS system to Hive and it is generating CREATE & INSERT statements in uppercase. I am working with the tool SME's as well if they can enfore a lowercase schema during code generation.
... View more
Labels:
- Labels:
-
Apache Hive
10-16-2017
06:11 PM
Thank you @subash sharma . I will check that as well..
... View more
10-05-2017
02:23 PM
Update: In my recent changes, you will find the "get_user_query(job_tracker_url)" commented out. Instead, I am fetching the Tracking URL for the job which is taking more time. But, the code is still there. Feel free to play with it.
... View more
10-05-2017
02:19 PM
Hi, Subash, The code will handle passing the job tracker URL. ( it is the job tracker URL ). In resource-manager -> Click on job -> Click on Application Master(Tracking Url). However, after some testing, I found the above code is working MR code only as Tez and Spark jobs redirects to another URL. I need to research a bit to make it work for all kind of apps. I hope this helps! -- Shashant
... View more
09-25-2017
06:31 PM
@Saurabh: Why you want to create 1000+ policies? Can you try grouping the requirements and reduce the policies? Eg: achieve them via making user groups or DB level policies with different user permission set. Ranger API is def a solution for this but I would suggest to revisit the requirement and rule out as many duplicates as you can. Just a suggestion, not saying your requirement is invalid!!!
... View more
09-25-2017
06:21 PM
Check out HW Data Flow(Nifi). I believe it has some processors which can convert JSON object to flow-file and flow-file to SQL(I guess you want to insert into Hive). Processor: ConvertJSONToSQL Description: https://nifi.apache.org/docs.html
... View more
09-25-2017
06:18 PM
HW does not support latest Hue version with HDP anymore. However, HW provides Ambari Views as an alternative (I prefer this option as one less tool to maintain). Ambari Views : https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-views/content/ch_understanding_ambari_views.html Migrating from Hue to Ambari Views: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-views/content/creating_a_htv_instance.html To your original question, I was not able to get Hue with the latest version of HDP. Found several Kerberos issue, ldap integration, etc. However, the same hue.ini worked with an older version of Hue.
... View more
09-21-2017
05:43 PM
Update: I built a custom solution for this by using YARN RM API and by parsing job_tracker & hdfs config properties. Here is the solution for ref: RM-Parser Feel free to fork it or provide any suggestions.
... View more
09-21-2017
05:32 PM
Update: The above worked after correcting the unix user:group mappings and OS level caching. There was a caching issue in RHEL6, which linux admin fixed and after YARN restart everything was in place.
... View more
09-13-2017
02:25 PM
Hello, I am trying to implement yarn user:queue mapping where user1:queue1 and user2:queue2 and mapreduce.queue.name=default. Now all user1 MR jobs are going in default queue whereas user2 MR jobs are going to queue2. I'm confused with the behavior, either both user's MR job should go to default queue or to their respective queue. I have ranger hive policy enabled for the users and yarn queue-override-mapping flag is set to false. HDP version: 2.5.3 Thank you
... View more
Labels:
- Labels:
-
Apache YARN
09-13-2017
01:49 PM
@Thomas Larsson I agree if I set yarn queue-mappings-override property to true. Then I cannot override queue if I want to for some job. Any solution for that?
... View more
09-08-2017
01:19 AM
Hi All, I am exploring YARN REST api's and looking to get hive query from the configuration. I know how to get the query from the UI but can anyone suggest how to get the query from yarn REST api. I am using http://<rm-host>:8088/ws/v1/cluster/apps?states=running&user=hive Thank you.
... View more
Labels:
- Labels:
-
Apache YARN
-
Cloudera Manager
09-05-2017
05:44 PM
NM, I resolved this by downgrading python version from 2.7.5-58 to 2.7.5-48. Earlier we were running 2.7.5-48, OS patching upgraded python version to 2.7.5-58 which started throwing SSL exception. Hope this helps someone in future.
... View more
09-05-2017
04:16 PM
@Chiranjeevi Nimmala What change you made in order to resolve this issue? Should I change to verify=enable or do I need to make some changes in Ambari server/agent confs? Thank you.
... View more
07-13-2017
06:56 PM
1 Kudo
I am running into Java Heap Space issue while running a complex query against a big dataset. The same query works fine from Hive CLI. If I run small queries they run fine in beeline. Is there any way I can increase beeline Java heap size for all users so they do not run out of memory?
... View more
Labels:
- Labels:
-
Apache Hive
04-12-2017
07:03 PM
@arjun more I did not notice the last line. Yes, I took that approach.
... View more
04-12-2017
03:30 PM
The above approach will not work as it requires 'hdp-select'. According to HW, HDP is not allowed on a cluster where HDF is installed and vice-versa. An adequate solution is, to install Apache Hadoop (version same as your HDP). Steps I followed:
Setup Java, A running HDF cluster will not require it but no harm in cross check. Download Hadoop from Apache Mirror, Unpack it and move it to the desired location. Set HADOOP_HOME and HADOOP_CONF_DIR in /etc/profile. By default conf. is set to HADOOP_HOME/etc/hadoop. It is good to keep your configs separate. I created HADOOP_HOME/conf. Important step: Copy existing hdp confs (/etc/hadoop/conf) to HADOOP_HOME/conf. Do not format or start Hadoop as we are connecting to an existing cluster. Last step, set HADOOP_HOME/bin in your user profile file (usually .bash_profile/.profile). That's it, try 'hadoop' or 'hdfs' command. I hope this will help somebody in future!
... View more
04-12-2017
03:07 PM
Thank you @arjun more. But, this is not working on a HDF cluster. I took another approach and install Apache client manually.
... View more
04-11-2017
02:34 PM
Hello, I was following this community post to install Hadoop client without yum. But, with latest hdp repo 2.5.3.0 I am getting the below exception. I want to install HDFS client on our HDF cluster to access HDP cluster hdfs. Any suggestions on approaches or how to do it? I installed the repo using below url's: http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-hdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-libhdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-yarn-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-mapreduce-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-client-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/zookeeper/zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/bigtop-jsvc/bigtop-jsvc-1.0.15-37.el6.x86_64.rpm Install command --> rpm -Uvh hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-hdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-client-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-mapreduce-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-libhdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-yarn-2.7.3.2.5.3.0-37.el6.x86_64.rpm zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch.rpm bigtop-jsvc-1.0.15-37.el6.x86_64.rpm error: Failed dependencies: ranger_2_5_3_0_37-hdfs-plugin is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 ranger_2_5_3_0_37-yarn-plugin is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 hdp-select >= 2.5.3.0-37 is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 spark_2_5_3_0_37-yarn-shuffle is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 spark2_2_5_3_0_37-yarn-shuffle is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 nc is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 hdp-select >= 2.5.3.0-37 is needed by zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch Thank you in advance!
... View more
Labels:
- Labels:
-
Apache Hadoop
04-06-2017
06:53 PM
Thank you @Namit Maheshwari.
... View more
04-06-2017
06:53 PM
Thank you guys for the prompt response. Root cause: Somehow Kerberos admin session was expired which was not creating/setup any key tabs. Resolution: I fixed it by restarting Ambari. After that, regeneration of ticket resolved the problem.
... View more
04-06-2017
05:25 PM
I installed Ranger & Ambari-Infra in my cluster via Ambari but installed hung up on Setup Keytab. Although, the service installed properly but "ranger-usersync" failed due to service.keytab does not exist. Can anyone suggest how to re-generate keytab for a particular service? HDP - 2.5.3, Ambari - 2.4.2.0
... View more
Labels:
- Labels:
-
Apache Ranger
03-27-2017
08:29 PM
@vperiasamy -- Agree, that's what I understood hard way 🙂 I though Ranger will sync groups with users as well as users without groups. So should I disable group search first and keep user & user-group mapping. Any suggestions?
... View more
03-27-2017
08:10 PM
@vperiasamy - Check my update. I believe the problem was "Users not belonging to a group". But, I did not understand why only users with groups were pulled. I would love to know the details if someone has an explanation how Ranger UserSync works.
... View more
03-27-2017
08:08 PM
1 Kudo
Update -- I got the new users by disabling "Group Search First" and "Enable User Search". Although, I would like to know what the difference between them and how exactly Ranger UserSync syncs users and group. Thank you!
... View more
03-27-2017
07:37 PM
Hi, I have Ranger UserSync enabled with LDAP. Everything was fine until I added a new user in LDAP. After adding, I am unable to find the user in Ranger. I tried restarting user-sync multiple times but no success. I checked LDAP using search filters and the user exists with correct details. Any suggestions for debugging this issue or has anyone faced something like this in past? Nothing helpful in ranger/user-sync & ranger/admin logs. Looks like users with groups are added. I see logs from "LdapUserGroupBuilder" class.
... View more
Labels:
- Labels:
-
Apache Ranger
02-09-2017
10:16 PM
Thank you @mbalakrishnan. A simple use case is to limit the space for different teams. I do not want one team to utilize the entire cluster space. For eg: Team A 25%, Team B 25% and Team C 50%. Something like YARN Capacity queues. Space quota per team for cost management. I will check the same for HDFS. Can you refer some links to check HDFS user level quota?
... View more
02-09-2017
04:20 PM
Update: I checked Ranger as well but did not find any setup related to space limitation.
... View more