About shashant_panwar

shashant_panwar · ‎10-27-2017

Hello, I am looking to export extracts from hive databases in csv or txt format using hql files. In Hive, you can achieve this by running below command: hive -f test.hql >> test.csv I tried the below command in Spark which works for a smaller dataset but fails if I try to run a complex query on a bigger data. I get out of memory exception from Driver. spark-sql --name my-extract --num-executors 20 --master yarn --executor-memory 5G --driver-memory 10G --queue batch -S -f test.hql >> test.csv I saw some solutions using Scala code and submit it through spark-submit, but users are more comfortable with SQL and running hive -f kinda export. Thanks in advance!!

shashant_panwar · ‎10-16-2017

I create a table and tried to insert the data but got the below exception due to case sensitivity. INSERT INTO TABLE X(A,B,C) select A,B,C FROM Y; FAILED: SemanticException 1:27 '[A, B, C]' in insert schema specification are not found among regular columns of default.X nor dynamic partition columns.. Error encountered near token 'C' Whereas the below successful: INSERT INTO TABLE X(a,b,c) select A,B,C FROM Y; After some investigation, I found Hive metastore store table info in lowercase and is case-sensitive. Is there any way I can make sure JDBC drive can handle this scenario? We are using a tool to push data from RDBMS system to Hive and it is generating CREATE & INSERT statements in uppercase. I am working with the tool SME's as well if they can enfore a lowercase schema during code generation.

shashant_panwar · ‎10-16-2017

Thank you @subash sharma . I will check that as well..

shashant_panwar · ‎10-05-2017

Update: In my recent changes, you will find the "get_user_query(job_tracker_url)" commented out. Instead, I am fetching the Tracking URL for the job which is taking more time. But, the code is still there. Feel free to play with it.

shashant_panwar · ‎10-05-2017

Hi, Subash, The code will handle passing the job tracker URL. ( it is the job tracker URL ). In resource-manager -> Click on job -> Click on Application Master(Tracking Url). However, after some testing, I found the above code is working MR code only as Tez and Spark jobs redirects to another URL. I need to research a bit to make it work for all kind of apps. I hope this helps! -- Shashant

shashant_panwar · ‎09-25-2017

@Saurabh: Why you want to create 1000+ policies? Can you try grouping the requirements and reduce the policies? Eg: achieve them via making user groups or DB level policies with different user permission set. Ranger API is def a solution for this but I would suggest to revisit the requirement and rule out as many duplicates as you can. Just a suggestion, not saying your requirement is invalid!!!

shashant_panwar · ‎09-25-2017

Check out HW Data Flow(Nifi). I believe it has some processors which can convert JSON object to flow-file and flow-file to SQL(I guess you want to insert into Hive). Processor: ConvertJSONToSQL Description: https://nifi.apache.org/docs.html

shashant_panwar · ‎09-25-2017

HW does not support latest Hue version with HDP anymore. However, HW provides Ambari Views as an alternative (I prefer this option as one less tool to maintain). Ambari Views : https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-views/content/ch_understanding_ambari_views.html Migrating from Hue to Ambari Views: https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-views/content/creating_a_htv_instance.html To your original question, I was not able to get Hue with the latest version of HDP. Found several Kerberos issue, ldap integration, etc. However, the same hue.ini worked with an older version of Hue.

shashant_panwar · ‎09-21-2017

Update: I built a custom solution for this by using YARN RM API and by parsing job_tracker & hdfs config properties. Here is the solution for ref: RM-Parser Feel free to fork it or provide any suggestions.

shashant_panwar · ‎09-21-2017

Update: The above worked after correcting the unix user:group mappings and OS level caching. There was a caching issue in RHEL6, which linux admin fixed and after YARN restart everything was in place.

shashant_panwar · ‎09-13-2017

Hello, I am trying to implement yarn user:queue mapping where user1:queue1 and user2:queue2 and mapreduce.queue.name=default. Now all user1 MR jobs are going in default queue whereas user2 MR jobs are going to queue2. I'm confused with the behavior, either both user's MR job should go to default queue or to their respective queue. I have ranger hive policy enabled for the users and yarn queue-override-mapping flag is set to false. HDP version: 2.5.3 Thank you

shashant_panwar · ‎09-13-2017

@Thomas Larsson I agree if I set yarn queue-mappings-override property to true. Then I cannot override queue if I want to for some job. Any solution for that?

shashant_panwar · ‎09-08-2017

Hi All, I am exploring YARN REST api's and looking to get hive query from the configuration. I know how to get the query from the UI but can anyone suggest how to get the query from yarn REST api. I am using http://<rm-host>:8088/ws/v1/cluster/apps?states=running&user=hive Thank you.

shashant_panwar · ‎09-05-2017

NM, I resolved this by downgrading python version from 2.7.5-58 to 2.7.5-48. Earlier we were running 2.7.5-48, OS patching upgraded python version to 2.7.5-58 which started throwing SSL exception. Hope this helps someone in future.

shashant_panwar · ‎09-05-2017

@Chiranjeevi Nimmala What change you made in order to resolve this issue? Should I change to verify=enable or do I need to make some changes in Ambari server/agent confs? Thank you.

shashant_panwar · ‎07-13-2017

I am running into Java Heap Space issue while running a complex query against a big dataset. The same query works fine from Hive CLI. If I run small queries they run fine in beeline. Is there any way I can increase beeline Java heap size for all users so they do not run out of memory?

shashant_panwar · ‎06-06-2017

Ambari will take care of the creation of Kerberos ticket for a newly added service. In order to run any job in Kerberos enable environment, you need to have the authorization to access and run the job which means access to the valid ticket. Check this link : https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_Security_Guide/content/_kerberos_overview.html

shashant_panwar · ‎06-06-2017

Check this documentation: Ambari Views Documentation

shashant_panwar · ‎04-12-2017

@arjun more I did not notice the last line. Yes, I took that approach.

shashant_panwar · ‎04-12-2017

The above approach will not work as it requires 'hdp-select'. According to HW, HDP is not allowed on a cluster where HDF is installed and vice-versa. An adequate solution is, to install Apache Hadoop (version same as your HDP). Steps I followed: Setup Java, A running HDF cluster will not require it but no harm in cross check. Download Hadoop from Apache Mirror, Unpack it and move it to the desired location. Set HADOOP_HOME and HADOOP_CONF_DIR in /etc/profile. By default conf. is set to HADOOP_HOME/etc/hadoop. It is good to keep your configs separate. I created HADOOP_HOME/conf. Important step: Copy existing hdp confs (/etc/hadoop/conf) to HADOOP_HOME/conf. Do not format or start Hadoop as we are connecting to an existing cluster. Last step, set HADOOP_HOME/bin in your user profile file (usually .bash_profile/.profile). That's it, try 'hadoop' or 'hdfs' command. I hope this will help somebody in future!

shashant_panwar · ‎04-12-2017

Thank you @arjun more. But, this is not working on a HDF cluster. I took another approach and install Apache client manually.

shashant_panwar · ‎04-11-2017

Hello, I was following this community post to install Hadoop client without yum. But, with latest hdp repo 2.5.3.0 I am getting the below exception. I want to install HDFS client on our HDF cluster to access HDP cluster hdfs. Any suggestions on approaches or how to do it? I installed the repo using below url's: http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-hdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-libhdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-yarn-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-mapreduce-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/hadoop/hadoop_2_5_3_0_37-client-2.7.3.2.5.3.0-37.el6.x86_64.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/zookeeper/zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch.rpm http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/bigtop-jsvc/bigtop-jsvc-1.0.15-37.el6.x86_64.rpm Install command --> rpm -Uvh hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-hdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-client-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-mapreduce-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-libhdfs-2.7.3.2.5.3.0-37.el6.x86_64.rpm hadoop_2_5_3_0_37-yarn-2.7.3.2.5.3.0-37.el6.x86_64.rpm zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch.rpm bigtop-jsvc-1.0.15-37.el6.x86_64.rpm error: Failed dependencies: ranger_2_5_3_0_37-hdfs-plugin is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 ranger_2_5_3_0_37-yarn-plugin is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 hdp-select >= 2.5.3.0-37 is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 spark_2_5_3_0_37-yarn-shuffle is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 spark2_2_5_3_0_37-yarn-shuffle is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 nc is needed by hadoop_2_5_3_0_37-2.7.3.2.5.3.0-37.el6.x86_64 hdp-select >= 2.5.3.0-37 is needed by zookeeper_2_5_3_0_37-3.4.6.2.5.3.0-37.el6.noarch Thank you in advance!

shashant_panwar · ‎04-06-2017

Thank you @Namit Maheshwari.

shashant_panwar · ‎04-06-2017

Thank you guys for the prompt response. Root cause: Somehow Kerberos admin session was expired which was not creating/setup any key tabs. Resolution: I fixed it by restarting Ambari. After that, regeneration of ticket resolved the problem.

shashant_panwar · ‎04-06-2017

I installed Ranger & Ambari-Infra in my cluster via Ambari but installed hung up on Setup Keytab. Although, the service installed properly but "ranger-usersync" failed due to service.keytab does not exist. Can anyone suggest how to re-generate keytab for a particular service? HDP - 2.5.3, Ambari - 2.4.2.0

shashant_panwar · ‎03-27-2017

@vperiasamy -- Agree, that's what I understood hard way 🙂 I though Ranger will sync groups with users as well as users without groups. So should I disable group search first and keep user & user-group mapping. Any suggestions?

shashant_panwar · ‎03-27-2017

@vperiasamy - Check my update. I believe the problem was "Users not belonging to a group". But, I did not understand why only users with groups were pulled. I would love to know the details if someone has an explanation how Ranger UserSync works.

shashant_panwar · ‎03-27-2017

Update -- I got the new users by disabling "Group Search First" and "Enable User Search". Although, I would like to know what the difference between them and how exactly Ranger UserSync syncs users and group. Thank you!

shashant_panwar · ‎03-27-2017

Hi, I have Ranger UserSync enabled with LDAP. Everything was fine until I added a new user in LDAP. After adding, I am unable to find the user in Ranger. I tried restarting user-sync multiple times but no success. I checked LDAP using search filters and the user exists with correct details. Any suggestions for debugging this issue or has anyone faced something like this in past? Nothing helpful in ranger/user-sync & ranger/admin logs. Looks like users with groups are added. I see logs from "LdapUserGroupBuilder" class.

shashant_panwar · ‎02-10-2017

Thank you @mbalakrishnan

Online	Offline
Last Visited	‎11-21-2017 04:53 PM

Date Registered	‎07-15-2016 05:50 PM
Last Visited	‎11-21-2017 04:53 PM
Total Messages Posted	43
Total Kudos Received	3

Re: Get hive query from resource manager using yar...

Re: Install HDFS client without using yum on HDF?

Run .hql file in Spark and store result as txt or ...

Hive Insert statement with or without select case ...

Re: Get hive query from resource manager using yar...

Re: Get hive query from resource manager using yar...

Re: Get hive query from resource manager using yar...

Re: How to create many ranger policies and add use...

Re: Convert Json to sql format

Re: Hue 4 on HDP 2.6.x

Re: Get hive query from resource manager using yar...

Re: Yarn user queue mapping

Yarn user queue mapping

Re: Yarn queue Mapping

Get hive query from resource manager using yarn RE...

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED]...

Re: Ambari agent- [SSL: CERTIFICATE_VERIFY_FAILED]...

How to increase beeline java heap size?

Re: Using HDP after enabling kerberos

Re: Using HDP after enabling kerberos

Re: Install HDFS client without using yum on HDF?

Re: Install HDFS client without using yum on HDF?

Re: Install HDFS client without using yum on HDF?

Install HDFS client without using yum on HDF?

Re: Regenerate Kerberos Keytab for a particular se...

Re: Regenerate Kerberos Keytab for a particular se...

Regenerate Kerberos Keytab for a particular servic...

Re: Ranger UserSync Issue - LDAP

Re: Ranger UserSync Issue - LDAP

Re: Ranger UserSync Issue - LDAP

Ranger UserSync Issue - LDAP

Re: Hive Database Size