About Adija1

Adija1 · ‎03-30-2016

Hello We added 5 new DataNodes to our HDP cluster. We run the balancer manually so the newly added DNs will be balanced with the rest (manually because of the Bug in Ambari which is described here https://community.hortonworks.com/articles/4595/balancer-not-working-in-hdfs-ha.html) with the default threshold of 10, and changed the default dfs.datanode.balance.bandwidthPerSec to 100000000. However, it seems that the progress of balancing data from the old DNs to the new ones is roughly 20-30gb a day. It's been running 3 days straight and the newly added disks have only 63gb~ of data. Is there a way to increase the balancer's speed ?? By the way - All of the jobs were stopped so the cluster is completely idle. Thanks in advance. Adi

Adija1 · ‎02-11-2016

@Neeraj Sabharwal Thank you so much for helping !!

Adija1 · ‎02-11-2016

@Ancil McBarnett Ancil you hit the nail right on the head ! Here are the final results: 1. When changing hive auth to LDAP in Ambari - a blank space is added to hive.server2.authentication.ldap.baseDN. That is a problem and must be taken care of! 2. Disabling doAs fixes the auth and also maintains the end user that queries ! (So i'm not sure what's the purpose of this value...) 3. hive.server2.authentication.ldap.Domain doesn't work. Only after removing it and using FQDN for authentication works. Thank you Ancil and thank you all for helping out. This forum is awesome !!

Adija1 · ‎02-11-2016

@Ancil McBarnett Hi Ancil and thank you for your response! Good observation about the blank space ! I removed it and yet unable to authenticate (using beeline). However - i was able to authenticate after disabling the doAs !!! The doAs is important to me because if i understand correct, when doAs is disabled - all of the Hive queries are performed as the hive user and not as the end user. It's a problematic for auditing. I need to know exactly who performed the queries.... I wanna experiment a little bit more and get back to you all with my results. Thanks!!

Adija1 · ‎02-10-2016

@Alex Miller As far as i know HDP 2.3.4 which the latest available form Hortonworks comes with hive 1.2.1 so if i want a higher version of hive i need to install it separately and it won't be managed under Ambari. I'd rather stay with the current 1.2.1 which is pretty high. Anyways - i really would be glad to get in touch with anyone on this planet who has managed to configure AD authentication for hive :)

Adija1 · ‎02-10-2016

@Neeraj Sabharwal I created "hive" user in LDAP with the same password as in my linux machine that runs hive. Still problem remains. Every minute repeatedly the hiveserver2.log shows: LDAP: error code 49 - 80090308: LdapErr: DSID-0C0903A9, comment: AcceptSecurityContext error, data 52e, There must be someone out there who managed to get hive authentication with Active Directory...

Adija1 · ‎02-10-2016

@Neeraj Sabharwal I will reset it's password and create this user in my LDAP. Will update on the result.

Adija1 · ‎02-10-2016

@Neeraj Sabharwal The error means that hive on startup tried to authenticate with a user but it's credentials are not correct. My guess is that the hive which is running on local user "hive" is trying to authenticate using this user (hive) and it doesn't exist on LDAP. Theoretically if i will create "hive" user in my LDAP i guess it will work. The problem is that i'm not sure what it's password is....

Adija1 · ‎02-10-2016

@Neeraj Sabharwal Not exactly. 2016-02-10 14:38:33,237 ERROR [HiveServer2-Handler-Pool: Thread-51]: transport.TSaslTransport (TSaslTransport.java:open(315)) - SASL negotiation failure javax.security.sasl.SaslException: Error validating the login [Caused by javax.security.sasl.AuthenticationException: Error validating LDAP user [Caused by javax.naming.Authe nticationException: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C0903A9, comment: AcceptSecurityContext error, data 52e, v1db1^@]]]

Adija1 · ‎02-10-2016

@Neeraj Sabharwal Disregard the HUE error. It is probably because of old version of HUE that doesn't support LDAP. But still beeline doesn't work and i get the same error you get...

Online	Offline
Last Visited	‎09-20-2018 02:06 PM

Member Since	‎02-04-2016 06:49 AM
Last Visited	‎09-20-2018 02:06 PM
Posts	132
Kudos received	52

Cloudera Community

Re: Connection failed to DataNode:50075 sometimes

Re: decommision datanode and keep other service

Re: Ambari-Agent high cpu & Datanode without heart...

Re: Permissions problem in Capacity Scheduler view...

Re: Ranger Audit stopped working after server rebo...

balancer is slow...

Re: LDAP: error code 49 when setting LDAP auth for...

Re: LDAP: error code 49 when setting LDAP auth for...

Re: LDAP: error code 49 when setting LDAP auth for...

Re: LDAP: error code 49 when setting LDAP auth for...

Re: LDAP: error code 49 when setting LDAP auth for...

Re: LDAP: error code 49 when setting LDAP auth for...

Re: LDAP: error code 49 when setting LDAP auth for...

Re: LDAP: error code 49 when setting LDAP auth for...

Re: LDAP: error code 49 when setting LDAP auth for...