We have set a Once way trust from AD to KDC and validated that we were able to get tickets from AD. Kerberos Realm - ABC.NET AD Domain - XZY.COM Validation kinit ad_user@XYZ.COM List Showed the valid ticket. Now in just few hrs the trust is broken the same command, kinit ad_user@XYZ.COM, gives kinit: Cannot find KDC for realm "XYZ.COM" while getting initial credentials. Thanks in advance. Just want to hit myself on the face. ... View more

We are running Ranger on Oracle. Faced a common error (NN logs and Hive logs) ORA-12899: Value too large for column "RANGERDBA"."XA.AUDIT"."REQUEST.DATA" (actual: 2465, maximum:2000) We changed the REQUEST_DATA to VARCHAR2(3000) from VARCHAR2(2000) Now the new error is ORA-12899: Value too large for column "RANGERDBA"."XA.AUDIT"."REQUEST.DATA" (actual: 3465, maximum:3000) This a a fresh Installation on Oracle only and not migrated from MySql. In my past exp we had the same issue where the DB was migrated from MySql and there were some NON ASCII chars .. we fixed it by changing to VARCHAR2 (2000 CHAR) from VARCHAR2(2000). Will the same solution be able to fix this. What can cause this issue. Thanks Mayank ... View more

While securing Ambari Sever for Https, we can successfully login to https and default port 8443, however the https is stoked out and says This page is insecure (broken HTTPS). We are using wildcard certs initially in .cer format however have to convert it to .pem format using openssl. What is the preferred format and encryption for the Certs. The current error says 1) SHA-1 Certificate The certificate for this site expires in 2017 or later, and the certificate chain contains a certificate signed using SHA-1. 2) Certificate Error There are issues with the site's certificate chain (net::ERR_CERT_COMMON_NAME_INVALID). Thanks Mayank ... View more

Hello experts, While running balancer utility as HDFS user i'm getting below error. HDFS Balancer - Access denied for user hdfs-prod. Superuser privilege is required Note - I'm running as user hdfs against a kerberized cluster. My principal name is hdfs-prod@ABC.NET dfs.permissions.superusergroup = hdfs and hdfs groups hdfs shows hdfs : hadoop hdfs I'm sure I'm missing something here. Thanks Mayank ... View more

Hello Gurus, One of our clients is asking for Zepplin and Kerberized the cluster. Below are my doubts (I have searched the forums however not found anything solid) 1) What is the current state of security around Zeppelin 2) Can we run Zellpin in a Kerberized environment. Thanks Mayank ... View more

Hello experts, I'm trying to understand the total size / block size used by HDFS Snapshot. I have a dir like /user/x/data and a hdfs ls tells me it has 1.1 TB So If I take a snapshot of /user/x/data will the snapshot consumes same space and how much block size is used by it. My earlier output from hdfs dfsadmin -report was 19.6 TB and after taking snapshot it was still same. If snapshots takes same space as of the source why the report does't changes. Thanks Mayank ... View more

Hello experts, I feel confident with Kerberos Authentication, however a recent article has created a panic among few customers, I would like to understand if there is a real threat and how others are thinking through it. http://news4security.com/posts/2015/12/old-microsoft-kerberos-vulnerability-gets-new-spotlight/ The article talks about various ways to attack Kerberos and obtain or pass forged tickets. Would be real helpful if security experts can clear the air, specially what these threat means in hadoop world (if any) Thanks Mayank ... View more

What are the best practices around copying data between two clusters located in different datacenter on different LAN, the scope is to limit loops. ... View more

One of our clients have asked us to move the prefixed log location to a different mounted point for all the service logs, for example the prefix log location for hdfs is moved from /var/log/hadoop to /hdp/logs/hadoop via api calls. Everything restarted smoothly however only one NM is coming up out of 5, and a manual restart only works on the first NM. All other NM are through the same error, below; STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r ef0582ca14b8177a3cbb6376807545272677d730; compiled by 'jenkins' on 2015-12-16T03:01Z STARTUP_MSG: java = 1.7.0_67 ************************************************************/ 2016-01-26 15:01:25,155 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - registered UNIX signal handlers for [TERM, HUP, INT] 2016-01-26 15:01:26,283 INFO recovery.NMLeveldbStateStoreService (NMLeveldbStateStoreService.java:initStorage(927)) - Using state database at /hdp/logs/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state for recovery 2016-01-26 15:01:26,313 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService failed in state INITED; cause: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /hdp/logs/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/LOCK: Resource temporarily unavailable org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /hdp/logs/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/LOCK: Resource temporarily unavailable at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:178) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:220) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:537) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:585) 2016-01-26 15:01:26,316 INFO service.AbstractService (AbstractService.java:noteFailure(272)) - Service NodeManager failed in state INITED; cause: org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /hdp/logs/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/LOCK: Resource temporarily unavailable org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /hdp/logs/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/LOCK: Resource temporarily unavailable at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:178) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:220) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:537) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:585) Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /hdp/logs/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/LOCK: Resource temporarily unavailable at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 5 more 2016-01-26 15:01:26,317 FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(540)) - Error starting NodeManager org.apache.hadoop.service.ServiceStateException: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /hdp/logs/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/LOCK: Resource temporarily unavailable at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:178) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:220) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:537) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:585) Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /hdp/logs/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state/LOCK: Resource temporarily unavailable at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930) at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 5 more 2016-01-26 15:01:26,319 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NodeManager at bvluxhdpdn05.conocophillips.net/158.139.121.115 ************************************************************/ " As we can see it is not complaining about LOCK file bot present but unavailable, as whichever NM starts first acquire this LOCK (remember this is a single mount point and not local file-system) If I change the log location back to local file-system even for example /tmp/yarnlogs its works smooth since all the NM get access to LOCK file on local file-system where ever they are installed. Has someone faces this issue and can you please suggest a fix to this. Thanks Mayank ... View more

Few weeks back we were working with a customer and configured Solr for Ranger, customer decided to skip Solr until it came as a GA feature with Ambari, after uninstalling and removing the Solr properties customer has upgraded Ambari and HDP stack, now the ambari-server logs are flooded with below error, a solution/fix would be helpful. ERROR [qtp-ambari-client-2103] ClusterImpl:2145 - Config inconsistency exists: unknown configType=solr-env Thanks Mayank ... View more