Hi, we have BI tools in which the configuration file has primary namenode and seondary namenode hardcoded. We are seeing a lot of "read operation in standby not allowed errors" - i guess this will occur when client tries to first check which namenode is active and could throw error if it tries to conenct to standby. But we are seeing more than usual when it comes to this error. But jobs are running fine. I would like to know if it is cluster side issue with namenode failing over too many times? or BI tool config issue where we are hardcoding nn and snn, instead of service name. Thanks. ... View more

Hi, We use an application which uses Hive and spark jobs. We want to monitor from our application (edge node of hadoop) to check if the connection to hive and spark working or not using script... hive uses beeline with logins,we cannot use login in the script, is there any other way to monitor hive and spark jdbc connections? using curl? Please suggest. ... View more

I have a user user1 with group group1 I use a third party tool which writes spark event logs to a directory in hdfs. Currently using the user home directory to write the logs temporarily. I created this directory like: Hdfs dfs -mkdir /user/user1/sparkeventlogs Right now this is created under user1:group2 Changed ownership to the right group: Hdfs dfs -chown -R user1:group1 /user/user1/sparkeventlogs I also added ACL to the above directory using setfacl command and when I do getfacl it gives me correct user and group assigned. User and group both have rwx permissions Now when the job is run, it the giving permission denied with a user who ran it under that AD group saying User1:group2 drwx——— When actually it is User1:group1 drwxrwx—- We have ranger enabled but I don’t have access to it Thanks for you help ... View more

Hi All, I am trying to understand how does a tool like Atscale(data wrangling tool) connect to Hive metastore? There is a weird behavior in the log files of Atscale, when you try to wrangle data through Atscale, it is connecting and disconnecting from hive metastore every few seconds.... is this normal? when a user submits a query when is the connection made to hive metastore and when does it ends? and how often can this happen? in case of Atscale, there could be aggregates, not sure how this works.. Thanks in advance. ... View more

Hi all, As soon as i enabled hive acid properties and restarted hive services, i am getting these logs in hiveserver2 and resource manager. although resource manager is configured 8050 port, it is looking for 8032 and retrying ... also hive is unable to submit an application in a queue. 2018-05-21 14:17:01,965 INFO [main]: retry.RetryInvocationHandler (RetryInvocationHandler.java:log(267)) - Exception while invoking ApplicationClientProtocolPBClientImpl.submitApplication over rm1 after 24 failover attempts. Trying to failover immediately. org.apache.hadoop.security.AccessControlException: User hive does not have permission to submit application_1526926153151_0001 to queue AdHoc at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:292) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:585) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:239) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy42.submitApplication(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:259) at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72) at org.apache.tez.client.TezClient.start(TezClient.java:437) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:117) at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:76) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:488) at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:87) at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:720) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:593) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): User hive does not have permission to submit application_1526926153151_0001 to queue AdHoc at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:292) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:585) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552) at org.apache.hadoop.ipc.Client.call(Client.java:1496) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy41.submitApplication(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:236) ... 23 more 2018-05-21 14:18:13,106 INFO [main]: server.HiveServer2 (HiveServer2.java:stop(405)) - Shutting down HiveServer2 2018-05-21 14:18:13,106 INFO [main]: thrift.ThriftCLIService (ThriftCLIService.java:stop(218)) - Thrift server has stopped 2018-05-21 14:18:13,106 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped. 2018-05-21 14:18:13,107 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped. 2018-05-21 14:18:13,107 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped. 2018-05-21 14:18:23,108 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped. 2018-05-21 14:18:23,108 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(824)) - 0: Shutting down the object store... 2018-05-21 14:18:23,108 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(393)) - ugi=hive ip=unknown-ip-addr cmd=Shutting down the object store... 2018-05-21 14:18:23,109 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(824)) - 0: Metastore shutdown complete. 2018-05-21 14:18:23,109 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(393)) - ugi=hive ip=unknown-ip-addr cmd=Metastore shutdown complete. 2018-05-21 14:18:23,109 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped. 2018-05-21 14:18:23,117 WARN [main-EventThread]: server.HiveServer2 (HiveServer2.java:process(335)) - This HiveServer2 instance is now de-registered from ZooKeeper. The server will be shut down after the last client sesssion completes. 2018-05-21 14:18:23,117 WARN [main-EventThread]: server.HiveServer2 (HiveServer2.java:process(343)) - This instance of HiveServer2 has been removed from the list of server instances available for dynamic service discovery. The last client session has ended - will shutdown now. 2018-05-21 14:18:23,120 INFO [main]: zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x2620074a7ab1b48 closed 2018-05-21 14:18:23,120 INFO [main]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(361)) - Server instance removed from ZooKeeper. 2018-05-21 14:18:23,120 INFO [main-EventThread]: server.HiveServer2 (HiveServer2.java:stop(405)) - Shutting down HiveServer2 2018-05-21 14:18:23,120 WARN [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(508)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds org.apache.hadoop.security.AccessControlException: User hive does not have permission to submit application_1526926153151_0001 to queue AdHoc ... View more

Hi, i just enabled hive interactive querying on hdp 2.5.3, i notice hive cli is automaticlaly disabled, i am not able to login regularly with hive cli. I can only do a jdbc login through beeline.... but i am not sure how to run queries on hive llap? how is running a query or writing scripts to run some jobs is done with hive llap? Please help. ... View more

Hi, We have 2 different hadoop clusters(cloudera & hdp) , i have to run an import query on Impala twce a day to import that data into our hadoop cluster. What is the best way to do this? I see sqoop only imports from relational db, may be distcp but what if i want to do a query with where conditions? Is there any other way? if i decide to get all data from partition using distcp anyways, wghat ports should be open on both clusters? Thanks in advance. ... View more

Hi all, i am using sqoop --direct mode now to copy nz tables into hive. If i run 4 parallel sqoop jobs, how does it affect hdp cluster and also how does it effect nz database? also, is there a better and fastest way to import data from nz? ... View more

i have few questions about capacity scheduler: 1. when i have parent queue and child queues, and when i want to use a child queue, do i need to mention the whole qeue path? like: tez.queue.name = parentqueue.childqueue or just childqueue is fine? 2. i do not have yarn acl enabled, but i want to use queue mappings to certain users, is this how it works: queue A: no mention of queue mappings to any user queueB: no mention of queue mappings to any user queueC:user2 and user3 are mapped to queueC Here queue override is disabled. IN this case a universal user (user1) can still access all queues right? because he doesnt have user mappings set, whereas user2 and user3 can ONLY access queueC. Please correct me. 3. do i need to enable yarn acl? because with yarn acl disabled i could do queue mappings but i could easily override the quueues even though they are disabled? if i want a user to access all queues in this case i just need to mention it in the acl submit applications of all queues with that user name. correct? ... View more

Hi, I am using sqoop to import data from netezza, everything works fine but issue is it uses queue 'A'(which is mentioned in the sqoop import) for pulling data from nz and then while loading data into hive, it is using "default" queue. It is not a huge issue but i am having issues with yarn.scheduler.capacity.maximum-am-resource-percent where the queue is overloaded with different jobs. There seems to be 2 solutions here, one being increase the max am resource percent for the default queue or change the load to hive in the sqoop job to use queue 'A'. how can i change it? ... View more

Hi all, I have RM HA enabled. Everythign was working fine until 2 times my resource manager(one of the two) stops with the following exception. When i restart it, it starts back again. How to resolve this? (happened 2 times in the past 2 weeks) 2018-03-05 19:32:42,343 FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(190)) - Error in dispatcher thread java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b8f06c5 rejected from java.util.concurrent.ThreadPoolExecutor@68b47cd2[Shutting down, pool size = 34, active threads = 12, queued tasks = 0, completed tasks = 73728] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at org.apache.hadoop.registry.server.services.RegistryAdminService.submit(RegistryAdminService.java:176) at org.apache.hadoop.registry.server.integration.RMRegistryOperationsService.purgeRecordsAsync(RMRegistryOperationsService.java:200) at org.apache.hadoop.registry.server.integration.RMRegistryOperationsService.purgeRecordsAsync(RMRegistryOperationsService.java:170) at org.apache.hadoop.registry.server.integration.RMRegistryOperationsService.purgeAppAttemptRecords(RMRegistryOperationsService.java:95) at org.apache.hadoop.registry.server.integration.RMRegistryOperationsService.onApplicationAttemptUnregistered(RMRegistryOperationsService.java:109) at org.apache.hadoop.yarn.server.resourcemanager.registry.RMRegistryService.handleAppAttemptEvent(RMRegistryService.java:143) at org.apache.hadoop.yarn.server.resourcemanager.registry.RMRegistryService$AppEventHandler.handle(RMRegistryService.java:183) at org.apache.hadoop.yarn.server.resourcemanager.registry.RMRegistryService$AppEventHandler.handle(RMRegistryService.java:177) at org.apache.hadoop.yarn.event.AsyncDispatcher$MultiListenerHandler.handle(AsyncDispatcher.java:279) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110) at java.lang.Thread.run(Thread.java:745) 2018-03-05 19:32:42,349 INFO event.AsyncDispatcher (AsyncDispatcher.java:run(293)) - Exiting, bbye.. Thanks. ... View more