Hi, can someone help explain about dead executors in Spark and their behavior?Can it slow down a job? How to prevent dead executors? ... View more

Hi All, Need help finding yarn queue usage of all queues from either rm rest api or ams api.   Basically i need to run a script to pull this data and see if the  yarn queues are going beyond a threshold value and send out an alert.   I cannot use ambari alerts here. Need it in a script form. Appreciate help. ... View more

Hi, we have BI tools in which the configuration file has primary namenode and seondary namenode hardcoded. We are seeing a lot of "read operation in standby not allowed errors" - i guess this will occur when client tries to first check which namenode is active and could throw error if it tries to conenct to standby. But we are seeing more than usual when it comes to this error. But jobs are running fine. I would like to know if it is cluster side issue with namenode failing over too many times? or BI tool config issue where we are hardcoding nn and snn, instead of service name. Thanks. ... View more

Hi All, I have access to grafana and other stuff but we use a BI tool which uses separate yarn queue, i want to write a script or get data into csv which contains the queue utilization (only 1 specific queue) and other metrics related to this queue as a separate report either from grafana or any other means. Can somebody suggest a solution? I would like to update the metrics every 5 min. Thanks. ... View more

Hi, We use an application which uses Hive and spark jobs. We want to monitor from our application (edge node of hadoop) to check if the connection to hive and spark working or not using script... hive uses beeline with logins,we cannot use login in the script, is there any other way to monitor hive and spark jdbc connections? using curl? Please suggest. ... View more

I have a user user1 with group group1 I use a third party tool which writes spark event logs to a directory in hdfs. Currently using the user home directory to write the logs temporarily. I created this directory like: Hdfs dfs -mkdir /user/user1/sparkeventlogs Right now this is created under user1:group2 Changed ownership to the right group: Hdfs dfs -chown -R user1:group1 /user/user1/sparkeventlogs I also added ACL to the above directory using setfacl command and when I do getfacl it gives me correct user and group assigned. User and group both have rwx permissions Now when the job is run, it the giving permission denied with a user who ran it under that AD group saying User1:group2 drwx——— When actually it is User1:group1 drwxrwx—- We have ranger enabled but I don’t have access to it Thanks for you help ... View more

Hi All, I am trying to understand how does a tool like Atscale(data wrangling tool) connect to Hive metastore? There is a weird behavior in the log files of Atscale, when you try to wrangle data through Atscale, it is connecting and disconnecting from hive metastore every few seconds.... is this normal? when a user submits a query when is the connection made to hive metastore and when does it ends? and how often can this happen? in case of Atscale, there could be aggregates, not sure how this works.. Thanks in advance. ... View more

Hi all, As soon as i enabled hive acid properties and restarted hive services, i am getting these logs in hiveserver2 and resource manager. although resource manager is configured 8050 port, it is looking for 8032 and retrying ... also hive is unable to submit an application in a queue. 2018-05-21 14:17:01,965 INFO [main]: retry.RetryInvocationHandler (RetryInvocationHandler.java:log(267)) - Exception while invoking ApplicationClientProtocolPBClientImpl.submitApplication over rm1 after 24 failover attempts. Trying to failover immediately. org.apache.hadoop.security.AccessControlException: User hive does not have permission to submit application_1526926153151_0001 to queue AdHoc at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:292) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:585) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53) at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:239) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy42.submitApplication(Unknown Source) at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:259) at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72) at org.apache.tez.client.TezClient.start(TezClient.java:437) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:196) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:117) at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:76) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:488) at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:87) at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:720) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:593) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): User hive does not have permission to submit application_1526926153151_0001 to queue AdHoc at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:380) at org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:292) at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:585) at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218) at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552) at org.apache.hadoop.ipc.Client.call(Client.java:1496) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy41.submitApplication(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.submitApplication(ApplicationClientProtocolPBClientImpl.java:236) ... 23 more 2018-05-21 14:18:13,106 INFO [main]: server.HiveServer2 (HiveServer2.java:stop(405)) - Shutting down HiveServer2 2018-05-21 14:18:13,106 INFO [main]: thrift.ThriftCLIService (ThriftCLIService.java:stop(218)) - Thrift server has stopped 2018-05-21 14:18:13,106 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:ThriftBinaryCLIService is stopped. 2018-05-21 14:18:13,107 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:OperationManager is stopped. 2018-05-21 14:18:13,107 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:SessionManager is stopped. 2018-05-21 14:18:23,108 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:CLIService is stopped. 2018-05-21 14:18:23,108 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(824)) - 0: Shutting down the object store... 2018-05-21 14:18:23,108 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(393)) - ugi=hive ip=unknown-ip-addr cmd=Shutting down the object store... 2018-05-21 14:18:23,109 INFO [main]: metastore.HiveMetaStore (HiveMetaStore.java:logInfo(824)) - 0: Metastore shutdown complete. 2018-05-21 14:18:23,109 INFO [main]: HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(393)) - ugi=hive ip=unknown-ip-addr cmd=Metastore shutdown complete. 2018-05-21 14:18:23,109 INFO [main]: service.AbstractService (AbstractService.java:stop(125)) - Service:HiveServer2 is stopped. 2018-05-21 14:18:23,117 WARN [main-EventThread]: server.HiveServer2 (HiveServer2.java:process(335)) - This HiveServer2 instance is now de-registered from ZooKeeper. The server will be shut down after the last client sesssion completes. 2018-05-21 14:18:23,117 WARN [main-EventThread]: server.HiveServer2 (HiveServer2.java:process(343)) - This instance of HiveServer2 has been removed from the list of server instances available for dynamic service discovery. The last client session has ended - will shutdown now. 2018-05-21 14:18:23,120 INFO [main]: zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x2620074a7ab1b48 closed 2018-05-21 14:18:23,120 INFO [main]: server.HiveServer2 (HiveServer2.java:removeServerInstanceFromZooKeeper(361)) - Server instance removed from ZooKeeper. 2018-05-21 14:18:23,120 INFO [main-EventThread]: server.HiveServer2 (HiveServer2.java:stop(405)) - Shutting down HiveServer2 2018-05-21 14:18:23,120 WARN [main]: server.HiveServer2 (HiveServer2.java:startHiveServer2(508)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds org.apache.hadoop.security.AccessControlException: User hive does not have permission to submit application_1526926153151_0001 to queue AdHoc ... View more

Hi, i just enabled hive interactive querying on hdp 2.5.3, i notice hive cli is automaticlaly disabled, i am not able to login regularly with hive cli. I can only do a jdbc login through beeline.... but i am not sure how to run queries on hive llap? how is running a query or writing scripts to run some jobs is done with hive llap? Please help. ... View more

Hi, We have 2 different hadoop clusters(cloudera & hdp) , i have to run an import query on Impala twce a day to import that data into our hadoop cluster. What is the best way to do this? I see sqoop only imports from relational db, may be distcp but what if i want to do a query with where conditions? Is there any other way? if i decide to get all data from partition using distcp anyways, wghat ports should be open on both clusters? Thanks in advance. ... View more

Hi all, i am using sqoop --direct mode now to copy nz tables into hive. If i run 4 parallel sqoop jobs, how does it affect hdp cluster and also how does it effect nz database? also, is there a better and fastest way to import data from nz? ... View more

Hi, can somebody explain me how conda packages and env work ? i am using hdp 2.5.3 with python 2.6 I am planning to use nltk which seem to work with python 3 and above. Is there any other suggestion other than conda env for a production environment? Thanks. ... View more