Hi! Some time ago we decided to move few of our udf packages to auxlib reloadable directory to enable updating some functions without restart of Hiveservers. Since then however we experience random errors like shown in following example: 2019-04-04 22:33:18,103 INFO org.apache.hadoop.hive.ql.Driver: [HiveServer2-Handler-Pool: Thread-117]: Completed compiling command(queryId=hive_20190404223333_ed9b3085-fc91-42b1-9ca4-5224cd838aec); Time taken: 0.481 seconds 2019-04-04 22:33:18,103 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Handler-Pool: Thread-117]: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver> 2019-04-04 22:33:18,103 INFO org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Handler-Pool: Thread-117]: </PERFLOG method=releaseLocks start=1554409998103 end=1554409998103 duration=0 from=org.apache.hado op.hive.ql.Driver> 2019-04-04 22:33:18,105 INFO org.apache.hive.service.cli.operation.OperationManager: [HiveServer2-Handler-Pool: Thread-117]: Closing operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=3 8b7d0d3-d925-48b1-bdb7-b7c3334cc7d8] 2019-04-04 22:33:18,109 WARN org.apache.hive.service.cli.thrift.ThriftCLIService: [HiveServer2-Handler-Pool: Thread-117]: Error executing statement: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Generate Map Join Task Error: Unable to find class: XXXXX Serialization trace: genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator) reducer (org.apache.hadoop.hive.ql.plan.ReduceWork) reduceWork (org.apache.hadoop.hive.ql.plan.MapredWork) at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:187) at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:271) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:337) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:439) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:416) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:282) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:501) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313) at org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:763) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hive.ql.parse.SemanticException: Generate Map Join Task Error: Unable to find class: XXXXX Serialization trace: genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc) colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.JoinOperator) reducer (org.apache.hadoop.hive.ql.plan.ReduceWork) reduceWork (org.apache.hadoop.hive.ql.plan.MapredWork) at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:516) at org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:179) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180) at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125) at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79) at org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107) at org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:273) at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:225) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10315) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10108) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:558) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1356) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1343) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:185) ... 15 more XXXX denotes here the class name. Have you seen such erratic behaviour before? The eror occurs for functions in one package only, although there are 4 packages in reloadable directory. The only difference between them is that the malfunctioning is significantly bigger than the other (it's shaded fat jar). There are some network resources suggesting the size may be a problem, but it isn't  the same problem as ours: https://stackoverflow.com/questions/54572121/how-do-i-fix-this-kryo-exception-when-using-a-udf-on-hive https://stackoverflow.com/questions/32448575/why-does-kryo-throw-classnotfoundexception-for-class-in-fat-jar-cdh-yarn   Is there anything we could do to investigate this case further and get more information about the problem? ... View more

Hi, I'd like to ask about CDH support for long running applications on YARN. We are trying to setup gobblin to work with CDH 5.11.2, but regularly after 2 days we get message that AMRMToken is invalid: CEST INFO  [AMRM Heartbeater thread] org.apache.hadoop.io.retry.RetryInvocationHandler  - Exception while invoking allocate of class ApplicationMasterProtocolPBClientImpl over rm365. Trying to fail over immediately. org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattempt_1535039367371_143153_000001     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)     at java.lang.reflect.Constructor.newInstance(Constructor.java:408)     at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)     at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)     at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)     at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)     at java.lang.reflect.Method.invoke(Method.java:483)     at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)     at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)     at com.sun.proxy.$Proxy20.allocate(Unknown Source)     at org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:277)     at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224) Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Invalid AMRMToken from appattempt_1535039367371_143153_000001     at org.apache.hadoop.ipc.Client.call(Client.java:1502)     at org.apache.hadoop.ipc.Client.call(Client.java:1439)     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)     at com.sun.proxy.$Proxy19.allocate(Unknown Source)     at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) Do you know why would AM fail to refresh the token before its 48h expiration period passes? I found this ticket that, I think, would provide support for AM token refreshment: https://issues.apache.org/jira/browse/YARN-941 It hasn't been solved yet, so I wonder if this functionality has been implemented in another ticket, or is not supported by Yarn yet? I've found also a bug that perhaps relates to the same issue: https://issues.apache.org/jira/browse/YARN-3429 This ticket is solved, but not included in any CDH5 version yet. Do you think we are hitting this issue? If so - when would it be incorporated into CDH? ... View more