Log Type: syslog_dag_1479255529349_182491_1_post Log Upload Time: Thu Dec 15 00:32:28 -0500 2016 Log Length: 14796 2016-12-15 00:32:19,777 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Central Dispatcher queue size after DAG completion, before cleanup: 0 2016-12-15 00:32:19,777 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Waiting for next DAG to be submitted. 2016-12-15 00:32:19,778 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Cleaning up DAG: name=select count(1) from (select * from t_so...a(Stage-1), with id=dag_1479255529349_182491_1 2016-12-15 00:32:19,778 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Completed cleanup for DAG: name=select count(1) from (select * from t_so...a(Stage-1), with id=dag_1479255529349_182491_1 2016-12-15 00:32:21,552 [INFO] [Socket Reader #1 for port 46306] |ipc.Server|: Auth successful for clbhhdev@PRODDFS.PF.DISCOVERFINANCIAL.COM (auth:SIMPLE) 2016-12-15 00:32:21,565 [INFO] [Socket Reader #1 for port 46306] |authorize.ServiceAuthorizationManager|: Authorization successful for clbhhdev@PRODDFS.PF.DISCOVERFINANCIAL.COM (auth:TOKEN) for protocol=interface org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB 2016-12-15 00:32:21,568 [INFO] [IPC Server handler 0 on 46306] |client.DAGClientHandler|: Received message to shutdown AM 2016-12-15 00:32:21,568 [INFO] [IPC Server handler 0 on 46306] |rm.TaskSchedulerEventHandler|: TaskScheduler notified that it should unregister from RM 2016-12-15 00:32:21,568 [INFO] [IPC Server handler 0 on 46306] |app.DAGAppMaster|: No current running DAG, shutting down the AM 2016-12-15 00:32:21,568 [INFO] [IPC Server handler 0 on 46306] |app.DAGAppMaster|: DAGAppMasterShutdownHandler invoked 2016-12-15 00:32:21,568 [INFO] [IPC Server handler 0 on 46306] |app.DAGAppMaster|: Handling DAGAppMaster shutdown 2016-12-15 00:32:21,569 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Sleeping for 5 seconds before shutting down 2016-12-15 00:32:24,671 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: Free: pendingRequests: 0 delayedContainers: 2 heartbeats: 201 lastPreemptionHeartbeat: 200 2016-12-15 00:32:26,569 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Calling stop for all the services 2016-12-15 00:32:26,570 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Initiating stop of YarnTaskScheduler 2016-12-15 00:32:26,570 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Releasing held containers 2016-12-15 00:32:26,570 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Removing all pending taskRequests 2016-12-15 00:32:26,571 [INFO] [AMShutdownThread] |history.HistoryEventHandler|: Stopping HistoryEventHandler 2016-12-15 00:32:26,571 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Stopping RecoveryService 2016-12-15 00:32:26,571 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Handle the remaining events in queue, queue size=0 2016-12-15 00:32:26,571 [INFO] [RecoveryEventHandlingThread] |recovery.RecoveryService|: EventQueue take interrupted. Returning 2016-12-15 00:32:26,571 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Closing Summary Stream 2016-12-15 00:32:26,572 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1479255529349_182491_1][Event:CONTAINER_STOPPED]: containerId=container_e26_1479255529349_182491_01_000003, stoppedTime=1481779946572, exitStatus=0 2016-12-15 00:32:26,572 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1479255529349_182491_1][Event:CONTAINER_STOPPED]: containerId=container_e26_1479255529349_182491_01_000002, stoppedTime=1481779946572, exitStatus=0 2016-12-15 00:32:26,573 [INFO] [ContainerLauncher #2] |launcher.ContainerLauncherImpl|: Stopping container_e26_1479255529349_182491_01_000003 2016-12-15 00:32:26,573 [INFO] [ContainerLauncher #3] |launcher.ContainerLauncherImpl|: Stopping container_e26_1479255529349_182491_01_000002 2016-12-15 00:32:26,573 [INFO] [ContainerLauncher #2] |impl.ContainerManagementProtocolProxy|: Opening proxy : rwld585.rw.discoverfinancial.com:45454 2016-12-15 00:32:26,574 [INFO] [ContainerLauncher #3] |impl.ContainerManagementProtocolProxy|: Opening proxy : rwld586.rw.discoverfinancial.com:45454 2016-12-15 00:32:26,579 [WARN] [AMShutdownThread] |recovery.RecoveryService|: Ignoring error while closing summary stream. The recovery directory at hdfs://dshdp-dev-cluster/tmp/hive/clbhhdev/_tez_session_dir/33aacc40-07c3-4a5a-9a1a-2dbaf1f77bc2/.tez/application_1479255529349_182491/recovery/1 has already been deleted externally 2016-12-15 00:32:26,579 [INFO] [AMShutdownThread] |ats.ATSHistoryLoggingService|: Stopping ATSService, eventQueueBacklog=0 2016-12-15 00:32:26,583 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: AllocatedContainerManager Thread interrupted 2016-12-15 00:32:26,585 [WARN] [ContainerLauncher #2] |common.AsyncDispatcher|: AsyncDispatcher thread interrupted java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:338) at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:230) at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:388) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-12-15 00:32:26,585 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0 , trackingURL=https://rwld407.rw.discoverfinancial.com:8443/#/main/views/TEZ/0.7.0.2.3.0.0-236/TEZ_CLUSTER_INSTANCE?viewPath=%2F%23%2Ftez-app%2Fapplication_1479255529349_182491 2016-12-15 00:32:26,586 [WARN] [ContainerLauncher #3] |common.AsyncDispatcher|: AsyncDispatcher thread interrupted java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:338) at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:238) at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:388) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2016-12-15 00:32:26,586 [WARN] [ContainerLauncher #2] |launcher.ContainerLauncherImpl|: cleanup failed for container container_e26_1479255529349_182491_01_000003 : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:343) at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:230) at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:388) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:338) ... 5 more 2016-12-15 00:32:26,587 [ERROR] [ContainerLauncher #3] |yarn.YarnUncaughtExceptionHandler|: Thread Thread[ContainerLauncher #3,5,main] threw an Exception. org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:343) at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:238) at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:388) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219) at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340) at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338) at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:338) ... 5 more 2016-12-15 00:32:26,590 [INFO] [AMShutdownThread] |impl.AMRMClientImpl|: Waiting for application to be successfully unregistered. 2016-12-15 00:32:26,691 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Successfully unregistered application from RM 2016-12-15 00:32:26,691 [INFO] [AMRM Callback Handler Thread] |impl.AMRMClientAsyncImpl|: Interrupted while waiting for queue java.lang.InterruptedException at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:287) 2016-12-15 00:32:26,696 [INFO] [AMShutdownThread] |mortbay.log|: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0 2016-12-15 00:32:26,797 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 33408 2016-12-15 00:32:26,798 [INFO] [IPC Server listener on 33408] |ipc.Server|: Stopping IPC Server listener on 33408 2016-12-15 00:32:26,798 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 46306 2016-12-15 00:32:26,798 [INFO] [IPC Server listener on 46306] |ipc.Server|: Stopping IPC Server listener on 46306 2016-12-15 00:32:26,798 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder 2016-12-15 00:32:26,799 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder 2016-12-15 00:32:26,801 [INFO] [Thread-4] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked 2016-12-15 00:32:26,801 [INFO] [Thread-4] |app.DAGAppMaster|: The shutdown handler is still running, waiting for it to complete 2016-12-15 00:32:26,806 [WARN] [AMShutdownThread] |ipc.Client|: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563) at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:727) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492) at org.apache.hadoop.ipc.Client.call(Client.java:1402) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy18.delete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:542) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104) at com.sun.proxy.$Proxy19.delete(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2098) at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:765) at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:761) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:761) at org.apache.tez.dag.app.DAGAppMaster$4.run(DAGAppMaster.java:1898) at org.apache.tez.dag.app.DAGAppMaster$4.run(DAGAppMaster.java:1894) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.tez.dag.app.DAGAppMaster.serviceStop(DAGAppMaster.java:1894) at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221) at org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterShutdownHandler$AMShutdownRunnable.run(DAGAppMaster.java:845) at java.lang.Thread.run(Thread.java:745) 2016-12-15 00:32:26,810 [WARN] [AMShutdownThread] |app.DAGAppMaster|: Failed to delete tez scratch data dir, path=hdfs://dshdp-dev-cluster/tmp/hive/clbhhdev/_tez_session_dir/33aacc40-07c3-4a5a-9a1a-2dbaf1f77bc2/.tez/application_1479255529349_182491 2016-12-15 00:32:26,810 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Exiting DAGAppMaster..GoodBye! 2016-12-15 00:32:26,811 [INFO] [Thread-4] |app.DAGAppMaster|: The shutdown handler has completed