Log Type: syslog_dag_1479255529349_182491_1_post
Log Upload Time: Thu Dec 15 00:32:28 -0500 2016
Log Length: 14796
2016-12-15 00:32:19,777 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Central Dispatcher queue size after DAG completion, before cleanup: 0
2016-12-15 00:32:19,777 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Waiting for next DAG to be submitted.
2016-12-15 00:32:19,778 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Cleaning up DAG: name=select count(1) from (select * from t_so...a(Stage-1), with id=dag_1479255529349_182491_1
2016-12-15 00:32:19,778 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Completed cleanup for DAG: name=select count(1) from (select * from t_so...a(Stage-1), with id=dag_1479255529349_182491_1
2016-12-15 00:32:21,552 [INFO] [Socket Reader #1 for port 46306] |ipc.Server|: Auth successful for clbhhdev@PRODDFS.PF.DISCOVERFINANCIAL.COM (auth:SIMPLE)
2016-12-15 00:32:21,565 [INFO] [Socket Reader #1 for port 46306] |authorize.ServiceAuthorizationManager|: Authorization successful for clbhhdev@PRODDFS.PF.DISCOVERFINANCIAL.COM (auth:TOKEN) for protocol=interface org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPB
2016-12-15 00:32:21,568 [INFO] [IPC Server handler 0 on 46306] |client.DAGClientHandler|: Received message to shutdown AM
2016-12-15 00:32:21,568 [INFO] [IPC Server handler 0 on 46306] |rm.TaskSchedulerEventHandler|: TaskScheduler notified that it should unregister from RM
2016-12-15 00:32:21,568 [INFO] [IPC Server handler 0 on 46306] |app.DAGAppMaster|: No current running DAG, shutting down the AM
2016-12-15 00:32:21,568 [INFO] [IPC Server handler 0 on 46306] |app.DAGAppMaster|: DAGAppMasterShutdownHandler invoked
2016-12-15 00:32:21,568 [INFO] [IPC Server handler 0 on 46306] |app.DAGAppMaster|: Handling DAGAppMaster shutdown
2016-12-15 00:32:21,569 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Sleeping for 5 seconds before shutting down
2016-12-15 00:32:24,671 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:8192, vCores:2> Free: <memory:0, vCores:0> pendingRequests: 0 delayedContainers: 2 heartbeats: 201 lastPreemptionHeartbeat: 200
2016-12-15 00:32:26,569 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Calling stop for all the services
2016-12-15 00:32:26,570 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Initiating stop of YarnTaskScheduler
2016-12-15 00:32:26,570 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Releasing held containers
2016-12-15 00:32:26,570 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Removing all pending taskRequests
2016-12-15 00:32:26,571 [INFO] [AMShutdownThread] |history.HistoryEventHandler|: Stopping HistoryEventHandler
2016-12-15 00:32:26,571 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Stopping RecoveryService
2016-12-15 00:32:26,571 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Handle the remaining events in queue, queue size=0
2016-12-15 00:32:26,571 [INFO] [RecoveryEventHandlingThread] |recovery.RecoveryService|: EventQueue take interrupted. Returning
2016-12-15 00:32:26,571 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Closing Summary Stream
2016-12-15 00:32:26,572 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1479255529349_182491_1][Event:CONTAINER_STOPPED]: containerId=container_e26_1479255529349_182491_01_000003, stoppedTime=1481779946572, exitStatus=0
2016-12-15 00:32:26,572 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1479255529349_182491_1][Event:CONTAINER_STOPPED]: containerId=container_e26_1479255529349_182491_01_000002, stoppedTime=1481779946572, exitStatus=0
2016-12-15 00:32:26,573 [INFO] [ContainerLauncher #2] |launcher.ContainerLauncherImpl|: Stopping container_e26_1479255529349_182491_01_000003
2016-12-15 00:32:26,573 [INFO] [ContainerLauncher #3] |launcher.ContainerLauncherImpl|: Stopping container_e26_1479255529349_182491_01_000002
2016-12-15 00:32:26,573 [INFO] [ContainerLauncher #2] |impl.ContainerManagementProtocolProxy|: Opening proxy : rwld585.rw.discoverfinancial.com:45454
2016-12-15 00:32:26,574 [INFO] [ContainerLauncher #3] |impl.ContainerManagementProtocolProxy|: Opening proxy : rwld586.rw.discoverfinancial.com:45454
2016-12-15 00:32:26,579 [WARN] [AMShutdownThread] |recovery.RecoveryService|: Ignoring error while closing summary stream. The recovery directory at hdfs://dshdp-dev-cluster/tmp/hive/clbhhdev/_tez_session_dir/33aacc40-07c3-4a5a-9a1a-2dbaf1f77bc2/.tez/application_1479255529349_182491/recovery/1 has already been deleted externally
2016-12-15 00:32:26,579 [INFO] [AMShutdownThread] |ats.ATSHistoryLoggingService|: Stopping ATSService, eventQueueBacklog=0
2016-12-15 00:32:26,583 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: AllocatedContainerManager Thread interrupted
2016-12-15 00:32:26,585 [WARN] [ContainerLauncher #2] |common.AsyncDispatcher|: AsyncDispatcher thread interrupted
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
	at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:338)
	at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:230)
	at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:388)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
2016-12-15 00:32:26,585 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=1, successfulDAGs=1, failedDAGs=0, killedDAGs=0
, trackingURL=https://rwld407.rw.discoverfinancial.com:8443/#/main/views/TEZ/0.7.0.2.3.0.0-236/TEZ_CLUSTER_INSTANCE?viewPath=%2F%23%2Ftez-app%2Fapplication_1479255529349_182491
2016-12-15 00:32:26,586 [WARN] [ContainerLauncher #3] |common.AsyncDispatcher|: AsyncDispatcher thread interrupted
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
	at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:338)
	at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:238)
	at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:388)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
2016-12-15 00:32:26,586 [WARN] [ContainerLauncher #2] |launcher.ContainerLauncherImpl|: cleanup failed for container container_e26_1479255529349_182491_01_000003 : org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException
	at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:343)
	at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:230)
	at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:388)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
	at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:338)
	... 5 more

2016-12-15 00:32:26,587 [ERROR] [ContainerLauncher #3] |yarn.YarnUncaughtExceptionHandler|: Thread Thread[ContainerLauncher #3,5,main] threw an Exception.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.InterruptedException
	at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:343)
	at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$Container.kill(ContainerLauncherImpl.java:238)
	at org.apache.tez.dag.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:388)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1219)
	at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:340)
	at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:338)
	at org.apache.tez.common.AsyncDispatcher$GenericEventHandler.handle(AsyncDispatcher.java:338)
	... 5 more
2016-12-15 00:32:26,590 [INFO] [AMShutdownThread] |impl.AMRMClientImpl|: Waiting for application to be successfully unregistered.
2016-12-15 00:32:26,691 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Successfully unregistered application from RM
2016-12-15 00:32:26,691 [INFO] [AMRM Callback Handler Thread] |impl.AMRMClientAsyncImpl|: Interrupted while waiting for queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2052)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:287)
2016-12-15 00:32:26,696 [INFO] [AMShutdownThread] |mortbay.log|: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0
2016-12-15 00:32:26,797 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 33408
2016-12-15 00:32:26,798 [INFO] [IPC Server listener on 33408] |ipc.Server|: Stopping IPC Server listener on 33408
2016-12-15 00:32:26,798 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 46306
2016-12-15 00:32:26,798 [INFO] [IPC Server listener on 46306] |ipc.Server|: Stopping IPC Server listener on 46306
2016-12-15 00:32:26,798 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder
2016-12-15 00:32:26,799 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder
2016-12-15 00:32:26,801 [INFO] [Thread-4] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked
2016-12-15 00:32:26,801 [INFO] [Thread-4] |app.DAGAppMaster|: The shutdown handler is still running, waiting for it to complete
2016-12-15 00:32:26,806 [WARN] [AMShutdownThread] |ipc.Client|: Exception encountered while connecting to the server : 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563)
	at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:727)
	at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492)
	at org.apache.hadoop.ipc.Client.call(Client.java:1402)
	at org.apache.hadoop.ipc.Client.call(Client.java:1363)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy18.delete(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:542)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
	at com.sun.proxy.$Proxy19.delete(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2098)
	at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:765)
	at org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:761)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:761)
	at org.apache.tez.dag.app.DAGAppMaster$4.run(DAGAppMaster.java:1898)
	at org.apache.tez.dag.app.DAGAppMaster$4.run(DAGAppMaster.java:1894)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.tez.dag.app.DAGAppMaster.serviceStop(DAGAppMaster.java:1894)
	at org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
	at org.apache.tez.dag.app.DAGAppMaster$DAGAppMasterShutdownHandler$AMShutdownRunnable.run(DAGAppMaster.java:845)
	at java.lang.Thread.run(Thread.java:745)
2016-12-15 00:32:26,810 [WARN] [AMShutdownThread] |app.DAGAppMaster|: Failed to delete tez scratch data dir, path=hdfs://dshdp-dev-cluster/tmp/hive/clbhhdev/_tez_session_dir/33aacc40-07c3-4a5a-9a1a-2dbaf1f77bc2/.tez/application_1479255529349_182491
2016-12-15 00:32:26,810 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Exiting DAGAppMaster..GoodBye!
2016-12-15 00:32:26,811 [INFO] [Thread-4] |app.DAGAppMaster|: The shutdown handler has completed