Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

I am exporting data from teradata into Hadoop. but my query is getting failed by giving an error " Failed to write data"

Highlighted

I am exporting data from teradata into Hadoop. but my query is getting failed by giving an error " Failed to write data"

New Contributor

Tools

Log Type: syslog

Log Upload Time: Tue Mar 08 22:59:27 -0800 2016

Log Length: 4931

.When I'm looking into the error logs, it is telling me mapreduce tasks are failing. Please look at the below logs and help me in resolving the issue.

2016-03-08 22:47:07,414 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
2016-03-08 22:47:07,499 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2016-03-08 22:47:07,499 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2016-03-08 22:47:07,509 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2016-03-08 22:47:07,510 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1457504560070_0004, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@175b9425)
2016-03-08 22:47:07,556 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: RM_DELEGATION_TOKEN, Service: 39.7.48.2:8032,39.7.48.3:8032, Ident: (owner=hive, renewer=oozie mr token, realUser=oozie, issueDate=1457506410968, maxDate=1458111210968, sequenceNumber=908, masterKeyId=280)
2016-03-08 22:47:07,599 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2016-03-08 22:47:07,848 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /data1/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004,/data2/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004,/data3/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004,/data4/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004,/data5/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004,/data6/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004,/data7/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004,/data8/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004,/data9/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004,/data10/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004,/data12/hadoop/yarn/local/usercache/hive/appcache/application_1457504560070_0004
2016-03-08 22:47:08,132 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2016-03-08 22:47:08,623 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2016-03-08 22:47:08,840 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: com.teradata.dynaload.hcatalog.mapper.TDInputFormat$TeradataInputSplit@2ece4966
2016-03-08 22:47:08,844 INFO [main] com.teradata.dynaload.hcatalog.mapper.TDInputFormat$TeradataRecordReader: recordreader class com.teradata.dynaload.hcatalog.mapper.TDInputFormat$TeradataRecordReaderinitialize time is:  1457506028844
2016-03-08 22:47:09,512 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 300417020(1201668080)
2016-03-08 22:47:09,512 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 1146
2016-03-08 22:47:09,512 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 841167680
2016-03-08 22:47:09,512 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 1201668096
2016-03-08 22:47:09,512 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 300417020; length = 75104256
2016-03-08 22:47:09,515 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-03-08 22:47:09,518 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
2016-03-08 22:47:09,518 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
2016-03-08 22:47:09,848 WARN [main] org.apache.hadoop.hive.conf.HiveConf: HiveConf of name hive.metastore.local does not exist
2016-03-08 22:47:09,914 INFO [main] hive.metastore: Trying to connect to metastore with URI thrift://apus2.labs.teradata.com:9083
2016-03-08 22:47:09,951 INFO [main] hive.metastore: Connected to metastore.
2016-03-08 22:47:10,407 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
2016-03-08 22:47:10,452 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 1
2016-03-08 22:47:10,453 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.work.output.dir is deprecated. Instead, use mapreduce.task.output.dir
2016-03-08 22:47:10,453 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
2016-03-08 22:47:10,457 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class  

 Application Logs:

Log Type: syslog
Log Upload Time: Tue Mar 08 22:59:27 -0800 2016
Log Length: 66959
2016-03-08 22:53:46,354 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1457504560070_0004_000001
2016-03-08 22:53:46,602 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-03-08 22:53:46,639 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2016-03-08 22:53:46,639 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: mapreduce.job, Service: 39.7.48.4:35971, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@37d4349f)
2016-03-08 22:53:46,670 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 4 cluster_timestamp: 1457504560070 } attemptId: 1 } keyId: 462508970)
2016-03-08 22:53:46,741 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 39.7.48.2:8032,39.7.48.3:8032, Ident: (owner=hive, renewer=oozie mr token, realUser=oozie, issueDate=1457506410968, maxDate=1458111210968, sequenceNumber=908, masterKeyId=280)
2016-03-08 22:53:46,756 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2016-03-08 22:53:47,371 WARN [main] org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2016-03-08 22:53:47,536 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
2016-03-08 22:53:47,588 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.NullOutputFormat$2
2016-03-08 22:53:47,608 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2016-03-08 22:53:47,609 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2016-03-08 22:53:47,609 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2016-03-08 22:53:47,610 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2016-03-08 22:53:47,610 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2016-03-08 22:53:47,611 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2016-03-08 22:53:47,611 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2016-03-08 22:53:47,612 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2016-03-08 22:53:47,640 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore -  ignoring
2016-03-08 22:53:47,662 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore -  ignoring
2016-03-08 22:53:47,685 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore -  ignoring
2016-03-08 22:53:47,710 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
2016-03-08 22:53:47,773 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2016-03-08 22:53:48,129 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties
2016-03-08 22:53:48,226 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2016-03-08 22:53:48,226 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2016-03-08 22:53:48,233 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1457504560070_0004 to jobTokenSecretManager
2016-03-08 22:53:48,361 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1457504560070_0004 because: not enabled; too much RAM;
2016-03-08 22:53:48,374 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1457504560070_0004 = 7. Number of splits = 7
2016-03-08 22:53:48,374 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1457504560070_0004 = 1
2016-03-08 22:53:48,375 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1457504560070_0004Job Transitioned from NEW to INITED
2016-03-08 22:53:48,376 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1457504560070_0004.
2016-03-08 22:53:48,397 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2016-03-08 22:53:48,407 INFO [Socket Reader #1 for port 52062] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 52062
2016-03-08 22:53:48,426 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2016-03-08 22:53:48,427 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2016-03-08 22:53:48,427 INFO [IPC Server listener on 52062] org.apache.hadoop.ipc.Server: IPC Server listener on 52062: starting
2016-03-08 22:53:48,428 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at apus7.labs.teradata.com/39.7.48.8:52062
2016-03-08 22:53:48,505 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-03-08 22:53:48,513 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2016-03-08 22:53:48,518 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
2016-03-08 22:53:48,524 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2016-03-08 22:53:48,564 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
2016-03-08 22:53:48,564 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
2016-03-08 22:53:48,567 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
2016-03-08 22:53:48,567 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2016-03-08 22:53:48,575 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 49303
2016-03-08 22:53:48,575 INFO [main] org.mortbay.log: jetty-6.1.26.hwx
2016-03-08 22:53:48,604 INFO [main] org.mortbay.log: Extract jar:file:/data11/hadoop/yarn/local/filecache/11/mapreduce.tar.gz/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.7.1.2.3.0.0-2557.jar!/webapps/mapreduce to ./tmp/Jetty_0_0_0_0_49303_mapreduce____.emb3w8/webapp
2016-03-08 22:53:48,842 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:49303
2016-03-08 22:53:48,842 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 49303
2016-03-08 22:53:49,098 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2016-03-08 22:53:49,103 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2016-03-08 22:53:49,104 INFO [Socket Reader #1 for port 46067] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 46067
2016-03-08 22:53:49,107 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2016-03-08 22:53:49,107 INFO [IPC Server listener on 46067] org.apache.hadoop.ipc.Server: IPC Server listener on 46067: starting
2016-03-08 22:53:49,127 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2016-03-08 22:53:49,127 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2016-03-08 22:53:49,127 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2016-03-08 22:53:49,236 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: maxContainerCapability: <memory:6144, vCores:8>
2016-03-08 22:53:49,236 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: queue: default
2016-03-08 22:53:49,239 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
2016-03-08 22:53:49,240 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
2016-03-08 22:53:49,242 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
2016-03-08 22:53:49,247 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1457504560070_0004Job Transitioned from INITED to SETUP
2016-03-08 22:53:49,248 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
2016-03-08 22:53:49,249 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1457504560070_0004Job Transitioned from SETUP to RUNNING
2016-03-08 22:53:49,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus6.labs.teradata.com to /default-rack
2016-03-08 22:53:49,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000000 Task Transitioned from NEW to SCHEDULED
2016-03-08 22:53:49,298 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus4.labs.teradata.com to /default-rack
2016-03-08 22:53:49,299 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000001 Task Transitioned from NEW to SCHEDULED
2016-03-08 22:53:49,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus8.labs.teradata.com to /default-rack
2016-03-08 22:53:49,316 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000002 Task Transitioned from NEW to SCHEDULED
2016-03-08 22:53:49,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus9.labs.teradata.com to /default-rack
2016-03-08 22:53:49,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000003 Task Transitioned from NEW to SCHEDULED
2016-03-08 22:53:49,364 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus3.labs.teradata.com to /default-rack
2016-03-08 22:53:49,364 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000004 Task Transitioned from NEW to SCHEDULED
2016-03-08 22:53:49,382 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus5.labs.teradata.com to /default-rack
2016-03-08 22:53:49,382 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000005 Task Transitioned from NEW to SCHEDULED
2016-03-08 22:53:49,399 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus7.labs.teradata.com to /default-rack
2016-03-08 22:53:49,399 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000006 Task Transitioned from NEW to SCHEDULED
2016-03-08 22:53:49,400 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_r_000000 Task Transitioned from NEW to SCHEDULED
2016-03-08 22:53:49,401 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2016-03-08 22:53:49,402 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2016-03-08 22:53:49,402 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2016-03-08 22:53:49,402 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2016-03-08 22:53:49,402 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000004_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2016-03-08 22:53:49,402 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000005_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2016-03-08 22:53:49,402 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000006_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2016-03-08 22:53:49,403 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2016-03-08 22:53:49,404 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:2048, vCores:1>
2016-03-08 22:53:49,411 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:4096, vCores:1>
2016-03-08 22:53:49,412 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1457504560070_0004, File: hdfs://C423A:8020/user/hive/.staging/job_1457504560070_0004/job_1457504560070_0004_1.jhist
2016-03-08 22:53:50,319 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:7 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2016-03-08 22:53:50,379 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1457504560070_0004: ask=9 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:153600, vCores:1> knownNMs=7
2016-03-08 22:53:50,380 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:153600, vCores:1>
2016-03-08 22:53:50,380 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1
2016-03-08 22:53:51,401 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 7
2016-03-08 22:53:51,403 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e50_1457504560070_0004_01_000002 to attempt_1457504560070_0004_m_000003_0
2016-03-08 22:53:51,405 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e50_1457504560070_0004_01_000003 to attempt_1457504560070_0004_m_000004_0
2016-03-08 22:53:51,406 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e50_1457504560070_0004_01_000004 to attempt_1457504560070_0004_m_000001_0
2016-03-08 22:53:51,406 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e50_1457504560070_0004_01_000005 to attempt_1457504560070_0004_m_000002_0
2016-03-08 22:53:51,406 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e50_1457504560070_0004_01_000006 to attempt_1457504560070_0004_m_000000_0
2016-03-08 22:53:51,407 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e50_1457504560070_0004_01_000008 to attempt_1457504560070_0004_m_000005_0
2016-03-08 22:53:51,407 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved apus6.labs.teradata.com to /default-rack
2016-03-08 22:53:51,408 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e50_1457504560070_0004_01_000007 to attempt_1457504560070_0004_m_000006_0
2016-03-08 22:53:51,408 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:110592, vCores:1>
2016-03-08 22:53:51,408 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 1
2016-03-08 22:53:51,408 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:7 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:7 ContRel:0 HostLocal:6 RackLocal:1
2016-03-08 22:53:51,476 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus9.labs.teradata.com to /default-rack
2016-03-08 22:53:51,503 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://C423A/user/hive/.staging/job_1457504560070_0004/job.jar
2016-03-08 22:53:51,506 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hive/.staging/job_1457504560070_0004/job.xml
2016-03-08 22:53:51,543 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/hive-hcat-core-1.2.1.2.3.0.0-2557.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/hive-hcat-core-1.2.1.2.3.0.0-2557.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,545 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/terajdbc4.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/terajdbc4.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,546 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/tdgssconfig.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/tdgssconfig.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,548 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/joda-time-1.6.2.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/joda-time-1.6.2.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,549 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/hive-exec-1.2.1.2.3.0.0-2557.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/hive-exec-1.2.1.2.3.0.0-2557.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,550 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/hive-metastore-1.2.1.2.3.0.0-2557.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/hive-metastore-1.2.1.2.3.0.0-2557.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,552 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/libfb303-0.9.0.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/libfb303-0.9.0.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,553 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/hive-serde-1.2.1.2.3.0.0-2557.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/hive-serde-1.2.1.2.3.0.0-2557.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,555 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/tdptl.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/tdptl.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,556 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/hive-site.xml conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/hive-site.xml This will be an error in Hadoop 2.0
2016-03-08 22:53:51,558 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/hive-common-1.2.1.2.3.0.0-2557.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/hive-common-1.2.1.2.3.0.0-2557.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,559 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/tdrowconverter.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/tdrowconverter.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,561 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.util.MRApps: cache file (mapreduce.job.cache.files) hdfs://C423A/apps/querygrid/lib/tdefssp.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://C423A/user/hive/.staging/job_1457504560070_0004/libjars/tdefssp.jar This will be an error in Hadoop 2.0
2016-03-08 22:53:51,561 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #2 tokens and #1 secret keys for NM use for launching container
2016-03-08 22:53:51,561 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 2
2016-03-08 22:53:51,562 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
2016-03-08 22:53:51,677 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2016-03-08 22:53:51,682 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus3.labs.teradata.com to /default-rack
2016-03-08 22:53:51,683 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000004_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2016-03-08 22:53:51,684 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus4.labs.teradata.com to /default-rack
2016-03-08 22:53:51,684 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2016-03-08 22:53:51,685 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus8.labs.teradata.com to /default-rack
2016-03-08 22:53:51,686 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2016-03-08 22:53:51,686 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus6.labs.teradata.com to /default-rack
2016-03-08 22:53:51,687 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2016-03-08 22:53:51,687 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus5.labs.teradata.com to /default-rack
2016-03-08 22:53:51,688 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000005_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2016-03-08 22:53:51,689 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved apus6.labs.teradata.com to /default-rack
2016-03-08 22:53:51,689 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000006_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
2016-03-08 22:53:51,703 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e50_1457504560070_0004_01_000002 taskAttempt attempt_1457504560070_0004_m_000003_0
2016-03-08 22:53:51,703 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e50_1457504560070_0004_01_000003 taskAttempt attempt_1457504560070_0004_m_000004_0
2016-03-08 22:53:51,705 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e50_1457504560070_0004_01_000004 taskAttempt attempt_1457504560070_0004_m_000001_0
2016-03-08 22:53:51,706 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e50_1457504560070_0004_01_000005 taskAttempt attempt_1457504560070_0004_m_000002_0
2016-03-08 22:53:51,706 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e50_1457504560070_0004_01_000006 taskAttempt attempt_1457504560070_0004_m_000000_0
2016-03-08 22:53:51,707 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e50_1457504560070_0004_01_000008 taskAttempt attempt_1457504560070_0004_m_000005_0
2016-03-08 22:53:51,709 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1457504560070_0004_m_000001_0
2016-03-08 22:53:51,709 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1457504560070_0004_m_000004_0
2016-03-08 22:53:51,709 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1457504560070_0004_m_000000_0
2016-03-08 22:53:51,709 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1457504560070_0004_m_000005_0
2016-03-08 22:53:51,709 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1457504560070_0004_m_000003_0
2016-03-08 22:53:51,709 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1457504560070_0004_m_000002_0
2016-03-08 22:53:51,710 INFO [ContainerLauncher #2] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus4.labs.teradata.com:45454
2016-03-08 22:53:51,710 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e50_1457504560070_0004_01_000007 taskAttempt attempt_1457504560070_0004_m_000006_0
2016-03-08 22:53:51,711 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1457504560070_0004_m_000006_0
2016-03-08 22:53:51,745 INFO [ContainerLauncher #6] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus6.labs.teradata.com:45454
2016-03-08 22:53:51,747 INFO [ContainerLauncher #3] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus8.labs.teradata.com:45454
2016-03-08 22:53:51,749 INFO [ContainerLauncher #0] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus9.labs.teradata.com:45454
2016-03-08 22:53:51,753 INFO [ContainerLauncher #5] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus5.labs.teradata.com:45454
2016-03-08 22:53:51,756 INFO [ContainerLauncher #4] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus6.labs.teradata.com:45454
2016-03-08 22:53:51,759 INFO [ContainerLauncher #1] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus3.labs.teradata.com:45454
2016-03-08 22:53:51,847 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1457504560070_0004_m_000001_0 : 13562
2016-03-08 22:53:51,847 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1457504560070_0004_m_000005_0 : 13562
2016-03-08 22:53:51,847 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1457504560070_0004_m_000002_0 : 13562
2016-03-08 22:53:51,847 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1457504560070_0004_m_000006_0 : 13562
2016-03-08 22:53:51,847 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1457504560070_0004_m_000004_0 : 13562
2016-03-08 22:53:51,847 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1457504560070_0004_m_000000_0 : 13562
2016-03-08 22:53:51,847 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1457504560070_0004_m_000003_0 : 13562
2016-03-08 22:53:51,850 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1457504560070_0004_m_000004_0] using containerId: [container_e50_1457504560070_0004_01_000003 on NM: [apus3.labs.teradata.com:45454]
2016-03-08 22:53:51,855 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000004_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2016-03-08 22:53:51,855 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1457504560070_0004_m_000003_0] using containerId: [container_e50_1457504560070_0004_01_000002 on NM: [apus9.labs.teradata.com:45454]
2016-03-08 22:53:51,855 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000003_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2016-03-08 22:53:51,856 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1457504560070_0004_m_000006_0] using containerId: [container_e50_1457504560070_0004_01_000007 on NM: [apus6.labs.teradata.com:45454]
2016-03-08 22:53:51,856 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000006_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2016-03-08 22:53:51,856 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1457504560070_0004_m_000005_0] using containerId: [container_e50_1457504560070_0004_01_000008 on NM: [apus5.labs.teradata.com:45454]
2016-03-08 22:53:51,857 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000005_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2016-03-08 22:53:51,857 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1457504560070_0004_m_000002_0] using containerId: [container_e50_1457504560070_0004_01_000005 on NM: [apus8.labs.teradata.com:45454]
2016-03-08 22:53:51,857 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2016-03-08 22:53:51,858 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1457504560070_0004_m_000000_0] using containerId: [container_e50_1457504560070_0004_01_000006 on NM: [apus6.labs.teradata.com:45454]
2016-03-08 22:53:51,858 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2016-03-08 22:53:51,859 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1457504560070_0004_m_000001_0] using containerId: [container_e50_1457504560070_0004_01_000004 on NM: [apus4.labs.teradata.com:45454]
2016-03-08 22:53:51,859 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
2016-03-08 22:53:51,860 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000004 Task Transitioned from SCHEDULED to RUNNING
2016-03-08 22:53:51,860 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000003 Task Transitioned from SCHEDULED to RUNNING
2016-03-08 22:53:51,860 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000006 Task Transitioned from SCHEDULED to RUNNING
2016-03-08 22:53:51,861 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000005 Task Transitioned from SCHEDULED to RUNNING
2016-03-08 22:53:51,861 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000002 Task Transitioned from SCHEDULED to RUNNING
2016-03-08 22:53:51,861 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000000 Task Transitioned from SCHEDULED to RUNNING
2016-03-08 22:53:51,862 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000001 Task Transitioned from SCHEDULED to RUNNING
2016-03-08 22:53:52,410 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1457504560070_0004: ask=9 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:110592, vCores:1> knownNMs=7
2016-03-08 22:53:53,373 INFO [Socket Reader #1 for port 46067] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1457504560070_0004 (auth:SIMPLE)
2016-03-08 22:53:53,377 INFO [Socket Reader #1 for port 46067] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1457504560070_0004 (auth:SIMPLE)
2016-03-08 22:53:53,379 INFO [Socket Reader #1 for port 46067] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1457504560070_0004 (auth:SIMPLE)
2016-03-08 22:53:53,386 INFO [Socket Reader #1 for port 46067] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1457504560070_0004 (auth:SIMPLE)
2016-03-08 22:53:53,402 INFO [IPC Server handler 2 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1457504560070_0004_m_54975581388805 asked for a task
2016-03-08 22:53:53,403 INFO [IPC Server handler 2 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1457504560070_0004_m_54975581388805 given task: attempt_1457504560070_0004_m_000002_0
2016-03-08 22:53:53,403 INFO [IPC Server handler 1 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1457504560070_0004_m_54975581388802 asked for a task
2016-03-08 22:53:53,404 INFO [IPC Server handler 1 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1457504560070_0004_m_54975581388802 given task: attempt_1457504560070_0004_m_000003_0
2016-03-08 22:53:53,404 INFO [IPC Server handler 4 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1457504560070_0004_m_54975581388804 asked for a task
2016-03-08 22:53:53,404 INFO [IPC Server handler 0 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1457504560070_0004_m_54975581388803 asked for a task
2016-03-08 22:53:53,405 INFO [IPC Server handler 4 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1457504560070_0004_m_54975581388804 given task: attempt_1457504560070_0004_m_000001_0
2016-03-08 22:53:53,405 INFO [IPC Server handler 0 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1457504560070_0004_m_54975581388803 given task: attempt_1457504560070_0004_m_000004_0
2016-03-08 22:53:53,459 INFO [Socket Reader #1 for port 46067] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1457504560070_0004 (auth:SIMPLE)
2016-03-08 22:53:53,475 INFO [IPC Server handler 3 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1457504560070_0004_m_54975581388808 asked for a task
2016-03-08 22:53:53,475 INFO [IPC Server handler 3 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1457504560070_0004_m_54975581388808 given task: attempt_1457504560070_0004_m_000005_0
2016-03-08 22:53:53,499 INFO [Socket Reader #1 for port 46067] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1457504560070_0004 (auth:SIMPLE)
2016-03-08 22:53:53,503 INFO [Socket Reader #1 for port 46067] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1457504560070_0004 (auth:SIMPLE)
2016-03-08 22:53:53,513 INFO [IPC Server handler 5 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1457504560070_0004_m_54975581388806 asked for a task
2016-03-08 22:53:53,513 INFO [IPC Server handler 5 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1457504560070_0004_m_54975581388806 given task: attempt_1457504560070_0004_m_000000_0
2016-03-08 22:53:53,516 INFO [IPC Server handler 6 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1457504560070_0004_m_54975581388807 asked for a task
2016-03-08 22:53:53,517 INFO [IPC Server handler 6 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1457504560070_0004_m_54975581388807 given task: attempt_1457504560070_0004_m_000006_0
2016-03-08 22:54:00,283 INFO [IPC Server handler 5 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1457504560070_0004_m_000002_0 is : 0.0
2016-03-08 22:54:00,284 INFO [IPC Server handler 6 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1457504560070_0004_m_000004_0 is : 0.0
2016-03-08 22:54:00,284 INFO [IPC Server handler 7 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1457504560070_0004_m_000003_0 is : 0.0
2016-03-08 22:54:00,288 INFO [IPC Server handler 9 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1457504560070_0004_m_000005_0 is : 0.0
2016-03-08 22:54:00,295 INFO [IPC Server handler 8 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1457504560070_0004_m_000001_0 is : 0.0
2016-03-08 22:54:00,401 INFO [IPC Server handler 11 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1457504560070_0004_m_000000_0 is : 0.0
2016-03-08 22:54:00,407 INFO [IPC Server handler 10 on 46067] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1457504560070_0004_m_000006_0 is : 0.0
2016-03-08 22:59:19,116 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1457504560070_0004_m_000000_0: AttemptID:attempt_1457504560070_0004_m_000000_0 Timed out after 300 secs
2016-03-08 22:59:19,118 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2016-03-08 22:59:19,118 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1457504560070_0004_m_000004_0: AttemptID:attempt_1457504560070_0004_m_000004_0 Timed out after 300 secs
2016-03-08 22:59:19,118 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000004_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2016-03-08 22:59:19,118 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1457504560070_0004_m_000003_0: AttemptID:attempt_1457504560070_0004_m_000003_0 Timed out after 300 secs
2016-03-08 22:59:19,119 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000003_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2016-03-08 22:59:19,119 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1457504560070_0004_m_000002_0: AttemptID:attempt_1457504560070_0004_m_000002_0 Timed out after 300 secs
2016-03-08 22:59:19,119 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000002_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2016-03-08 22:59:19,119 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1457504560070_0004_m_000001_0: AttemptID:attempt_1457504560070_0004_m_000001_0 Timed out after 300 secs
2016-03-08 22:59:19,120 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000001_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2016-03-08 22:59:19,120 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1457504560070_0004_m_000006_0: AttemptID:attempt_1457504560070_0004_m_000006_0 Timed out after 300 secs
2016-03-08 22:59:19,120 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000006_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2016-03-08 22:59:19,120 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1457504560070_0004_m_000005_0: AttemptID:attempt_1457504560070_0004_m_000005_0 Timed out after 300 secs
2016-03-08 22:59:19,120 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000005_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2016-03-08 22:59:19,122 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e50_1457504560070_0004_01_000005 taskAttempt attempt_1457504560070_0004_m_000002_0
2016-03-08 22:59:19,122 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e50_1457504560070_0004_01_000004 taskAttempt attempt_1457504560070_0004_m_000001_0
2016-03-08 22:59:19,122 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e50_1457504560070_0004_01_000008 taskAttempt attempt_1457504560070_0004_m_000005_0
2016-03-08 22:59:19,123 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1457504560070_0004_m_000001_0
2016-03-08 22:59:19,123 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e50_1457504560070_0004_01_000003 taskAttempt attempt_1457504560070_0004_m_000004_0
2016-03-08 22:59:19,123 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1457504560070_0004_m_000002_0
2016-03-08 22:59:19,122 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e50_1457504560070_0004_01_000006 taskAttempt attempt_1457504560070_0004_m_000000_0
2016-03-08 22:59:19,122 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e50_1457504560070_0004_01_000007 taskAttempt attempt_1457504560070_0004_m_000006_0
2016-03-08 22:59:19,124 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1457504560070_0004_m_000004_0
2016-03-08 22:59:19,123 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1457504560070_0004_m_000005_0
2016-03-08 22:59:19,123 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e50_1457504560070_0004_01_000002 taskAttempt attempt_1457504560070_0004_m_000003_0
2016-03-08 22:59:19,125 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1457504560070_0004_m_000006_0
2016-03-08 22:59:19,125 INFO [ContainerLauncher #5] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus4.labs.teradata.com:45454
2016-03-08 22:59:19,125 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1457504560070_0004_m_000000_0
2016-03-08 22:59:19,126 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1457504560070_0004_m_000003_0
2016-03-08 22:59:19,127 INFO [ContainerLauncher #9] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus9.labs.teradata.com:45454
2016-03-08 22:59:19,129 INFO [ContainerLauncher #7] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus6.labs.teradata.com:45454
2016-03-08 22:59:19,130 INFO [ContainerLauncher #3] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus6.labs.teradata.com:45454
2016-03-08 22:59:19,131 INFO [ContainerLauncher #2] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus5.labs.teradata.com:45454
2016-03-08 22:59:19,134 INFO [ContainerLauncher #8] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus3.labs.teradata.com:45454
2016-03-08 22:59:19,137 INFO [ContainerLauncher #6] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : apus8.labs.teradata.com:45454
2016-03-08 22:59:19,169 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000004_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2016-03-08 22:59:19,170 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000006_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2016-03-08 22:59:19,170 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000000_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2016-03-08 22:59:19,170 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000003_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2016-03-08 22:59:19,171 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000001_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2016-03-08 22:59:19,171 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000002_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2016-03-08 22:59:19,171 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000005_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2016-03-08 22:59:19,172 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2016-03-08 22:59:19,172 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2016-03-08 22:59:19,173 INFO [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2016-03-08 22:59:19,173 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2016-03-08 22:59:19,173 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2016-03-08 22:59:19,173 INFO [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2016-03-08 22:59:19,173 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2016-03-08 22:59:19,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000004_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2016-03-08 22:59:19,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000006_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2016-03-08 22:59:19,175 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000000_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2016-03-08 22:59:19,176 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000001_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2016-03-08 22:59:19,176 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000005_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2016-03-08 22:59:19,176 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000003_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2016-03-08 22:59:19,176 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_m_000002_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2016-03-08 22:59:19,187 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000004 Task Transitioned from RUNNING to FAILED
2016-03-08 22:59:19,188 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000006 Task Transitioned from RUNNING to FAILED
2016-03-08 22:59:19,188 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000000 Task Transitioned from RUNNING to FAILED
2016-03-08 22:59:19,189 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000001 Task Transitioned from RUNNING to FAILED
2016-03-08 22:59:19,189 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000005 Task Transitioned from RUNNING to FAILED
2016-03-08 22:59:19,189 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000003 Task Transitioned from RUNNING to FAILED
2016-03-08 22:59:19,190 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_m_000002 Task Transitioned from RUNNING to FAILED
2016-03-08 22:59:19,190 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node apus3.labs.teradata.com
2016-03-08 22:59:19,193 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
2016-03-08 22:59:19,193 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:1 failedReduces:0
2016-03-08 22:59:19,194 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1457504560070_0004Job Transitioned from RUNNING to FAIL_WAIT
2016-03-08 22:59:19,194 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node apus6.labs.teradata.com
2016-03-08 22:59:19,195 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node apus6.labs.teradata.com
2016-03-08 22:59:19,195 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node apus4.labs.teradata.com
2016-03-08 22:59:19,195 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node apus5.labs.teradata.com
2016-03-08 22:59:19,195 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node apus9.labs.teradata.com
2016-03-08 22:59:19,195 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node apus8.labs.teradata.com
2016-03-08 22:59:19,195 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_r_000000 Task Transitioned from SCHEDULED to KILL_WAIT
2016-03-08 22:59:19,195 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1457504560070_0004_r_000000_0 TaskAttempt Transitioned from UNASSIGNED to KILLED
2016-03-08 22:59:19,196 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE
2016-03-08 22:59:19,196 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1457504560070_0004_r_000000 Task Transitioned from KILL_WAIT to KILLED
2016-03-08 22:59:19,196 ERROR [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Could not deallocate container for task attemptId attempt_1457504560070_0004_r_000000_0
2016-03-08 22:59:19,197 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1457504560070_0004Job Transitioned from FAIL_WAIT to FAIL_ABORT
2016-03-08 22:59:19,197 INFO [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT
2016-03-08 22:59:19,201 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1457504560070_0004Job Transitioned from FAIL_ABORT to FAILED
2016-03-08 22:59:19,202 INFO [Thread-89] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry
2016-03-08 22:59:19,203 INFO [Thread-89] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
2016-03-08 22:59:19,203 INFO [Thread-89] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified that shouldUnregistered is: true
2016-03-08 22:59:19,203 INFO [Thread-89] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
2016-03-08 22:59:19,203 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
2016-03-08 22:59:19,203 INFO [Thread-89] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
2016-03-08 22:59:19,204 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 15
2016-03-08 22:59:19,280 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event MAP_ATTEMPT_FAILED
2016-03-08 22:59:19,285 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event MAP_ATTEMPT_FAILED
2016-03-08 22:59:19,290 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event MAP_ATTEMPT_FAILED
2016-03-08 22:59:19,295 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event MAP_ATTEMPT_FAILED
2016-03-08 22:59:19,300 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event MAP_ATTEMPT_FAILED
2016-03-08 22:59:19,304 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event MAP_ATTEMPT_FAILED
2016-03-08 22:59:19,308 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2016-03-08 22:59:19,312 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2016-03-08 22:59:19,313 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2016-03-08 22:59:19,315 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2016-03-08 22:59:19,317 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2016-03-08 22:59:19,319 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2016-03-08 22:59:19,321 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2016-03-08 22:59:19,322 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event TASK_FAILED
2016-03-08 22:59:19,325 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_FAILED
2016-03-08 22:59:19,456 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://C423A:8020/user/hive/.staging/job_1457504560070_0004/job_1457504560070_0004_1.jhist to hdfs://C423A:8020/mr-history/tmp/hive/job_1457504560070_0004-1457506422934-hive-oozie%3Aaction%3AT%3Djava%3AW%3DTDExportMR%3AA%3Dexport%3AID%3D00001-1457506759193-0-0-FAILED-default-1457506429243.jhist_tmp
2016-03-08 22:59:19,550 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://C423A:8020/mr-history/tmp/hive/job_1457504560070_0004-1457506422934-hive-oozie%3Aaction%3AT%3Djava%3AW%3DTDExportMR%3AA%3Dexport%3AID%3D00001-1457506759193-0-0-FAILED-default-1457506429243.jhist_tmp
2016-03-08 22:59:19,562 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://C423A:8020/user/hive/.staging/job_1457504560070_0004/job_1457504560070_0004_1_conf.xml to hdfs://C423A:8020/mr-history/tmp/hive/job_1457504560070_0004_conf.xml_tmp
2016-03-08 22:59:19,614 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://C423A:8020/mr-history/tmp/hive/job_1457504560070_0004_conf.xml_tmp
2016-03-08 22:59:19,645 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://C423A:8020/mr-history/tmp/hive/job_1457504560070_0004.summary_tmp to hdfs://C423A:8020/mr-history/tmp/hive/job_1457504560070_0004.summary
2016-03-08 22:59:19,654 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://C423A:8020/mr-history/tmp/hive/job_1457504560070_0004_conf.xml_tmp to hdfs://C423A:8020/mr-history/tmp/hive/job_1457504560070_0004_conf.xml
2016-03-08 22:59:19,666 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://C423A:8020/mr-history/tmp/hive/job_1457504560070_0004-1457506422934-hive-oozie%3Aaction%3AT%3Djava%3AW%3DTDExportMR%3AA%3Dexport%3AID%3D00001-1457506759193-0-0-FAILED-default-1457506429243.jhist_tmp to hdfs://C423A:8020/mr-history/tmp/hive/job_1457504560070_0004-1457506422934-hive-oozie%3Aaction%3AT%3Djava%3AW%3DTDExportMR%3AA%3Dexport%3AID%3D00001-1457506759193-0-0-FAILED-default-1457506429243.jhist
2016-03-08 22:59:19,666 INFO [Thread-89] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
2016-03-08 22:59:19,671 INFO [Thread-89] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Task failed task_1457504560070_0004_m_000004
Job failed as tasks failed. failedMaps:1 failedReduces:0

2016-03-08 22:59:19,672 INFO [Thread-89] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: History url is http://apus2.labs.teradata.com:19888/jobhistory/job/job_1457504560070_0004
2016-03-08 22:59:19,680 INFO [Thread-89] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Waiting for application to be successfully unregistered.
2016-03-08 22:59:20,682 INFO [Thread-89] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:7 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:7 ContRel:0 HostLocal:6 RackLocal:1
2016-03-08 22:59:20,684 INFO [Thread-89] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://C423A /user/hive/.staging/job_1457504560070_0004
2016-03-08 22:59:20,711 INFO [Thread-89] org.apache.hadoop.ipc.Server: Stopping server on 46067
2016-03-08 22:59:20,712 INFO [IPC Server listener on 46067] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 46067
2016-03-08 22:59:20,712 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2016-03-08 22:59:20,714 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
3 REPLIES 3
Highlighted

Re: I am exporting data from teradata into Hadoop. but my query is getting failed by giving an error " Failed to write data"

Rising Star

@mukul jain Please post the query which you are using for exporting.

Highlighted

Re: I am exporting data from teradata into Hadoop. but my query is getting failed by giving an error " Failed to write data"

New Contributor

Thanks Mahesh for your response.

Please look at my query below:

INSERT INTO tiestpg.WebSales_Export_medium_text@dgtest SELECT * FROM tiestpg.WebSales_medium_text_jr2;

where dgtest is my foreign server where i'm providing all needed credentials to connect my database with hadoop database.

well, I think it's not a connection issue by looking at the logs. It seems container size issue. I reconfigured my container size severl times on different values, still the problem is not resolved. Can you please look at the problem.

Highlighted

Re: I am exporting data from teradata into Hadoop. but my query is getting failed by giving an error " Failed to write data"

Rising Star

Hi, were you able to sort out your issue?

please suggest the steps taken.

Thanks.

Rishit

Don't have an account?
Coming from Hortonworks? Activate your account here