Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

ALL hadoop-mapreduce-examples.jar fail cdh6

avatar
Explorer

Please help I've spent 3 days and still can't get an example to run including pi, terasort or even wordcount

New installation and have turned on HA for yarn and also hdfs, and installed the yarn MapReduce Framework JARs as well. Cluster looks health as seen below:

cluster1.png

 

If I try and run any of the examples ...they fail.

[hdfs@cdh-utility-1 opc]$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4
WARNING: Use "yarn jar" to launch YARN applications.
Number of Maps = 4
Samples per Map example.com= 4
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Starting Job
20/02/26 17:09:47 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1582677468069_0009
20/02/26 17:09:47 INFO input.FileInputFormat: Total input files to process : 4
20/02/26 17:09:47 INFO mapreduce.JobSubmitter: number of splits:4
20/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.zk-address is deprecated. Instead, use hadoop.zk.address
20/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
20/02/26 17:09:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1582677468069_0009
20/02/26 17:09:48 INFO mapreduce.JobSubmitter: Executing with tokens: []
20/02/26 17:09:48 INFO conf.Configuration: resource-types.xml not found
20/02/26 17:09:48 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
20/02/26 17:09:48 INFO impl.YarnClientImpl: Submitted application application_1582677468069_0009
20/02/26 17:09:48 INFO mapreduce.Job: The url to track the job: http://cdh-master-1.private.cdhvcn.examplevcn.com:8088/proxy/application_1582677468069_0009/
20/02/26 17:09:48 INFO mapreduce.Job: Running job: job_1582677468069_0009
20/02/26 17:09:54 INFO mapreduce.Job: Job job_1582677468069_0009 running in uber mode : false
20/02/26 17:09:54 INFO mapreduce.Job: map 0% reduce 0%
20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000002_0, Status : FAILED
[2020-02-26 17:09:55.814]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000004
Exit code: 1

[2020-02-26 17:09:55.815]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT


[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

 

20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000001_0, Status : FAILED
[2020-02-26 17:09:55.826]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000003
Exit code: 1

[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT


[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

 

20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000000_0, Status : FAILED
[2020-02-26 17:09:55.813]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000002
Exit code: 1

[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT


[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

 

20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000003_0, Status : FAILED
[2020-02-26 17:09:55.819]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000005
Exit code: 1

[2020-02-26 17:09:55.852]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT


[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT

1 ACCEPTED SOLUTION

avatar
Master Guru

That's great!  You should be able to replace the "NOT FOUND" values for those two fields with:

 

-Djava.net.preferIPv4Stack=true

This will configure it as CM usually has by default.

Not sure how the NOT FOUND ended up there.

View solution in original post

14 REPLIES 14

avatar
Master Guru

Hello @HadoopBD ,

 

Sorry to hear this has been causing you trouble.

I'll see if I can help with the investigation.

 

A few things you mention that are relevant:

- All examples fail, so that supports the assumption that it is something the jobs have in common that is contributing this issue.

- It also appears that the issue happen during container launch (as prelaunch.err contains the error)

- The failure indicates that somehow the class name was erroneously evaluated to the string "NOT" as seen here:
   Error: Could not find or load main class NOT

 

Since there is no class named "NOT" that implies whatever evaluation was done to attempt to execute that class was  incorrect.  For instance, you can get the same result if you run the following:

 

> java NOT
Error: Could not find or load main class NOT

So, the question goes to what happened before an attempt was made to launch a container.

One thing that can be an influence is the environment (env variables).  If this is an out-of-the-box installation, we would not expect this to happen, so if you have updated any YARN or HDFS configuration, it would be good note.

 

We could use a bit more information, so I would suggest getting the logs for the application like this:

# yarn logs -applicationId application_1582677468069_0009 > application_1582677468069_0009.log

 

This should allow you to look at all the logs for this application including any information that may have been missing from the job stdout.

 

Also consider trying to run the job from the same host as the Resource Manager to see if the failure is the same.  If there is something related to your client environment or hadoop configuration, the test may highlight that type of issue.

 

Cheers,

 

Ben

avatar
Explorer

I went ahead and tried on the RM server as well and same darn error.  I just don't get it.  

   As you had asked for the log .  Here it is:

  296	java.io.tmpdir: /data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001/tmp
   297	user.dir: /data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001
   298	user.name: yarn
   299	************************************************************/
   300	2020-02-26 17:09:49,953 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration
   301	2020-02-26 17:09:50,097 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 9 cluster_timestamp: 1582677468069 } attemptId: 1 } keyId: -1009427054)]
   302	2020-02-26 17:09:50,128 INFO [main] org.apache.hadoop.conf.Configuration: resource-types.xml not found
   303	2020-02-26 17:09:50,128 INFO [main] org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'.
   304	2020-02-26 17:09:50,136 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
   305	2020-02-26 17:09:50,138 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
   306	2020-02-26 17:09:50,173 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2
   307	2020-02-26 17:09:50,173 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
   308	2020-02-26 17:09:50,627 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   309	2020-02-26 17:09:50,662 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
   310	2020-02-26 17:09:50,830 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
   311	2020-02-26 17:09:50,831 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
   312	2020-02-26 17:09:50,831 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
   313	2020-02-26 17:09:50,832 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
   314	2020-02-26 17:09:50,832 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
   315	2020-02-26 17:09:50,833 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
   316	2020-02-26 17:09:50,834 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
   317	2020-02-26 17:09:50,834 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
   318	2020-02-26 17:09:50,864 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
   319	2020-02-26 17:09:50,881 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
   320	2020-02-26 17:09:50,895 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
   321	2020-02-26 17:09:50,909 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
   322	2020-02-26 17:09:50,946 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
   323	2020-02-26 17:09:51,187 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties
   324	2020-02-26 17:09:51,244 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
   325	2020-02-26 17:09:51,244 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
   326	2020-02-26 17:09:51,252 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1582677468069_0009 to jobTokenSecretManager
   327	2020-02-26 17:09:51,358 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1582677468069_0009 because: not enabled;
   328	2020-02-26 17:09:51,374 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1582677468069_0009 = 472. Number of splits = 4
   329	2020-02-26 17:09:51,375 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1582677468069_0009 = 1
   330	2020-02-26 17:09:51,375 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from NEW to INITED
   331	2020-02-26 17:09:51,376 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1582677468069_0009.
   332	2020-02-26 17:09:51,399 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
   333	2020-02-26 17:09:51,408 INFO [Socket Reader #1 for port 46625] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 46625
   334	2020-02-26 17:09:51,577 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
   335	2020-02-26 17:09:51,578 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
   336	2020-02-26 17:09:51,578 INFO [IPC Server listener on 46625] org.apache.hadoop.ipc.Server: IPC Server listener on 46625: starting
   337	2020-02-26 17:09:51,579 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at cdh-worker-3.private.cdhvcn.hiddenvcn.com/10.161.2.5:46625
   338	2020-02-26 17:09:51,609 INFO [main] org.eclipse.jetty.util.log: Logging initialized @2635ms
   339	2020-02-26 17:09:51,697 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
   340	2020-02-26 17:09:51,700 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
   341	2020-02-26 17:09:51,705 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
   342	2020-02-26 17:09:51,747 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
   343	2020-02-26 17:09:51,747 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
   344	2020-02-26 17:09:51,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
   345	2020-02-26 17:09:51,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
   346	2020-02-26 17:09:52,083 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
   347	2020-02-26 17:09:52,084 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 45277
   348	2020-02-26 17:09:52,085 INFO [main] org.eclipse.jetty.server.Server: jetty-9.3.20.v20170531
   349	2020-02-26 17:09:52,122 INFO [main] org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@39109136{/static,jar:file:/data4/yarn/nm/filecache/12/3.0.0-cdh6.1.1-mr-framework.tar.gz/hadoop-yarn-common-3.0.0-cdh6.1.1.jar!/webapps/static,AVAILABLE}
   350	2020-02-26 17:09:53,009 INFO [main] org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@1f939a0f{/,file:///data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001/tmp/jetty-0.0.0.0-45277-mapreduce-_-any-7944828710115519068.dir/webapp/,AVAILABLE}{/mapreduce}
   351	2020-02-26 17:09:53,017 INFO [main] org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@7112ce6{HTTP/1.1,[http/1.1]}{0.0.0.0:45277}
   352	2020-02-26 17:09:53,017 INFO [main] org.eclipse.jetty.server.Server: Started @4043ms
   353	2020-02-26 17:09:53,017 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app mapreduce started at 45277
   354	2020-02-26 17:09:53,022 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
   355	2020-02-26 17:09:53,023 INFO [Socket Reader #1 for port 34533] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34533
   356	2020-02-26 17:09:53,028 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
   357	2020-02-26 17:09:53,028 INFO [IPC Server listener on 34533] org.apache.hadoop.ipc.Server: IPC Server listener on 34533: starting
   358	2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
   359	2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
   360	2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
   361	2020-02-26 17:09:53,060 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the mappers will be scheduled using OPPORTUNISTIC containers
   362	2020-02-26 17:09:53,154 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: maxContainerCapability: <memory:8192, vCores:2>
   363	2020-02-26 17:09:53,154 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: queue: root.users.hdfs
   364	2020-02-26 17:09:53,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
   365	2020-02-26 17:09:53,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
   366	2020-02-26 17:09:53,169 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from INITED to SETUP
   367	2020-02-26 17:09:53,171 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
   368	2020-02-26 17:09:53,182 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from SETUP to RUNNING
   369	2020-02-26 17:09:53,274 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1582677468069_0009, File: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist
   370	2020-02-26 17:09:53,288 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from NEW to SCHEDULED
   371	2020-02-26 17:09:53,311 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from NEW to SCHEDULED
   372	2020-02-26 17:09:53,313 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from NEW to SCHEDULED
   373	2020-02-26 17:09:53,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from NEW to SCHEDULED
   374	2020-02-26 17:09:53,316 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from NEW to SCHEDULED
   375	2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
   376	2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED
   377	2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED
   378	2020-02-26 17:09:53,318 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED
   379	2020-02-26 17:09:53,318 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
   380	2020-02-26 17:09:53,319 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>
   381	2020-02-26 17:09:53,332 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:1024, vCores:1>
   382	2020-02-26 17:09:54,158 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
   383	2020-02-26 17:09:54,189 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=6 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:454656, vCores:63> knownNMs=4
   384	2020-02-26 17:09:54,192 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:454656, vCores:63>
   385	2020-02-26 17:09:54,192 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
   386	2020-02-26 17:09:55,206 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
   387	2020-02-26 17:09:55,209 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000002 to attempt_1582677468069_0009_m_000000_0
   388	2020-02-26 17:09:55,211 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000003 to attempt_1582677468069_0009_m_000001_0
   389	2020-02-26 17:09:55,212 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000004 to attempt_1582677468069_0009_m_000002_0
   390	2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000005 to attempt_1582677468069_0009_m_000003_0
   391	2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:450560, vCores:59>
   392	2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
   393	2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1
   394	2020-02-26 17:09:55,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://nameservice1/user/hdfs/.staging/job_1582677468069_0009/job.jar
   395	2020-02-26 17:09:55,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hdfs/.staging/job_1582677468069_0009/job.xml
   396	2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container
   397	2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1
   398	2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
   399	2020-02-26 17:09:55,308 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   400	2020-02-26 17:09:55,311 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   401	2020-02-26 17:09:55,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   402	2020-02-26 17:09:55,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   403	2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   404	2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   405	2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   406	2020-02-26 17:09:55,316 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   407	2020-02-26 17:09:55,318 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000002 taskAttempt attempt_1582677468069_0009_m_000000_0
   408	2020-02-26 17:09:55,318 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000003 taskAttempt attempt_1582677468069_0009_m_000001_0
   409	2020-02-26 17:09:55,318 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000004 taskAttempt attempt_1582677468069_0009_m_000002_0
   410	2020-02-26 17:09:55,319 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000005 taskAttempt attempt_1582677468069_0009_m_000003_0
   411	2020-02-26 17:09:55,321 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_0
   412	2020-02-26 17:09:55,321 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_0
   413	2020-02-26 17:09:55,321 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_0
   414	2020-02-26 17:09:55,321 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_0
   415	2020-02-26 17:09:55,404 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_0 : 13562
   416	2020-02-26 17:09:55,405 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_0 : 13562
   417	2020-02-26 17:09:55,405 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_0] using containerId: [container_e02_1582677468069_0009_01_000005 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
   418	2020-02-26 17:09:55,408 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_0 : 13562
   419	2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
   420	2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_0] using containerId: [container_e02_1582677468069_0009_01_000003 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
   421	2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
   422	2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_0] using containerId: [container_e02_1582677468069_0009_01_000002 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
   423	2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
   424	2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from SCHEDULED to RUNNING
   425	2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from SCHEDULED to RUNNING
   426	2020-02-26 17:09:55,411 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from SCHEDULED to RUNNING
   427	2020-02-26 17:09:55,412 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_0 : 13562
   428	2020-02-26 17:09:55,412 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_0] using containerId: [container_e02_1582677468069_0009_01_000004 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
   429	2020-02-26 17:09:55,413 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
   430	2020-02-26 17:09:55,413 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from SCHEDULED to RUNNING
   431	2020-02-26 17:09:56,221 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=6 release= 0 newContainers=0 finishedContainers=4 resourcelimit=<memory:454656, vCores:63> knownNMs=4
   432	2020-02-26 17:09:56,221 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000004
   433	2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000003
   434	2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000002
   435	2020-02-26 17:09:56,222 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000002_0: [2020-02-26 17:09:55.814]Exception from container-launch.
   436	Container id: container_e02_1582677468069_0009_01_000004
   437	Exit code: 1
       
   438	[2020-02-26 17:09:55.815]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   439	Last 4096 bytes of prelaunch.err :
   440	Last 4096 bytes of stderr :
   441	Error: Could not find or load main class NOT
       
       
   442	[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   443	Last 4096 bytes of prelaunch.err :
   444	Last 4096 bytes of stderr :
   445	Error: Could not find or load main class NOT
       
       
       
   446	2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000005
   447	2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:454656, vCores:63>
   448	2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
   449	2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1
   450	2020-02-26 17:09:56,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
   451	2020-02-26 17:09:56,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_0: [2020-02-26 17:09:55.826]Exception from container-launch.
   452	Container id: container_e02_1582677468069_0009_01_000003
   453	Exit code: 1
       
   454	[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   455	Last 4096 bytes of prelaunch.err :
   456	Last 4096 bytes of stderr :
   457	Error: Could not find or load main class NOT
       
       
   458	[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   459	Last 4096 bytes of prelaunch.err :
   460	Last 4096 bytes of stderr :
   461	Error: Could not find or load main class NOT
       
       
       
   462	2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
   463	2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_0: [2020-02-26 17:09:55.813]Exception from container-launch.
   464	Container id: container_e02_1582677468069_0009_01_000002
   465	Exit code: 1
       
   466	[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   467	Last 4096 bytes of prelaunch.err :
   468	Last 4096 bytes of stderr :
   469	Error: Could not find or load main class NOT
       
       
   470	[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   471	Last 4096 bytes of prelaunch.err :
   472	Last 4096 bytes of stderr :
   473	Error: Could not find or load main class NOT
       
       
       
   474	2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
   475	2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_0: [2020-02-26 17:09:55.819]Exception from container-launch.
   476	Container id: container_e02_1582677468069_0009_01_000005
   477	Exit code: 1
       
   478	[2020-02-26 17:09:55.852]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   479	Last 4096 bytes of prelaunch.err :
   480	Last 4096 bytes of stderr :
   481	Error: Could not find or load main class NOT
       
       
   482	[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   483	Last 4096 bytes of prelaunch.err :
   484	Last 4096 bytes of stderr :
   485	Error: Could not find or load main class NOT
       
       
       
   486	2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
   487	2020-02-26 17:09:56,228 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000004 taskAttempt attempt_1582677468069_0009_m_000002_0
   488	2020-02-26 17:09:56,237 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000003 taskAttempt attempt_1582677468069_0009_m_000001_0
   489	2020-02-26 17:09:56,240 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000002 taskAttempt attempt_1582677468069_0009_m_000000_0
   490	2020-02-26 17:09:56,241 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000005 taskAttempt attempt_1582677468069_0009_m_000003_0
   491	2020-02-26 17:09:56,243 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com
   492	2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from NEW to UNASSIGNED
   493	2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from NEW to UNASSIGNED
   494	2020-02-26 17:09:56,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com
   495	2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
   496	2020-02-26 17:09:56,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com
   497	2020-02-26 17:09:56,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted host cdh-worker-1.private.cdhvcn.hiddenvcn.com
   498	2020-02-26 17:09:56,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from NEW to UNASSIGNED
   499	2020-02-26 17:09:56,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_1 to list of failed maps
   500	2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000001_1 to list of failed maps
   501	2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000000_1 to list of failed maps
   502	2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000003_1 to list of failed maps
   503	2020-02-26 17:09:57,223 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1
   504	2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:339968, vCores:47> knownNMs=4
   505	2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=1 blacklistRemovals=0
   506	2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:339968, vCores:47>
   507	2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
   508	2020-02-26 17:09:58,231 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
   509	2020-02-26 17:09:58,232 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000006, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
   510	2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
   511	2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000006 to attempt_1582677468069_0009_m_000002_1
   512	2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000007, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
   513	2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
   514	2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000007 to attempt_1582677468069_0009_m_000001_1
   515	2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000008, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
   516	2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
   517	2020-02-26 17:09:58,233 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   518	2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000008 to attempt_1582677468069_0009_m_000000_1
   519	2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000009, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
   520	2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
   521	2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000009 to attempt_1582677468069_0009_m_000003_1
   522	2020-02-26 17:09:58,234 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   523	2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:335872, vCores:43>
   524	2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
   525	2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
   526	2020-02-26 17:09:58,235 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   527	2020-02-26 17:09:58,236 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   528	2020-02-26 17:09:58,236 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   529	2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   530	2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   531	2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   532	2020-02-26 17:09:58,238 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000006 taskAttempt attempt_1582677468069_0009_m_000002_1
   533	2020-02-26 17:09:58,238 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000009 taskAttempt attempt_1582677468069_0009_m_000003_1
   534	2020-02-26 17:09:58,238 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_1
   535	2020-02-26 17:09:58,238 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000007 taskAttempt attempt_1582677468069_0009_m_000001_1
   536	2020-02-26 17:09:58,238 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000008 taskAttempt attempt_1582677468069_0009_m_000000_1
   537	2020-02-26 17:09:58,238 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_1
   538	2020-02-26 17:09:58,238 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_1
   539	2020-02-26 17:09:58,238 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_1
   540	2020-02-26 17:09:58,265 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_1 : 13562
   541	2020-02-26 17:09:58,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_1] using containerId: [container_e02_1582677468069_0009_01_000009 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
   542	2020-02-26 17:09:58,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
   543	2020-02-26 17:09:58,273 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_1 : 13562
   544	2020-02-26 17:09:58,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_1] using containerId: [container_e02_1582677468069_0009_01_000007 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
   545	2020-02-26 17:09:58,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
   546	2020-02-26 17:09:58,280 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_1 : 13562
   547	2020-02-26 17:09:58,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_1] using containerId: [container_e02_1582677468069_0009_01_000006 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
   548	2020-02-26 17:09:58,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
   549	2020-02-26 17:09:58,289 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_1 : 13562
   550	2020-02-26 17:09:58,289 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_1] using containerId: [container_e02_1582677468069_0009_01_000008 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
   551	2020-02-26 17:09:58,289 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
   552	2020-02-26 17:09:59,238 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=3 resourcelimit=<memory:338944, vCores:46> knownNMs=4
   553	2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000008
   554	2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000009
   555	2020-02-26 17:09:59,239 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_1: [2020-02-26 17:09:58.706]Exception from container-launch.
   556	Container id: container_e02_1582677468069_0009_01_000008
   557	Exit code: 1
       
   558	[2020-02-26 17:09:58.708]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   559	Last 4096 bytes of prelaunch.err :
   560	Last 4096 bytes of stderr :
   561	Error: Could not find or load main class NOT
       
       
   562	[2020-02-26 17:09:58.708]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   563	Last 4096 bytes of prelaunch.err :
   564	Last 4096 bytes of stderr :
   565	Error: Could not find or load main class NOT
       
       
       
   566	2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000007
   567	2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:338944, vCores:46>
   568	2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
   569	2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
   570	2020-02-26 17:09:59,239 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
   571	2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_1: [2020-02-26 17:09:58.721]Exception from container-launch.
   572	Container id: container_e02_1582677468069_0009_01_000009
   573	Exit code: 1
       
   574	[2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   575	Last 4096 bytes of prelaunch.err :
   576	Last 4096 bytes of stderr :
   577	Error: Could not find or load main class NOT
       
       
   578	[2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   579	Last 4096 bytes of prelaunch.err :
   580	Last 4096 bytes of stderr :
   581	Error: Could not find or load main class NOT
       
       
       
   582	2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
   583	2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_1: [2020-02-26 17:09:58.712]Exception from container-launch.
   584	Container id: container_e02_1582677468069_0009_01_000007
   585	Exit code: 1
       
   586	[2020-02-26 17:09:58.732]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   587	Last 4096 bytes of prelaunch.err :
   588	Last 4096 bytes of stderr :
   589	Error: Could not find or load main class NOT
       
       
   590	[2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   591	Last 4096 bytes of prelaunch.err :
   592	Last 4096 bytes of stderr :
   593	Error: Could not find or load main class NOT
       
       
       
   594	2020-02-26 17:09:59,241 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
   595	2020-02-26 17:09:59,241 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000008 taskAttempt attempt_1582677468069_0009_m_000000_1
   596	2020-02-26 17:09:59,242 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000009 taskAttempt attempt_1582677468069_0009_m_000003_1
   597	2020-02-26 17:09:59,244 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000007 taskAttempt attempt_1582677468069_0009_m_000001_1
   598	2020-02-26 17:09:59,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com
   599	2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from NEW to UNASSIGNED
   600	2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com
   601	2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from NEW to UNASSIGNED
   602	2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com
   603	2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted host cdh-worker-3.private.cdhvcn.hiddenvcn.com
   604	2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from NEW to UNASSIGNED
   605	2020-02-26 17:09:59,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000000_2 to list of failed maps
   606	2020-02-26 17:09:59,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000003_2 to list of failed maps
   607	2020-02-26 17:09:59,248 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000001_2 to list of failed maps
   608	2020-02-26 17:10:00,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
   609	2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:229376, vCores:32> knownNMs=4
   610	2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=1 blacklistRemovals=0
   611	2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Ignore blacklisting set to true. Known: 4, Blacklisted: 2, 50%
   612	2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000006
   613	2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:229376, vCores:32>
   614	2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
   615	2020-02-26 17:10:00,243 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000002_1: [2020-02-26 17:09:58.725]Exception from container-launch.
   616	Container id: container_e02_1582677468069_0009_01_000006
   617	Exit code: 1
       
   618	[2020-02-26 17:09:58.734]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   619	Last 4096 bytes of prelaunch.err :
   620	Last 4096 bytes of stderr :
   621	Error: Could not find or load main class NOT
       
       
   622	[2020-02-26 17:09:58.734]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   623	Last 4096 bytes of prelaunch.err :
   624	Last 4096 bytes of stderr :
   625	Error: Could not find or load main class NOT
       
       
       
   626	2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
   627	2020-02-26 17:10:00,244 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
   628	2020-02-26 17:10:00,244 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000006 taskAttempt attempt_1582677468069_0009_m_000002_1
   629	2020-02-26 17:10:00,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_2 TaskAttempt Transitioned from NEW to UNASSIGNED
   630	2020-02-26 17:10:00,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_2 to list of failed maps
   631	2020-02-26 17:10:01,244 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
   632	2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=3 finishedContainers=0 resourcelimit=<memory:451584, vCores:60> knownNMs=4
   633	2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=0 blacklistRemovals=2
   634	2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 3
   635	2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000010, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
   636	2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
   637	2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000010 to attempt_1582677468069_0009_m_000000_2
   638	2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000011, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
   639	2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
   640	2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000011 to attempt_1582677468069_0009_m_000003_2
   641	2020-02-26 17:10:01,251 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   642	2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000012, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
   643	2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
   644	2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000012 to attempt_1582677468069_0009_m_000001_2
   645	2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:451584, vCores:60>
   646	2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
   647	2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:11 ContRel:0 HostLocal:3 RackLocal:1
   648	2020-02-26 17:10:01,252 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   649	2020-02-26 17:10:01,254 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   650	2020-02-26 17:10:01,254 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   651	2020-02-26 17:10:01,255 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
   652	2020-02-26 17:10:01,255 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
   653	2020-02-26 17:10:01,255 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000010 taskAttempt attempt_1582677468069_0009_m_000000_2
   654	2020-02-26 17:10:01,255 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000011 taskAttempt attempt_1582677468069_0009_m_000003_2
   655	2020-02-26 17:10:01,255 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000012 taskAttempt attempt_1582677468069_0009_m_000001_2
   656	2020-02-26 17:10:01,256 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_2
   657	2020-02-26 17:10:01,255 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_2
   658	2020-02-26 17:10:01,256 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_2
   659	2020-02-26 17:10:01,267 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_2 : 13562
   660	2020-02-26 17:10:01,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_2] using containerId: [container_e02_1582677468069_0009_01_000011 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
   661	2020-02-26 17:10:01,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
   662	2020-02-26 17:10:01,271 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_2 : 13562
   663	2020-02-26 17:10:01,271 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_2] using containerId: [container_e02_1582677468069_0009_01_000012 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
   664	2020-02-26 17:10:01,271 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
   665	2020-02-26 17:10:01,273 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_2 : 13562
   666	2020-02-26 17:10:01,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_2] using containerId: [container_e02_1582677468069_0009_01_000010 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
   667	2020-02-26 17:10:01,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
   668	2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=4 finishedContainers=3 resourcelimit=<memory:450560, vCores:59> knownNMs=4
   669	2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000011
   670	2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000012
   671	2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000010
   672	2020-02-26 17:10:02,257 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_2: [2020-02-26 17:10:01.685]Exception from container-launch.
   673	Container id: container_e02_1582677468069_0009_01_000011
   674	Exit code: 1
       
   675	[2020-02-26 17:10:01.686]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   676	Last 4096 bytes of prelaunch.err :
   677	Last 4096 bytes of stderr :
   678	Error: Could not find or load main class NOT
       
       
   679	[2020-02-26 17:10:01.691]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   680	Last 4096 bytes of prelaunch.err :
   681	Last 4096 bytes of stderr :
   682	Error: Could not find or load main class NOT
       
       
       
   683	2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
   684	2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041
   685	2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_2: [2020-02-26 17:10:01.698]Exception from container-launch.
   686	Container id: container_e02_1582677468069_0009_01_000012
   687	Exit code: 1
       
   688	[2020-02-26 17:10:01.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   689	Last 4096 bytes of prelaunch.err :
   690	Last 4096 bytes of stderr :
   691	Error: Could not find or load main class NOT
       
       
   692	[2020-02-26 17:10:01.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
   693	Last 4096 bytes of prelaunch.err :
   694	Last 4096 bytes of stderr :
   695	Error: Could not find or load main class NOT
       
       
       
   696	2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000013, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
   697	2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
   698	2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041
   699	2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000013 to attempt_1582677468069_0009_m_000002_2
   700	2020-02-26 17:10:02,259 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_2: [2020-02-26 17:10:01.688]Exception from container-launch.
   701	Container id: container_e02_1582677468069_0009_01_000010
   702	Exit code: 1

 

avatar
Master Guru

Hi @HadoopBD ,

 

The logs provided don't contain the environment or any clues that would help us understand what may have been passed to the command that is attempting to launch a container.  It might have been missed in my previous message, but a good way of getting some more detail about the Application Masters, containers, etc. would be to collect logs via the "yarn logs" command.

 

For instance:

yarn logs -applicationId application_1582677468069_0009 > application_1582677468069_0009.log

 

Resource Manager logs tell us some things, but not the whole picture.  If you can run the above, the output may be pretty big, but if you can take a look and see if you can find the string "NOT" in there that might be a start.

 

If you are on Cloudera Manager 6.3 or higher, you can try the following to collect more information about the container launch:

 

(1)

Via Cloudera Manager, set the following configuration to 600 (10 minutes): Localized Dir Deletion Delay.  This will tell the Node Manager to delay 10 minutes before cleaning up the container launcher.  This will help us review the files used in the failed container launch

 

(2)

 

Set the following YARN configuration: Enable Container Launch Debug Information.  Check the box to enable it.  This will allow you to collect extra container launch information in the "yarn logs -applicationId" output.

 

(3)

 

SAVE your changes and then Restart YARN service from CM

 

(4)

 

Run a test mapreduce job (pi for instance)

 

(5)

 

After it fails, run the following to collect the aggregated logs for the job:

 

yarn logs -applicationId <app_id>

 

NOTE:  you can direct the output to a file so you can search in the file.

 

(6)

 

Look for "launch_container" in the output to find the launch information.

 

Again, the output might be pretty big so you can try adding it here or look for things that may be relevant. 

avatar
Super Guru
@HadoopBD ,

If possible, if you can attach log as a file to this thread, it would be better, as the log messages make the page lengthy and hard to navigate around.

Cheers

avatar
Explorer

I don't see anything here at all for doing an attachment, just links 😞  😞 .  so I'll apologize now.  But, here is more of the log.  



753 [2020-02-26 17:10:02.569]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
754 Last 4096 bytes of prelaunch.err :
755 Last 4096 bytes of stderr :
756 Error: Could not find or load main class NOT



757 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000015
758 2020-02-26 17:10:03,267 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container container_e02_1582677468069_0009_01_000015
759 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000016
760 2020-02-26 17:10:03,267 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container container_e02_1582677468069_0009_01_000016
761 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
762 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000017, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
763 2020-02-26 17:10:03,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041
764 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
765 2020-02-26 17:10:03,267 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000013 taskAttempt attempt_1582677468069_0009_m_000002_2
766 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000017 to attempt_1582677468069_0009_m_000003_3
767 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:453632, vCores:62>
768 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
769 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:16 ContRel:3 HostLocal:3 RackLocal:1
770 2020-02-26 17:10:03,269 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
771 2020-02-26 17:10:03,270 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
772 2020-02-26 17:10:03,270 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from NEW to UNASSIGNED
773 2020-02-26 17:10:03,270 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000017 taskAttempt attempt_1582677468069_0009_m_000003_3
774 2020-02-26 17:10:03,270 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_3 to list of failed maps
775 2020-02-26 17:10:03,270 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_3
776 2020-02-26 17:10:03,281 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_3 : 13562
777 2020-02-26 17:10:03,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_3] using containerId: [container_e02_1582677468069_0009_01_000017 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
778 2020-02-26 17:10:03,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 TaskAttempt Transitioned from ASSIGNED to RUNNING
779 2020-02-26 17:10:04,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:16 ContRel:3 HostLocal:3 RackLocal:1
780 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=3 finishedContainers=1 resourcelimit=<memory:451584, vCores:60> knownNMs=4
781 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000017
782 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 3
783 2020-02-26 17:10:04,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_3: [2020-02-26 17:10:03.568]Exception from container-launch.
784 Container id: container_e02_1582677468069_0009_01_000017
785 Exit code: 1

786 [2020-02-26 17:10:03.570]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
787 Last 4096 bytes of prelaunch.err :
788 Last 4096 bytes of stderr :
789 Error: Could not find or load main class NOT


790 [2020-02-26 17:10:03.570]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
791 Last 4096 bytes of prelaunch.err :
792 Last 4096 bytes of stderr :
793 Error: Could not find or load main class NOT



794 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000018, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
795 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
796 2020-02-26 17:10:04,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041
797 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000018 to attempt_1582677468069_0009_m_000001_3
798 2020-02-26 17:10:04,274 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000017 taskAttempt attempt_1582677468069_0009_m_000003_3
799 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000019, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
800 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
801 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000019 to attempt_1582677468069_0009_m_000000_3
802 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000020, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
803 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
804 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000020 to attempt_1582677468069_0009_m_000002_3
805 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:451584, vCores:60>
806 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
807 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:19 ContRel:3 HostLocal:3 RackLocal:1
808 2020-02-26 17:10:04,276 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from RUNNING to FAILED
809 2020-02-26 17:10:04,277 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
810 2020-02-26 17:10:04,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
811 2020-02-26 17:10:04,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
812 2020-02-26 17:10:04,279 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
813 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
814 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
815 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
816 2020-02-26 17:10:04,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
817 2020-02-26 17:10:04,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from RUNNING to FAIL_WAIT
818 2020-02-26 17:10:04,282 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000018 taskAttempt attempt_1582677468069_0009_m_000001_3
819 2020-02-26 17:10:04,282 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000019 taskAttempt attempt_1582677468069_0009_m_000000_3
820 2020-02-26 17:10:04,282 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_3
821 2020-02-26 17:10:04,283 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_3
822 2020-02-26 17:10:04,283 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000020 taskAttempt attempt_1582677468069_0009_m_000002_3
823 2020-02-26 17:10:04,283 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_3
824 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from RUNNING to KILL_WAIT
825 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from RUNNING to KILL_WAIT
826 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from RUNNING to KILL_WAIT
827 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from SCHEDULED to KILL_WAIT
828 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP
829 2020-02-26 17:10:04,284 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP
830 2020-02-26 17:10:04,284 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP
831 2020-02-26 17:10:04,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_r_000000_0 TaskAttempt Transitioned from UNASSIGNED to KILLED
832 2020-02-26 17:10:04,285 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE
833 2020-02-26 17:10:04,285 ERROR [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Could not deallocate container for task attemptId attempt_1582677468069_0009_r_000000_0
834 2020-02-26 17:10:04,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from KILL_WAIT to KILLED
835 2020-02-26 17:10:04,286 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000019 taskAttempt attempt_1582677468069_0009_m_000000_3
836 2020-02-26 17:10:04,286 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000018 taskAttempt attempt_1582677468069_0009_m_000001_3
837 2020-02-26 17:10:04,286 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000020 taskAttempt attempt_1582677468069_0009_m_000002_3
838 2020-02-26 17:10:04,297 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_3 : 13562
839 2020-02-26 17:10:04,298 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000001_3
840 2020-02-26 17:10:04,305 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_3 : 13562
841 2020-02-26 17:10:04,306 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000002_3
842 2020-02-26 17:10:04,310 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_3 : 13562
843 2020-02-26 17:10:04,310 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000000_3
844 2020-02-26 17:10:04,324 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
845 2020-02-26 17:10:04,324 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
846 2020-02-26 17:10:04,325 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
847 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
848 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
849 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
850 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000000_3
851 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000001_3
852 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000002_3
853 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_3] using containerId: [container_e02_1582677468069_0009_01_000019 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
854 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED
855 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_3] using containerId: [container_e02_1582677468069_0009_01_000018 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
856 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED
857 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_3] using containerId: [container_e02_1582677468069_0009_01_000020 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
858 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED
859 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from KILL_WAIT to KILLED
860 2020-02-26 17:10:04,333 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from KILL_WAIT to KILLED
861 2020-02-26 17:10:04,333 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from KILL_WAIT to KILLED
862 2020-02-26 17:10:04,334 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from FAIL_WAIT to FAIL_ABORT
863 2020-02-26 17:10:04,334 INFO [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT
864 2020-02-26 17:10:04,344 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from FAIL_ABORT to FAILED
865 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly, recording last MRAppMaster retry
866 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
867 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified that shouldUnregistered is: true
868 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
869 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
870 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
871 2020-02-26 17:10:04,346 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 1
872 2020-02-26 17:10:04,347 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_FAILED
873 2020-02-26 17:10:04,389 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp
874 2020-02-26 17:10:04,424 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied from: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist to done location: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp
875 2020-02-26 17:10:04,427 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Set historyUrl to http://cdh-utility-1.public.cdhvcn.hiddenvcn.com:19888/jobhistory/job/job_1582677468069_0009
876 2020-02-26 17:10:04,428 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1_conf.xml to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp
877 2020-02-26 17:10:04,460 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied from: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1_conf.xml to done location: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp
878 2020-02-26 17:10:04,466 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009.summary_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009.summary
879 2020-02-26 17:10:04,469 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml
880 2020-02-26 17:10:04,471 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist
881 2020-02-26 17:10:04,471 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
882 2020-02-26 17:10:04,476 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Task failed task_1582677468069_0009_m_000003
883 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

884 2020-02-26 17:10:04,476 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: History url is http://cdh-utility-1.public.cdhvcn.hiddenvcn.com:19888/jobhistory/job/job_1582677468069_0009
885 2020-02-26 17:10:04,481 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Waiting for application to be successfully unregistered.
886 2020-02-26 17:10:05,482 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:19 ContRel:3 HostLocal:3 RackLocal:1
887 2020-02-26 17:10:05,483 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://nameservice1 /user/hdfs/.staging/job_1582677468069_0009
888 2020-02-26 17:10:05,487 INFO [Thread-90] org.apache.hadoop.ipc.Server: Stopping server on 34533
889 2020-02-26 17:10:05,488 INFO [IPC Server listener on 34533] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 34533
890 2020-02-26 17:10:05,488 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
891 2020-02-26 17:10:05,490 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
892 2020-02-26 17:10:05,490 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
893 2020-02-26 17:10:10,491 INFO [Thread-90] org.apache.hadoop.ipc.Server: Stopping server on 46625
894 2020-02-26 17:10:10,491 INFO [IPC Server listener on 46625] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 46625
895 2020-02-26 17:10:10,491 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
896 2020-02-26 17:10:10,494 INFO [Thread-90] org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.w.WebAppContext@1f939a0f{/,null,UNAVAILABLE}{/mapreduce}
897 2020-02-26 17:10:10,497 INFO [Thread-90] org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@7112ce6{HTTP/1.1,[http/1.1]}{0.0.0.0:0}
898 2020-02-26 17:10:10,498 INFO [Thread-90] org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@39109136{/static,jar:file:/data4/yarn/nm/filecache/12/3.0.0-cdh6.1.1-mr-framework.tar.gz/hadoop-yarn-common-3.0.0-cdh6.1.1.jar!/webapps/static,UNAVAILABLE}

899 End of LogType:syslog
900 ***********************************************************************

avatar
Master Guru

Hello @HadoopBD ,

 

It appears to me that the log you provided may not have been captured with the steps I suggested before EricL's comment.  Can you confirm how you retrieved these logs.  From what I see, this only hints that there are problems launching containers, but in order to try to see why, we will need to capture more information.

 

Thanks,

 

Ben

 

avatar
Super Guru
Hi @HadoopBD,

The log you provided is RM log, not application log that Ben asked earlier. Please run below command:
yarn logs -applicationId application_1582677468069_0009 > application_1582677468069_0009.log

And then find the snippet that has the error, as the log will likely to be very big.

I also noticed that the log you provided earlier contained your domain name, if you have concerns, we can redact them for you. Please advise.

Thanks
Eric

avatar
Explorer

The snippets from the log that I uploaded were actually from the output of the yarn log command as both of you had requested.  If there is a way I can upload the full log I'd be more than happy to.  As for remove of the domain name I actually change them before I submit so they are not feasible anyway 🙂 🙂   

 

avatar
Master Guru

Hello @HadoopBD ,

 

Thanks for providing the logs, but they do not contain what we would expect if you had followed the steps to enabled container launch debug information.  I am guessing you missed my steps during the threaded conversation.

 

Basically, the standard logs show you some information, but not all.  We are missing the actual files and log information about how the "launch_container" processes was started and what was passed to the script use to execute the necessary java.  In order to capture that information, which will most likely give us some sort of clue about the cause of this issue.

 

The steps to retain container launching information and also allow "yarn logs" command to obtain them is in CM 6.3 so I wanted to find out if you had that version.

 

Here are the steps:

If you are on Cloudera Manager 6.3 or higher, you can try the following to collect more information about the container launch:

 

(1)

Via Cloudera Manager, set the following configuration to 600 (10 minutes): Localized Dir Deletion Delay.  This will tell the Node Manager to delay 10 minutes before cleaning up the container launcher.  This will help us review the files used in the failed container launch

 

(2)

 

Set the following YARN configuration: Enable Container Launch Debug Information.  Check the box to enable it.  This will allow you to collect extra container launch information in the "yarn logs -applicationId" output.

 

(3)

 

SAVE your changes and then Restart YARN service from CM

 

(4)

 

Run a test mapreduce job (pi for instance)

 

(5)

 

After it fails, run the following to collect the aggregated logs for the job:

 

yarn logs -applicationId <app_id>

 

NOTE:  you can direct the output to a file so you can search in the file.

 

(6)

 

Look for "launch_container" in the output to find the launch information.

 

I just ran through a test and a lot more details about how the command will be launched is available.  I truly believe it will help us assess a cause so we can find a solution.