Support Questions
- Cloudera Community
- Support
- Support Questions
- ALL hadoop-mapreduce-examples.jar fail cdh6
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
ALL hadoop-mapreduce-examples.jar fail cdh6
- Labels:
-
Apache YARN
-
Cloudera Manager
-
MapReduce
Created on
02-26-2020
09:14 AM
- last edited on
02-26-2020
01:26 PM
by
ask_bill_brooks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please help I've spent 3 days and still can't get an example to run including pi, terasort or even wordcount
New installation and have turned on HA for yarn and also hdfs, and installed the yarn MapReduce Framework JARs as well. Cluster looks health as seen below:
If I try and run any of the examples ...they fail.
[hdfs@cdh-utility-1 opc]$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4
WARNING: Use "yarn jar" to launch YARN applications.
Number of Maps = 4
Samples per Map example.com= 4
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Starting Job
20/02/26 17:09:47 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1582677468069_0009
20/02/26 17:09:47 INFO input.FileInputFormat: Total input files to process : 4
20/02/26 17:09:47 INFO mapreduce.JobSubmitter: number of splits:4
20/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.zk-address is deprecated. Instead, use hadoop.zk.address
20/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
20/02/26 17:09:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1582677468069_0009
20/02/26 17:09:48 INFO mapreduce.JobSubmitter: Executing with tokens: []
20/02/26 17:09:48 INFO conf.Configuration: resource-types.xml not found
20/02/26 17:09:48 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
20/02/26 17:09:48 INFO impl.YarnClientImpl: Submitted application application_1582677468069_0009
20/02/26 17:09:48 INFO mapreduce.Job: The url to track the job: http://cdh-master-1.private.cdhvcn.examplevcn.com:8088/proxy/application_1582677468069_0009/
20/02/26 17:09:48 INFO mapreduce.Job: Running job: job_1582677468069_0009
20/02/26 17:09:54 INFO mapreduce.Job: Job job_1582677468069_0009 running in uber mode : false
20/02/26 17:09:54 INFO mapreduce.Job: map 0% reduce 0%
20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000002_0, Status : FAILED
[2020-02-26 17:09:55.814]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000004
Exit code: 1
[2020-02-26 17:09:55.815]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT
[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT
20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000001_0, Status : FAILED
[2020-02-26 17:09:55.826]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000003
Exit code: 1
[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT
[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT
20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000000_0, Status : FAILED
[2020-02-26 17:09:55.813]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000002
Exit code: 1
[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT
[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT
20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000003_0, Status : FAILED
[2020-02-26 17:09:55.819]Exception from container-launch.
Container id: container_e02_1582677468069_0009_01_000005
Exit code: 1
[2020-02-26 17:09:55.852]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT
[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class NOT
Created 03-03-2020 08:12 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
That's great! You should be able to replace the "NOT FOUND" values for those two fields with:
-Djava.net.preferIPv4Stack=trueThis will configure it as CM usually has by default.
Not sure how the NOT FOUND ended up there.
Created 02-26-2020 02:43 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @HadoopBD ,
Sorry to hear this has been causing you trouble.
I'll see if I can help with the investigation.
A few things you mention that are relevant:
- All examples fail, so that supports the assumption that it is something the jobs have in common that is contributing this issue.
- It also appears that the issue happen during container launch (as prelaunch.err contains the error)
- The failure indicates that somehow the class name was erroneously evaluated to the string "NOT" as seen here:
Error: Could not find or load main class NOT
Since there is no class named "NOT" that implies whatever evaluation was done to attempt to execute that class was incorrect. For instance, you can get the same result if you run the following:
> java NOT
Error: Could not find or load main class NOT
So, the question goes to what happened before an attempt was made to launch a container.
One thing that can be an influence is the environment (env variables). If this is an out-of-the-box installation, we would not expect this to happen, so if you have updated any YARN or HDFS configuration, it would be good note.
We could use a bit more information, so I would suggest getting the logs for the application like this:
# yarn logs -applicationId application_1582677468069_0009 > application_1582677468069_0009.log
This should allow you to look at all the logs for this application including any information that may have been missing from the job stdout.
Also consider trying to run the job from the same host as the Resource Manager to see if the failure is the same. If there is something related to your client environment or hadoop configuration, the test may highlight that type of issue.
Cheers,
Ben
Created 02-27-2020 12:26 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I went ahead and tried on the RM server as well and same darn error. I just don't get it.
As you had asked for the log . Here it is:
296 java.io.tmpdir: /data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001/tmp
297 user.dir: /data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001
298 user.name: yarn
299 ************************************************************/
300 2020-02-26 17:09:49,953 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration
301 2020-02-26 17:09:50,097 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 9 cluster_timestamp: 1582677468069 } attemptId: 1 } keyId: -1009427054)]
302 2020-02-26 17:09:50,128 INFO [main] org.apache.hadoop.conf.Configuration: resource-types.xml not found
303 2020-02-26 17:09:50,128 INFO [main] org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'.
304 2020-02-26 17:09:50,136 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
305 2020-02-26 17:09:50,138 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
306 2020-02-26 17:09:50,173 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2
307 2020-02-26 17:09:50,173 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
308 2020-02-26 17:09:50,627 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
309 2020-02-26 17:09:50,662 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
310 2020-02-26 17:09:50,830 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
311 2020-02-26 17:09:50,831 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
312 2020-02-26 17:09:50,831 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
313 2020-02-26 17:09:50,832 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
314 2020-02-26 17:09:50,832 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
315 2020-02-26 17:09:50,833 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
316 2020-02-26 17:09:50,834 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
317 2020-02-26 17:09:50,834 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
318 2020-02-26 17:09:50,864 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
319 2020-02-26 17:09:50,881 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
320 2020-02-26 17:09:50,895 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
321 2020-02-26 17:09:50,909 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
322 2020-02-26 17:09:50,946 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
323 2020-02-26 17:09:51,187 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties
324 2020-02-26 17:09:51,244 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
325 2020-02-26 17:09:51,244 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
326 2020-02-26 17:09:51,252 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1582677468069_0009 to jobTokenSecretManager
327 2020-02-26 17:09:51,358 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1582677468069_0009 because: not enabled;
328 2020-02-26 17:09:51,374 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1582677468069_0009 = 472. Number of splits = 4
329 2020-02-26 17:09:51,375 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1582677468069_0009 = 1
330 2020-02-26 17:09:51,375 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from NEW to INITED
331 2020-02-26 17:09:51,376 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1582677468069_0009.
332 2020-02-26 17:09:51,399 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
333 2020-02-26 17:09:51,408 INFO [Socket Reader #1 for port 46625] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 46625
334 2020-02-26 17:09:51,577 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
335 2020-02-26 17:09:51,578 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
336 2020-02-26 17:09:51,578 INFO [IPC Server listener on 46625] org.apache.hadoop.ipc.Server: IPC Server listener on 46625: starting
337 2020-02-26 17:09:51,579 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at cdh-worker-3.private.cdhvcn.hiddenvcn.com/10.161.2.5:46625
338 2020-02-26 17:09:51,609 INFO [main] org.eclipse.jetty.util.log: Logging initialized @2635ms
339 2020-02-26 17:09:51,697 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
340 2020-02-26 17:09:51,700 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
341 2020-02-26 17:09:51,705 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
342 2020-02-26 17:09:51,747 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
343 2020-02-26 17:09:51,747 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
344 2020-02-26 17:09:51,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
345 2020-02-26 17:09:51,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
346 2020-02-26 17:09:52,083 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
347 2020-02-26 17:09:52,084 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 45277
348 2020-02-26 17:09:52,085 INFO [main] org.eclipse.jetty.server.Server: jetty-9.3.20.v20170531
349 2020-02-26 17:09:52,122 INFO [main] org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@39109136{/static,jar:file:/data4/yarn/nm/filecache/12/3.0.0-cdh6.1.1-mr-framework.tar.gz/hadoop-yarn-common-3.0.0-cdh6.1.1.jar!/webapps/static,AVAILABLE}
350 2020-02-26 17:09:53,009 INFO [main] org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@1f939a0f{/,file:///data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001/tmp/jetty-0.0.0.0-45277-mapreduce-_-any-7944828710115519068.dir/webapp/,AVAILABLE}{/mapreduce}
351 2020-02-26 17:09:53,017 INFO [main] org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@7112ce6{HTTP/1.1,[http/1.1]}{0.0.0.0:45277}
352 2020-02-26 17:09:53,017 INFO [main] org.eclipse.jetty.server.Server: Started @4043ms
353 2020-02-26 17:09:53,017 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app mapreduce started at 45277
354 2020-02-26 17:09:53,022 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
355 2020-02-26 17:09:53,023 INFO [Socket Reader #1 for port 34533] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34533
356 2020-02-26 17:09:53,028 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
357 2020-02-26 17:09:53,028 INFO [IPC Server listener on 34533] org.apache.hadoop.ipc.Server: IPC Server listener on 34533: starting
358 2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
359 2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
360 2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
361 2020-02-26 17:09:53,060 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the mappers will be scheduled using OPPORTUNISTIC containers
362 2020-02-26 17:09:53,154 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: maxContainerCapability: <memory:8192, vCores:2>
363 2020-02-26 17:09:53,154 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: queue: root.users.hdfs
364 2020-02-26 17:09:53,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
365 2020-02-26 17:09:53,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
366 2020-02-26 17:09:53,169 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from INITED to SETUP
367 2020-02-26 17:09:53,171 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
368 2020-02-26 17:09:53,182 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from SETUP to RUNNING
369 2020-02-26 17:09:53,274 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1582677468069_0009, File: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist
370 2020-02-26 17:09:53,288 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from NEW to SCHEDULED
371 2020-02-26 17:09:53,311 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from NEW to SCHEDULED
372 2020-02-26 17:09:53,313 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from NEW to SCHEDULED
373 2020-02-26 17:09:53,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from NEW to SCHEDULED
374 2020-02-26 17:09:53,316 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from NEW to SCHEDULED
375 2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
376 2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED
377 2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED
378 2020-02-26 17:09:53,318 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED
379 2020-02-26 17:09:53,318 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
380 2020-02-26 17:09:53,319 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>
381 2020-02-26 17:09:53,332 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:1024, vCores:1>
382 2020-02-26 17:09:54,158 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
383 2020-02-26 17:09:54,189 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=6 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:454656, vCores:63> knownNMs=4
384 2020-02-26 17:09:54,192 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:454656, vCores:63>
385 2020-02-26 17:09:54,192 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
386 2020-02-26 17:09:55,206 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
387 2020-02-26 17:09:55,209 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000002 to attempt_1582677468069_0009_m_000000_0
388 2020-02-26 17:09:55,211 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000003 to attempt_1582677468069_0009_m_000001_0
389 2020-02-26 17:09:55,212 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000004 to attempt_1582677468069_0009_m_000002_0
390 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000005 to attempt_1582677468069_0009_m_000003_0
391 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:450560, vCores:59>
392 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
393 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1
394 2020-02-26 17:09:55,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://nameservice1/user/hdfs/.staging/job_1582677468069_0009/job.jar
395 2020-02-26 17:09:55,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hdfs/.staging/job_1582677468069_0009/job.xml
396 2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container
397 2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1
398 2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
399 2020-02-26 17:09:55,308 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
400 2020-02-26 17:09:55,311 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
401 2020-02-26 17:09:55,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
402 2020-02-26 17:09:55,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
403 2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
404 2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
405 2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
406 2020-02-26 17:09:55,316 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
407 2020-02-26 17:09:55,318 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000002 taskAttempt attempt_1582677468069_0009_m_000000_0
408 2020-02-26 17:09:55,318 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000003 taskAttempt attempt_1582677468069_0009_m_000001_0
409 2020-02-26 17:09:55,318 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000004 taskAttempt attempt_1582677468069_0009_m_000002_0
410 2020-02-26 17:09:55,319 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000005 taskAttempt attempt_1582677468069_0009_m_000003_0
411 2020-02-26 17:09:55,321 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_0
412 2020-02-26 17:09:55,321 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_0
413 2020-02-26 17:09:55,321 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_0
414 2020-02-26 17:09:55,321 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_0
415 2020-02-26 17:09:55,404 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_0 : 13562
416 2020-02-26 17:09:55,405 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_0 : 13562
417 2020-02-26 17:09:55,405 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_0] using containerId: [container_e02_1582677468069_0009_01_000005 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
418 2020-02-26 17:09:55,408 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_0 : 13562
419 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
420 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_0] using containerId: [container_e02_1582677468069_0009_01_000003 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
421 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
422 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_0] using containerId: [container_e02_1582677468069_0009_01_000002 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
423 2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
424 2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from SCHEDULED to RUNNING
425 2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from SCHEDULED to RUNNING
426 2020-02-26 17:09:55,411 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from SCHEDULED to RUNNING
427 2020-02-26 17:09:55,412 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_0 : 13562
428 2020-02-26 17:09:55,412 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_0] using containerId: [container_e02_1582677468069_0009_01_000004 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
429 2020-02-26 17:09:55,413 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
430 2020-02-26 17:09:55,413 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from SCHEDULED to RUNNING
431 2020-02-26 17:09:56,221 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=6 release= 0 newContainers=0 finishedContainers=4 resourcelimit=<memory:454656, vCores:63> knownNMs=4
432 2020-02-26 17:09:56,221 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000004
433 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000003
434 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000002
435 2020-02-26 17:09:56,222 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000002_0: [2020-02-26 17:09:55.814]Exception from container-launch.
436 Container id: container_e02_1582677468069_0009_01_000004
437 Exit code: 1
438 [2020-02-26 17:09:55.815]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
439 Last 4096 bytes of prelaunch.err :
440 Last 4096 bytes of stderr :
441 Error: Could not find or load main class NOT
442 [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
443 Last 4096 bytes of prelaunch.err :
444 Last 4096 bytes of stderr :
445 Error: Could not find or load main class NOT
446 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000005
447 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:454656, vCores:63>
448 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
449 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1
450 2020-02-26 17:09:56,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
451 2020-02-26 17:09:56,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_0: [2020-02-26 17:09:55.826]Exception from container-launch.
452 Container id: container_e02_1582677468069_0009_01_000003
453 Exit code: 1
454 [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
455 Last 4096 bytes of prelaunch.err :
456 Last 4096 bytes of stderr :
457 Error: Could not find or load main class NOT
458 [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
459 Last 4096 bytes of prelaunch.err :
460 Last 4096 bytes of stderr :
461 Error: Could not find or load main class NOT
462 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
463 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_0: [2020-02-26 17:09:55.813]Exception from container-launch.
464 Container id: container_e02_1582677468069_0009_01_000002
465 Exit code: 1
466 [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
467 Last 4096 bytes of prelaunch.err :
468 Last 4096 bytes of stderr :
469 Error: Could not find or load main class NOT
470 [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
471 Last 4096 bytes of prelaunch.err :
472 Last 4096 bytes of stderr :
473 Error: Could not find or load main class NOT
474 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
475 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_0: [2020-02-26 17:09:55.819]Exception from container-launch.
476 Container id: container_e02_1582677468069_0009_01_000005
477 Exit code: 1
478 [2020-02-26 17:09:55.852]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
479 Last 4096 bytes of prelaunch.err :
480 Last 4096 bytes of stderr :
481 Error: Could not find or load main class NOT
482 [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
483 Last 4096 bytes of prelaunch.err :
484 Last 4096 bytes of stderr :
485 Error: Could not find or load main class NOT
486 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
487 2020-02-26 17:09:56,228 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000004 taskAttempt attempt_1582677468069_0009_m_000002_0
488 2020-02-26 17:09:56,237 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000003 taskAttempt attempt_1582677468069_0009_m_000001_0
489 2020-02-26 17:09:56,240 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000002 taskAttempt attempt_1582677468069_0009_m_000000_0
490 2020-02-26 17:09:56,241 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000005 taskAttempt attempt_1582677468069_0009_m_000003_0
491 2020-02-26 17:09:56,243 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com
492 2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from NEW to UNASSIGNED
493 2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from NEW to UNASSIGNED
494 2020-02-26 17:09:56,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com
495 2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
496 2020-02-26 17:09:56,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com
497 2020-02-26 17:09:56,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted host cdh-worker-1.private.cdhvcn.hiddenvcn.com
498 2020-02-26 17:09:56,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from NEW to UNASSIGNED
499 2020-02-26 17:09:56,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_1 to list of failed maps
500 2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000001_1 to list of failed maps
501 2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000000_1 to list of failed maps
502 2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000003_1 to list of failed maps
503 2020-02-26 17:09:57,223 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1
504 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:339968, vCores:47> knownNMs=4
505 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=1 blacklistRemovals=0
506 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:339968, vCores:47>
507 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
508 2020-02-26 17:09:58,231 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
509 2020-02-26 17:09:58,232 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000006, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
510 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
511 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000006 to attempt_1582677468069_0009_m_000002_1
512 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000007, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
513 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
514 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000007 to attempt_1582677468069_0009_m_000001_1
515 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000008, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
516 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
517 2020-02-26 17:09:58,233 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
518 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000008 to attempt_1582677468069_0009_m_000000_1
519 2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000009, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
520 2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
521 2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000009 to attempt_1582677468069_0009_m_000003_1
522 2020-02-26 17:09:58,234 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
523 2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:335872, vCores:43>
524 2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
525 2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
526 2020-02-26 17:09:58,235 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
527 2020-02-26 17:09:58,236 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
528 2020-02-26 17:09:58,236 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
529 2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
530 2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
531 2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
532 2020-02-26 17:09:58,238 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000006 taskAttempt attempt_1582677468069_0009_m_000002_1
533 2020-02-26 17:09:58,238 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000009 taskAttempt attempt_1582677468069_0009_m_000003_1
534 2020-02-26 17:09:58,238 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_1
535 2020-02-26 17:09:58,238 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000007 taskAttempt attempt_1582677468069_0009_m_000001_1
536 2020-02-26 17:09:58,238 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000008 taskAttempt attempt_1582677468069_0009_m_000000_1
537 2020-02-26 17:09:58,238 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_1
538 2020-02-26 17:09:58,238 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_1
539 2020-02-26 17:09:58,238 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_1
540 2020-02-26 17:09:58,265 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_1 : 13562
541 2020-02-26 17:09:58,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_1] using containerId: [container_e02_1582677468069_0009_01_000009 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
542 2020-02-26 17:09:58,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
543 2020-02-26 17:09:58,273 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_1 : 13562
544 2020-02-26 17:09:58,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_1] using containerId: [container_e02_1582677468069_0009_01_000007 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
545 2020-02-26 17:09:58,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
546 2020-02-26 17:09:58,280 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_1 : 13562
547 2020-02-26 17:09:58,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_1] using containerId: [container_e02_1582677468069_0009_01_000006 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
548 2020-02-26 17:09:58,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
549 2020-02-26 17:09:58,289 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_1 : 13562
550 2020-02-26 17:09:58,289 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_1] using containerId: [container_e02_1582677468069_0009_01_000008 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
551 2020-02-26 17:09:58,289 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
552 2020-02-26 17:09:59,238 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=3 resourcelimit=<memory:338944, vCores:46> knownNMs=4
553 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000008
554 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000009
555 2020-02-26 17:09:59,239 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_1: [2020-02-26 17:09:58.706]Exception from container-launch.
556 Container id: container_e02_1582677468069_0009_01_000008
557 Exit code: 1
558 [2020-02-26 17:09:58.708]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
559 Last 4096 bytes of prelaunch.err :
560 Last 4096 bytes of stderr :
561 Error: Could not find or load main class NOT
562 [2020-02-26 17:09:58.708]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
563 Last 4096 bytes of prelaunch.err :
564 Last 4096 bytes of stderr :
565 Error: Could not find or load main class NOT
566 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000007
567 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:338944, vCores:46>
568 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
569 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
570 2020-02-26 17:09:59,239 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
571 2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_1: [2020-02-26 17:09:58.721]Exception from container-launch.
572 Container id: container_e02_1582677468069_0009_01_000009
573 Exit code: 1
574 [2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
575 Last 4096 bytes of prelaunch.err :
576 Last 4096 bytes of stderr :
577 Error: Could not find or load main class NOT
578 [2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
579 Last 4096 bytes of prelaunch.err :
580 Last 4096 bytes of stderr :
581 Error: Could not find or load main class NOT
582 2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
583 2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_1: [2020-02-26 17:09:58.712]Exception from container-launch.
584 Container id: container_e02_1582677468069_0009_01_000007
585 Exit code: 1
586 [2020-02-26 17:09:58.732]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
587 Last 4096 bytes of prelaunch.err :
588 Last 4096 bytes of stderr :
589 Error: Could not find or load main class NOT
590 [2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
591 Last 4096 bytes of prelaunch.err :
592 Last 4096 bytes of stderr :
593 Error: Could not find or load main class NOT
594 2020-02-26 17:09:59,241 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
595 2020-02-26 17:09:59,241 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000008 taskAttempt attempt_1582677468069_0009_m_000000_1
596 2020-02-26 17:09:59,242 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000009 taskAttempt attempt_1582677468069_0009_m_000003_1
597 2020-02-26 17:09:59,244 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000007 taskAttempt attempt_1582677468069_0009_m_000001_1
598 2020-02-26 17:09:59,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com
599 2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from NEW to UNASSIGNED
600 2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com
601 2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from NEW to UNASSIGNED
602 2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com
603 2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted host cdh-worker-3.private.cdhvcn.hiddenvcn.com
604 2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from NEW to UNASSIGNED
605 2020-02-26 17:09:59,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000000_2 to list of failed maps
606 2020-02-26 17:09:59,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000003_2 to list of failed maps
607 2020-02-26 17:09:59,248 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000001_2 to list of failed maps
608 2020-02-26 17:10:00,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
609 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:229376, vCores:32> knownNMs=4
610 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=1 blacklistRemovals=0
611 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Ignore blacklisting set to true. Known: 4, Blacklisted: 2, 50%
612 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000006
613 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:229376, vCores:32>
614 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
615 2020-02-26 17:10:00,243 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000002_1: [2020-02-26 17:09:58.725]Exception from container-launch.
616 Container id: container_e02_1582677468069_0009_01_000006
617 Exit code: 1
618 [2020-02-26 17:09:58.734]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
619 Last 4096 bytes of prelaunch.err :
620 Last 4096 bytes of stderr :
621 Error: Could not find or load main class NOT
622 [2020-02-26 17:09:58.734]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
623 Last 4096 bytes of prelaunch.err :
624 Last 4096 bytes of stderr :
625 Error: Could not find or load main class NOT
626 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
627 2020-02-26 17:10:00,244 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
628 2020-02-26 17:10:00,244 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000006 taskAttempt attempt_1582677468069_0009_m_000002_1
629 2020-02-26 17:10:00,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_2 TaskAttempt Transitioned from NEW to UNASSIGNED
630 2020-02-26 17:10:00,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_2 to list of failed maps
631 2020-02-26 17:10:01,244 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
632 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=3 finishedContainers=0 resourcelimit=<memory:451584, vCores:60> knownNMs=4
633 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=0 blacklistRemovals=2
634 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 3
635 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000010, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
636 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
637 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000010 to attempt_1582677468069_0009_m_000000_2
638 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000011, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
639 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
640 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000011 to attempt_1582677468069_0009_m_000003_2
641 2020-02-26 17:10:01,251 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
642 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000012, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
643 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
644 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000012 to attempt_1582677468069_0009_m_000001_2
645 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:451584, vCores:60>
646 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
647 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:11 ContRel:0 HostLocal:3 RackLocal:1
648 2020-02-26 17:10:01,252 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
649 2020-02-26 17:10:01,254 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
650 2020-02-26 17:10:01,254 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
651 2020-02-26 17:10:01,255 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
652 2020-02-26 17:10:01,255 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
653 2020-02-26 17:10:01,255 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000010 taskAttempt attempt_1582677468069_0009_m_000000_2
654 2020-02-26 17:10:01,255 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000011 taskAttempt attempt_1582677468069_0009_m_000003_2
655 2020-02-26 17:10:01,255 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000012 taskAttempt attempt_1582677468069_0009_m_000001_2
656 2020-02-26 17:10:01,256 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_2
657 2020-02-26 17:10:01,255 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_2
658 2020-02-26 17:10:01,256 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_2
659 2020-02-26 17:10:01,267 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_2 : 13562
660 2020-02-26 17:10:01,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_2] using containerId: [container_e02_1582677468069_0009_01_000011 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
661 2020-02-26 17:10:01,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
662 2020-02-26 17:10:01,271 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_2 : 13562
663 2020-02-26 17:10:01,271 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_2] using containerId: [container_e02_1582677468069_0009_01_000012 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
664 2020-02-26 17:10:01,271 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
665 2020-02-26 17:10:01,273 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_2 : 13562
666 2020-02-26 17:10:01,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_2] using containerId: [container_e02_1582677468069_0009_01_000010 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
667 2020-02-26 17:10:01,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
668 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=4 finishedContainers=3 resourcelimit=<memory:450560, vCores:59> knownNMs=4
669 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000011
670 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000012
671 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000010
672 2020-02-26 17:10:02,257 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_2: [2020-02-26 17:10:01.685]Exception from container-launch.
673 Container id: container_e02_1582677468069_0009_01_000011
674 Exit code: 1
675 [2020-02-26 17:10:01.686]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
676 Last 4096 bytes of prelaunch.err :
677 Last 4096 bytes of stderr :
678 Error: Could not find or load main class NOT
679 [2020-02-26 17:10:01.691]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
680 Last 4096 bytes of prelaunch.err :
681 Last 4096 bytes of stderr :
682 Error: Could not find or load main class NOT
683 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
684 2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041
685 2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_2: [2020-02-26 17:10:01.698]Exception from container-launch.
686 Container id: container_e02_1582677468069_0009_01_000012
687 Exit code: 1
688 [2020-02-26 17:10:01.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
689 Last 4096 bytes of prelaunch.err :
690 Last 4096 bytes of stderr :
691 Error: Could not find or load main class NOT
692 [2020-02-26 17:10:01.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
693 Last 4096 bytes of prelaunch.err :
694 Last 4096 bytes of stderr :
695 Error: Could not find or load main class NOT
696 2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000013, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
697 2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
698 2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041
699 2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000013 to attempt_1582677468069_0009_m_000002_2
700 2020-02-26 17:10:02,259 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_2: [2020-02-26 17:10:01.688]Exception from container-launch.
701 Container id: container_e02_1582677468069_0009_01_000010
702 Exit code: 1
Created 02-27-2020 02:55 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @HadoopBD ,
The logs provided don't contain the environment or any clues that would help us understand what may have been passed to the command that is attempting to launch a container. It might have been missed in my previous message, but a good way of getting some more detail about the Application Masters, containers, etc. would be to collect logs via the "yarn logs" command.
For instance:
yarn logs -applicationId application_1582677468069_0009 > application_1582677468069_0009.log
Resource Manager logs tell us some things, but not the whole picture. If you can run the above, the output may be pretty big, but if you can take a look and see if you can find the string "NOT" in there that might be a start.
If you are on Cloudera Manager 6.3 or higher, you can try the following to collect more information about the container launch:
(1)
Via Cloudera Manager, set the following configuration to 600 (10 minutes): Localized Dir Deletion Delay. This will tell the Node Manager to delay 10 minutes before cleaning up the container launcher. This will help us review the files used in the failed container launch
(2)
Set the following YARN configuration: Enable Container Launch Debug Information. Check the box to enable it. This will allow you to collect extra container launch information in the "yarn logs -applicationId" output.
(3)
SAVE your changes and then Restart YARN service from CM
(4)
Run a test mapreduce job (pi for instance)
(5)
After it fails, run the following to collect the aggregated logs for the job:
yarn logs -applicationId <app_id>
NOTE: you can direct the output to a file so you can search in the file.
(6)
Look for "launch_container" in the output to find the launch information.
Again, the output might be pretty big so you can try adding it here or look for things that may be relevant.
Created 02-27-2020 05:59 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If possible, if you can attach log as a file to this thread, it would be better, as the log messages make the page lengthy and hard to navigate around.
Cheers
Created 02-28-2020 08:10 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I don't see anything here at all for doing an attachment, just links 😞 😞 . so I'll apologize now. But, here is more of the log.
753 [2020-02-26 17:10:02.569]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
754 Last 4096 bytes of prelaunch.err :
755 Last 4096 bytes of stderr :
756 Error: Could not find or load main class NOT
757 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000015
758 2020-02-26 17:10:03,267 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container container_e02_1582677468069_0009_01_000015
759 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000016
760 2020-02-26 17:10:03,267 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container container_e02_1582677468069_0009_01_000016
761 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1
762 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000017, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
763 2020-02-26 17:10:03,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041
764 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
765 2020-02-26 17:10:03,267 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000013 taskAttempt attempt_1582677468069_0009_m_000002_2
766 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000017 to attempt_1582677468069_0009_m_000003_3
767 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:453632, vCores:62>
768 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
769 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:16 ContRel:3 HostLocal:3 RackLocal:1
770 2020-02-26 17:10:03,269 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
771 2020-02-26 17:10:03,270 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
772 2020-02-26 17:10:03,270 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from NEW to UNASSIGNED
773 2020-02-26 17:10:03,270 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000017 taskAttempt attempt_1582677468069_0009_m_000003_3
774 2020-02-26 17:10:03,270 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_3 to list of failed maps
775 2020-02-26 17:10:03,270 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_3
776 2020-02-26 17:10:03,281 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_3 : 13562
777 2020-02-26 17:10:03,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_3] using containerId: [container_e02_1582677468069_0009_01_000017 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
778 2020-02-26 17:10:03,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 TaskAttempt Transitioned from ASSIGNED to RUNNING
779 2020-02-26 17:10:04,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:16 ContRel:3 HostLocal:3 RackLocal:1
780 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=3 finishedContainers=1 resourcelimit=<memory:451584, vCores:60> knownNMs=4
781 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000017
782 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 3
783 2020-02-26 17:10:04,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_3: [2020-02-26 17:10:03.568]Exception from container-launch.
784 Container id: container_e02_1582677468069_0009_01_000017
785 Exit code: 1
786 [2020-02-26 17:10:03.570]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
787 Last 4096 bytes of prelaunch.err :
788 Last 4096 bytes of stderr :
789 Error: Could not find or load main class NOT
790 [2020-02-26 17:10:03.570]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
791 Last 4096 bytes of prelaunch.err :
792 Last 4096 bytes of stderr :
793 Error: Could not find or load main class NOT
794 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000018, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
795 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
796 2020-02-26 17:10:04,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041
797 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000018 to attempt_1582677468069_0009_m_000001_3
798 2020-02-26 17:10:04,274 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000017 taskAttempt attempt_1582677468069_0009_m_000003_3
799 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000019, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
800 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
801 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000019 to attempt_1582677468069_0009_m_000000_3
802 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000020, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
803 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
804 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000020 to attempt_1582677468069_0009_m_000002_3
805 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:451584, vCores:60>
806 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
807 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:19 ContRel:3 HostLocal:3 RackLocal:1
808 2020-02-26 17:10:04,276 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from RUNNING to FAILED
809 2020-02-26 17:10:04,277 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
810 2020-02-26 17:10:04,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
811 2020-02-26 17:10:04,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
812 2020-02-26 17:10:04,279 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
813 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
814 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
815 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1
816 2020-02-26 17:10:04,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
817 2020-02-26 17:10:04,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from RUNNING to FAIL_WAIT
818 2020-02-26 17:10:04,282 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000018 taskAttempt attempt_1582677468069_0009_m_000001_3
819 2020-02-26 17:10:04,282 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000019 taskAttempt attempt_1582677468069_0009_m_000000_3
820 2020-02-26 17:10:04,282 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_3
821 2020-02-26 17:10:04,283 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_3
822 2020-02-26 17:10:04,283 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000020 taskAttempt attempt_1582677468069_0009_m_000002_3
823 2020-02-26 17:10:04,283 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_3
824 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from RUNNING to KILL_WAIT
825 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from RUNNING to KILL_WAIT
826 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from RUNNING to KILL_WAIT
827 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from SCHEDULED to KILL_WAIT
828 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP
829 2020-02-26 17:10:04,284 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP
830 2020-02-26 17:10:04,284 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP
831 2020-02-26 17:10:04,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_r_000000_0 TaskAttempt Transitioned from UNASSIGNED to KILLED
832 2020-02-26 17:10:04,285 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE
833 2020-02-26 17:10:04,285 ERROR [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Could not deallocate container for task attemptId attempt_1582677468069_0009_r_000000_0
834 2020-02-26 17:10:04,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from KILL_WAIT to KILLED
835 2020-02-26 17:10:04,286 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000019 taskAttempt attempt_1582677468069_0009_m_000000_3
836 2020-02-26 17:10:04,286 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000018 taskAttempt attempt_1582677468069_0009_m_000001_3
837 2020-02-26 17:10:04,286 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000020 taskAttempt attempt_1582677468069_0009_m_000002_3
838 2020-02-26 17:10:04,297 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_3 : 13562
839 2020-02-26 17:10:04,298 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000001_3
840 2020-02-26 17:10:04,305 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_3 : 13562
841 2020-02-26 17:10:04,306 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000002_3
842 2020-02-26 17:10:04,310 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_3 : 13562
843 2020-02-26 17:10:04,310 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000000_3
844 2020-02-26 17:10:04,324 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
845 2020-02-26 17:10:04,324 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
846 2020-02-26 17:10:04,325 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
847 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
848 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
849 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
850 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000000_3
851 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000001_3
852 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000002_3
853 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_3] using containerId: [container_e02_1582677468069_0009_01_000019 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
854 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED
855 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_3] using containerId: [container_e02_1582677468069_0009_01_000018 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
856 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED
857 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_3] using containerId: [container_e02_1582677468069_0009_01_000020 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
858 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED
859 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from KILL_WAIT to KILLED
860 2020-02-26 17:10:04,333 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from KILL_WAIT to KILLED
861 2020-02-26 17:10:04,333 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from KILL_WAIT to KILLED
862 2020-02-26 17:10:04,334 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from FAIL_WAIT to FAIL_ABORT
863 2020-02-26 17:10:04,334 INFO [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT
864 2020-02-26 17:10:04,344 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from FAIL_ABORT to FAILED
865 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly, recording last MRAppMaster retry
866 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true
867 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified that shouldUnregistered is: true
868 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true
869 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true
870 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services
871 2020-02-26 17:10:04,346 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 1
872 2020-02-26 17:10:04,347 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_FAILED
873 2020-02-26 17:10:04,389 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp
874 2020-02-26 17:10:04,424 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied from: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist to done location: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp
875 2020-02-26 17:10:04,427 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Set historyUrl to http://cdh-utility-1.public.cdhvcn.hiddenvcn.com:19888/jobhistory/job/job_1582677468069_0009
876 2020-02-26 17:10:04,428 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1_conf.xml to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp
877 2020-02-26 17:10:04,460 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied from: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1_conf.xml to done location: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp
878 2020-02-26 17:10:04,466 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009.summary_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009.summary
879 2020-02-26 17:10:04,469 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml
880 2020-02-26 17:10:04,471 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist
881 2020-02-26 17:10:04,471 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop()
882 2020-02-26 17:10:04,476 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Task failed task_1582677468069_0009_m_000003
883 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
884 2020-02-26 17:10:04,476 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: History url is http://cdh-utility-1.public.cdhvcn.hiddenvcn.com:19888/jobhistory/job/job_1582677468069_0009
885 2020-02-26 17:10:04,481 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Waiting for application to be successfully unregistered.
886 2020-02-26 17:10:05,482 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:19 ContRel:3 HostLocal:3 RackLocal:1
887 2020-02-26 17:10:05,483 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://nameservice1 /user/hdfs/.staging/job_1582677468069_0009
888 2020-02-26 17:10:05,487 INFO [Thread-90] org.apache.hadoop.ipc.Server: Stopping server on 34533
889 2020-02-26 17:10:05,488 INFO [IPC Server listener on 34533] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 34533
890 2020-02-26 17:10:05,488 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted
891 2020-02-26 17:10:05,490 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
892 2020-02-26 17:10:05,490 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted
893 2020-02-26 17:10:10,491 INFO [Thread-90] org.apache.hadoop.ipc.Server: Stopping server on 46625
894 2020-02-26 17:10:10,491 INFO [IPC Server listener on 46625] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 46625
895 2020-02-26 17:10:10,491 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
896 2020-02-26 17:10:10,494 INFO [Thread-90] org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.w.WebAppContext@1f939a0f{/,null,UNAVAILABLE}{/mapreduce}
897 2020-02-26 17:10:10,497 INFO [Thread-90] org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@7112ce6{HTTP/1.1,[http/1.1]}{0.0.0.0:0}
898 2020-02-26 17:10:10,498 INFO [Thread-90] org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@39109136{/static,jar:file:/data4/yarn/nm/filecache/12/3.0.0-cdh6.1.1-mr-framework.tar.gz/hadoop-yarn-common-3.0.0-cdh6.1.1.jar!/webapps/static,UNAVAILABLE}
899 End of LogType:syslog
900 ***********************************************************************
Created 02-28-2020 09:40 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @HadoopBD ,
It appears to me that the log you provided may not have been captured with the steps I suggested before EricL's comment. Can you confirm how you retrieved these logs. From what I see, this only hints that there are problems launching containers, but in order to try to see why, we will need to capture more information.
Thanks,
Ben
Created 02-28-2020 07:56 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The log you provided is RM log, not application log that Ben asked earlier. Please run below command:
yarn logs -applicationId application_1582677468069_0009 > application_1582677468069_0009.log
And then find the snippet that has the error, as the log will likely to be very big.
I also noticed that the log you provided earlier contained your domain name, if you have concerns, we can redact them for you. Please advise.
Thanks
Eric
Created 03-01-2020 10:33 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The snippets from the log that I uploaded were actually from the output of the yarn log command as both of you had requested. If there is a way I can upload the full log I'd be more than happy to. As for remove of the domain name I actually change them before I submit so they are not feasible anyway 🙂 🙂
Created 03-02-2020 10:23 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @HadoopBD ,
Thanks for providing the logs, but they do not contain what we would expect if you had followed the steps to enabled container launch debug information. I am guessing you missed my steps during the threaded conversation.
Basically, the standard logs show you some information, but not all. We are missing the actual files and log information about how the "launch_container" processes was started and what was passed to the script use to execute the necessary java. In order to capture that information, which will most likely give us some sort of clue about the cause of this issue.
The steps to retain container launching information and also allow "yarn logs" command to obtain them is in CM 6.3 so I wanted to find out if you had that version.
Here are the steps:
If you are on Cloudera Manager 6.3 or higher, you can try the following to collect more information about the container launch:
(1)
Via Cloudera Manager, set the following configuration to 600 (10 minutes): Localized Dir Deletion Delay. This will tell the Node Manager to delay 10 minutes before cleaning up the container launcher. This will help us review the files used in the failed container launch
(2)
Set the following YARN configuration: Enable Container Launch Debug Information. Check the box to enable it. This will allow you to collect extra container launch information in the "yarn logs -applicationId" output.
(3)
SAVE your changes and then Restart YARN service from CM
(4)
Run a test mapreduce job (pi for instance)
(5)
After it fails, run the following to collect the aggregated logs for the job:
yarn logs -applicationId <app_id>
NOTE: you can direct the output to a file so you can search in the file.
(6)
Look for "launch_container" in the output to find the launch information.
I just ran through a test and a lot more details about how the command will be launched is available. I truly believe it will help us assess a cause so we can find a solution.
