Cloudera Community

Announcements
Celebrating as our community reaches 100,000 members! Thank you!
HadoopBD
user rank
Explorer

Adding a javaagent in CDH 5 or 6.2

by Explorer in Support Questions
‎10-14-2020 02:20 PM
‎10-14-2020 02:20 PM
We are trying to pull metrics into Promtheus and for the life of me I spent most of lastnight trying to get CDH 6.2 to work with a javaagent.  I read a previous post where someone was have a heck of a time with jmx exporter and Ambari, so I'm assuming this is also an issue in CDH?   If I goto: HBase Service Environment Advanced Configuration Snippet (Safety Valve) and add this: HBASE_MASTER_OPTS=-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=10110 -javaagent:/run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar=9214:/run/cloudera-scm-agent/hbasemaster.yml" HBASE_REGIONSERVER_OPTS=-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=10111 -javaagent:/run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar=9215:/run/cloudera-scm-agent/regionserver.yml"   and then restart the service The service WILL not restart and stderr shows: Error opening zip file or JAR manifest missing : /run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar I've verified all files are there : -rw-rw-rw- 1 root root 405K Sep 4 12:20 jmx_prometheus_javaagent-0.14.0.jar -rwxr-xr-x 1 root root 283 Oct 14 02:04 namenode.jml -rwxr-xr-x 1 root root 283 Oct 14 02:04 datanode.jml -rwxr-xr-x 1 root root 283 Oct 14 02:04 regionserver.jml -rwxr-xr-x 1 root root 283 Oct 14 06:17 hbasemaster.jml -rwxr-xr-x 1 root root 283 Oct 14 06:57 yarn.jml I've tried to modify the hbase-env.sh directly, but of course there are so many and that I can't seem to get it to work.   Has anyone got this working in CDH5 or CDH6 and if so, can point me in the right direction ... View more

Re: ALL hadoop-mapreduce-examples.jar fail cdh6

by Explorer in Support Questions
‎03-03-2020 12:01 PM
‎03-03-2020 12:01 PM
yep already did and even looked for a tuning parameter based on the cloud shape I'm using and so ended up setting both of them to: -Djava.net.preferIPv4Stack=true -Xms2368m -Xmx2368m and now pi works.  Thanks a bunch.       ... View more

Re: ALL hadoop-mapreduce-examples.jar fail cdh6

by Explorer in Support Questions
‎03-02-2020 08:53 PM
‎03-02-2020 08:53 PM
@bgooley  You might be onto something there. Since I was not running 6.3...I tore down the cluster and rebuilt from scratch this time using CDH 6.3.2, enabled HA for yarn and hdfs, deployed yarn framework, created job history directory and created nodemanager application log directory, then set  Localized Dir Deletion Delay to 1800 and Enable Container Launch Debug Information..  Then ran this:     sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4   then pulled the logs and looked for the bash line  and here is what I found: echo "ls -l:" 1>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" ls -l 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" find -L . -maxdepth 5 -ls 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" echo "Launching container" exec /bin/bash -c "$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN NOT FOUND -Xmx820m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.0.2.7 33307 attempt_1583199702433_0138_m_000002_3 3298534883342 1>/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/stdout 2>/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/stderr " End of LogType:launch_container.sh ************************************************************************************   so I then went into Yarn and looked for Map Task Java Opts Base and found:   BINGO.  You hit the nail on the head.  Great find.   ... View more

Re: ALL hadoop-mapreduce-examples.jar fail cdh6

by Explorer in Support Questions
‎03-01-2020 10:33 AM
‎03-01-2020 10:33 AM
The snippets from the log that I uploaded were actually from the output of the yarn log command as both of you had requested.  If there is a way I can upload the full log I'd be more than happy to.  As for remove of the domain name I actually change them before I submit so they are not feasible anyway 🙂 🙂      ... View more

Re: ALL hadoop-mapreduce-examples.jar fail cdh6

by Explorer in Support Questions
‎02-28-2020 08:10 AM
‎02-28-2020 08:10 AM
I don't see anything here at all for doing an attachment, just links 😞  😞 .  so I'll apologize now.  But, here is more of the log.   753 [2020-02-26 17:10:02.569]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 754 Last 4096 bytes of prelaunch.err : 755 Last 4096 bytes of stderr : 756 Error: Could not find or load main class NOT 757 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000015 758 2020-02-26 17:10:03,267 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container container_e02_1582677468069_0009_01_000015 759 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000016 760 2020-02-26 17:10:03,267 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container container_e02_1582677468069_0009_01_000016 761 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 762 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000017, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 763 2020-02-26 17:10:03,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041 764 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 765 2020-02-26 17:10:03,267 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000013 taskAttempt attempt_1582677468069_0009_m_000002_2 766 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000017 to attempt_1582677468069_0009_m_000003_3 767 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:453632, vCores:62> 768 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 769 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:16 ContRel:3 HostLocal:3 RackLocal:1 770 2020-02-26 17:10:03,269 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 771 2020-02-26 17:10:03,270 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 772 2020-02-26 17:10:03,270 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from NEW to UNASSIGNED 773 2020-02-26 17:10:03,270 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000017 taskAttempt attempt_1582677468069_0009_m_000003_3 774 2020-02-26 17:10:03,270 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_3 to list of failed maps 775 2020-02-26 17:10:03,270 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_3 776 2020-02-26 17:10:03,281 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_3 : 13562 777 2020-02-26 17:10:03,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_3] using containerId: [container_e02_1582677468069_0009_01_000017 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 778 2020-02-26 17:10:03,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 TaskAttempt Transitioned from ASSIGNED to RUNNING 779 2020-02-26 17:10:04,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:16 ContRel:3 HostLocal:3 RackLocal:1 780 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=3 finishedContainers=1 resourcelimit=<memory:451584, vCores:60> knownNMs=4 781 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000017 782 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 3 783 2020-02-26 17:10:04,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_3: [2020-02-26 17:10:03.568]Exception from container-launch. 784 Container id: container_e02_1582677468069_0009_01_000017 785 Exit code: 1 786 [2020-02-26 17:10:03.570]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 787 Last 4096 bytes of prelaunch.err : 788 Last 4096 bytes of stderr : 789 Error: Could not find or load main class NOT 790 [2020-02-26 17:10:03.570]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 791 Last 4096 bytes of prelaunch.err : 792 Last 4096 bytes of stderr : 793 Error: Could not find or load main class NOT 794 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000018, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 795 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 796 2020-02-26 17:10:04,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041 797 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000018 to attempt_1582677468069_0009_m_000001_3 798 2020-02-26 17:10:04,274 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000017 taskAttempt attempt_1582677468069_0009_m_000003_3 799 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000019, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 800 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 801 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000019 to attempt_1582677468069_0009_m_000000_3 802 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000020, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 803 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 804 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000020 to attempt_1582677468069_0009_m_000002_3 805 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:451584, vCores:60> 806 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 807 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:19 ContRel:3 HostLocal:3 RackLocal:1 808 2020-02-26 17:10:04,276 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from RUNNING to FAILED 809 2020-02-26 17:10:04,277 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 810 2020-02-26 17:10:04,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 811 2020-02-26 17:10:04,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 812 2020-02-26 17:10:04,279 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 813 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 814 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 815 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1 816 2020-02-26 17:10:04,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 817 2020-02-26 17:10:04,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from RUNNING to FAIL_WAIT 818 2020-02-26 17:10:04,282 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000018 taskAttempt attempt_1582677468069_0009_m_000001_3 819 2020-02-26 17:10:04,282 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000019 taskAttempt attempt_1582677468069_0009_m_000000_3 820 2020-02-26 17:10:04,282 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_3 821 2020-02-26 17:10:04,283 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_3 822 2020-02-26 17:10:04,283 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000020 taskAttempt attempt_1582677468069_0009_m_000002_3 823 2020-02-26 17:10:04,283 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_3 824 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from RUNNING to KILL_WAIT 825 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from RUNNING to KILL_WAIT 826 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from RUNNING to KILL_WAIT 827 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from SCHEDULED to KILL_WAIT 828 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP 829 2020-02-26 17:10:04,284 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP 830 2020-02-26 17:10:04,284 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP 831 2020-02-26 17:10:04,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_r_000000_0 TaskAttempt Transitioned from UNASSIGNED to KILLED 832 2020-02-26 17:10:04,285 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE 833 2020-02-26 17:10:04,285 ERROR [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Could not deallocate container for task attemptId attempt_1582677468069_0009_r_000000_0 834 2020-02-26 17:10:04,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from KILL_WAIT to KILLED 835 2020-02-26 17:10:04,286 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000019 taskAttempt attempt_1582677468069_0009_m_000000_3 836 2020-02-26 17:10:04,286 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000018 taskAttempt attempt_1582677468069_0009_m_000001_3 837 2020-02-26 17:10:04,286 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000020 taskAttempt attempt_1582677468069_0009_m_000002_3 838 2020-02-26 17:10:04,297 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_3 : 13562 839 2020-02-26 17:10:04,298 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000001_3 840 2020-02-26 17:10:04,305 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_3 : 13562 841 2020-02-26 17:10:04,306 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000002_3 842 2020-02-26 17:10:04,310 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_3 : 13562 843 2020-02-26 17:10:04,310 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000000_3 844 2020-02-26 17:10:04,324 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 845 2020-02-26 17:10:04,324 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 846 2020-02-26 17:10:04,325 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 847 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 848 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 849 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 850 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000000_3 851 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000001_3 852 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000002_3 853 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_3] using containerId: [container_e02_1582677468069_0009_01_000019 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 854 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 855 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_3] using containerId: [container_e02_1582677468069_0009_01_000018 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 856 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 857 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_3] using containerId: [container_e02_1582677468069_0009_01_000020 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 858 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 859 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from KILL_WAIT to KILLED 860 2020-02-26 17:10:04,333 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from KILL_WAIT to KILLED 861 2020-02-26 17:10:04,333 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from KILL_WAIT to KILLED 862 2020-02-26 17:10:04,334 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from FAIL_WAIT to FAIL_ABORT 863 2020-02-26 17:10:04,334 INFO [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT 864 2020-02-26 17:10:04,344 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from FAIL_ABORT to FAILED 865 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly, recording last MRAppMaster retry 866 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true 867 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified that shouldUnregistered is: true 868 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true 869 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true 870 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services 871 2020-02-26 17:10:04,346 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 1 872 2020-02-26 17:10:04,347 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_FAILED 873 2020-02-26 17:10:04,389 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp 874 2020-02-26 17:10:04,424 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied from: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist to done location: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp 875 2020-02-26 17:10:04,427 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Set historyUrl to http://cdh-utility-1.public.cdhvcn.hiddenvcn.com:19888/jobhistory/job/job_1582677468069_0009 876 2020-02-26 17:10:04,428 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1_conf.xml to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp 877 2020-02-26 17:10:04,460 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied from: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1_conf.xml to done location: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp 878 2020-02-26 17:10:04,466 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009.summary_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009.summary 879 2020-02-26 17:10:04,469 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml 880 2020-02-26 17:10:04,471 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist 881 2020-02-26 17:10:04,471 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 882 2020-02-26 17:10:04,476 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Task failed task_1582677468069_0009_m_000003 883 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 884 2020-02-26 17:10:04,476 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: History url is http://cdh-utility-1.public.cdhvcn.hiddenvcn.com:19888/jobhistory/job/job_1582677468069_0009 885 2020-02-26 17:10:04,481 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Waiting for application to be successfully unregistered. 886 2020-02-26 17:10:05,482 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:19 ContRel:3 HostLocal:3 RackLocal:1 887 2020-02-26 17:10:05,483 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://nameservice1 /user/hdfs/.staging/job_1582677468069_0009 888 2020-02-26 17:10:05,487 INFO [Thread-90] org.apache.hadoop.ipc.Server: Stopping server on 34533 889 2020-02-26 17:10:05,488 INFO [IPC Server listener on 34533] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 34533 890 2020-02-26 17:10:05,488 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted 891 2020-02-26 17:10:05,490 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 892 2020-02-26 17:10:05,490 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted 893 2020-02-26 17:10:10,491 INFO [Thread-90] org.apache.hadoop.ipc.Server: Stopping server on 46625 894 2020-02-26 17:10:10,491 INFO [IPC Server listener on 46625] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 46625 895 2020-02-26 17:10:10,491 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 896 2020-02-26 17:10:10,494 INFO [Thread-90] org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.w.WebAppContext@1f939a0f{/,null,UNAVAILABLE}{/mapreduce} 897 2020-02-26 17:10:10,497 INFO [Thread-90] org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@7112ce6{HTTP/1.1,[http/1.1]}{0.0.0.0:0} 898 2020-02-26 17:10:10,498 INFO [Thread-90] org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@39109136{/static,jar:file:/data4/yarn/nm/filecache/12/3.0.0-cdh6.1.1-mr-framework.tar.gz/hadoop-yarn-common-3.0.0-cdh6.1.1.jar!/webapps/static,UNAVAILABLE} 899 End of LogType:syslog 900 *********************************************************************** ... View more

Re: ALL hadoop-mapreduce-examples.jar fail cdh6

by Explorer in Support Questions
‎02-27-2020 12:26 PM
‎02-27-2020 12:26 PM
I went ahead and tried on the RM server as well and same darn error.  I just don't get it.      As you had asked for the log .  Here it is: 296 java.io.tmpdir: /data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001/tmp 297 user.dir: /data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001 298 user.name: yarn 299 ************************************************************/ 300 2020-02-26 17:09:49,953 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration 301 2020-02-26 17:09:50,097 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 9 cluster_timestamp: 1582677468069 } attemptId: 1 } keyId: -1009427054)] 302 2020-02-26 17:09:50,128 INFO [main] org.apache.hadoop.conf.Configuration: resource-types.xml not found 303 2020-02-26 17:09:50,128 INFO [main] org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'. 304 2020-02-26 17:09:50,136 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter. 305 2020-02-26 17:09:50,138 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null 306 2020-02-26 17:09:50,173 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2 307 2020-02-26 17:09:50,173 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 308 2020-02-26 17:09:50,627 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 309 2020-02-26 17:09:50,662 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 310 2020-02-26 17:09:50,830 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler 311 2020-02-26 17:09:50,831 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher 312 2020-02-26 17:09:50,831 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher 313 2020-02-26 17:09:50,832 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher 314 2020-02-26 17:09:50,832 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler 315 2020-02-26 17:09:50,833 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher 316 2020-02-26 17:09:50,834 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter 317 2020-02-26 17:09:50,834 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter 318 2020-02-26 17:09:50,864 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020] 319 2020-02-26 17:09:50,881 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020] 320 2020-02-26 17:09:50,895 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020] 321 2020-02-26 17:09:50,909 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled 322 2020-02-26 17:09:50,946 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler 323 2020-02-26 17:09:51,187 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties 324 2020-02-26 17:09:51,244 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 325 2020-02-26 17:09:51,244 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started 326 2020-02-26 17:09:51,252 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1582677468069_0009 to jobTokenSecretManager 327 2020-02-26 17:09:51,358 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1582677468069_0009 because: not enabled; 328 2020-02-26 17:09:51,374 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1582677468069_0009 = 472. Number of splits = 4 329 2020-02-26 17:09:51,375 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1582677468069_0009 = 1 330 2020-02-26 17:09:51,375 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from NEW to INITED 331 2020-02-26 17:09:51,376 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1582677468069_0009. 332 2020-02-26 17:09:51,399 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 333 2020-02-26 17:09:51,408 INFO [Socket Reader #1 for port 46625] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 46625 334 2020-02-26 17:09:51,577 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server 335 2020-02-26 17:09:51,578 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 336 2020-02-26 17:09:51,578 INFO [IPC Server listener on 46625] org.apache.hadoop.ipc.Server: IPC Server listener on 46625: starting 337 2020-02-26 17:09:51,579 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at cdh-worker-3.private.cdhvcn.hiddenvcn.com/10.161.2.5:46625 338 2020-02-26 17:09:51,609 INFO [main] org.eclipse.jetty.util.log: Logging initialized @2635ms 339 2020-02-26 17:09:51,697 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 340 2020-02-26 17:09:51,700 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined 341 2020-02-26 17:09:51,705 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 342 2020-02-26 17:09:51,747 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce 343 2020-02-26 17:09:51,747 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static 344 2020-02-26 17:09:51,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/* 345 2020-02-26 17:09:51,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/* 346 2020-02-26 17:09:52,083 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules 347 2020-02-26 17:09:52,084 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 45277 348 2020-02-26 17:09:52,085 INFO [main] org.eclipse.jetty.server.Server: jetty-9.3.20.v20170531 349 2020-02-26 17:09:52,122 INFO [main] org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@39109136{/static,jar:file:/data4/yarn/nm/filecache/12/3.0.0-cdh6.1.1-mr-framework.tar.gz/hadoop-yarn-common-3.0.0-cdh6.1.1.jar!/webapps/static,AVAILABLE} 350 2020-02-26 17:09:53,009 INFO [main] org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@1f939a0f{/,file:///data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001/tmp/jetty-0.0.0.0-45277-mapreduce-_-any-7944828710115519068.dir/webapp/,AVAILABLE}{/mapreduce} 351 2020-02-26 17:09:53,017 INFO [main] org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@7112ce6{HTTP/1.1,[http/1.1]}{0.0.0.0:45277} 352 2020-02-26 17:09:53,017 INFO [main] org.eclipse.jetty.server.Server: Started @4043ms 353 2020-02-26 17:09:53,017 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app mapreduce started at 45277 354 2020-02-26 17:09:53,022 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 355 2020-02-26 17:09:53,023 INFO [Socket Reader #1 for port 34533] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34533 356 2020-02-26 17:09:53,028 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting 357 2020-02-26 17:09:53,028 INFO [IPC Server listener on 34533] org.apache.hadoop.ipc.Server: IPC Server listener on 34533: starting 358 2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true 359 2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3 360 2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33 361 2020-02-26 17:09:53,060 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the mappers will be scheduled using OPPORTUNISTIC containers 362 2020-02-26 17:09:53,154 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: maxContainerCapability: <memory:8192, vCores:2> 363 2020-02-26 17:09:53,154 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: queue: root.users.hdfs 364 2020-02-26 17:09:53,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500 365 2020-02-26 17:09:53,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10 366 2020-02-26 17:09:53,169 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from INITED to SETUP 367 2020-02-26 17:09:53,171 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP 368 2020-02-26 17:09:53,182 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from SETUP to RUNNING 369 2020-02-26 17:09:53,274 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1582677468069_0009, File: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist 370 2020-02-26 17:09:53,288 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from NEW to SCHEDULED 371 2020-02-26 17:09:53,311 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from NEW to SCHEDULED 372 2020-02-26 17:09:53,313 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from NEW to SCHEDULED 373 2020-02-26 17:09:53,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from NEW to SCHEDULED 374 2020-02-26 17:09:53,316 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from NEW to SCHEDULED 375 2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 376 2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED 377 2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED 378 2020-02-26 17:09:53,318 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED 379 2020-02-26 17:09:53,318 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED 380 2020-02-26 17:09:53,319 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1> 381 2020-02-26 17:09:53,332 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:1024, vCores:1> 382 2020-02-26 17:09:54,158 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0 383 2020-02-26 17:09:54,189 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=6 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:454656, vCores:63> knownNMs=4 384 2020-02-26 17:09:54,192 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:454656, vCores:63> 385 2020-02-26 17:09:54,192 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 386 2020-02-26 17:09:55,206 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4 387 2020-02-26 17:09:55,209 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000002 to attempt_1582677468069_0009_m_000000_0 388 2020-02-26 17:09:55,211 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000003 to attempt_1582677468069_0009_m_000001_0 389 2020-02-26 17:09:55,212 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000004 to attempt_1582677468069_0009_m_000002_0 390 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000005 to attempt_1582677468069_0009_m_000003_0 391 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:450560, vCores:59> 392 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 393 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1 394 2020-02-26 17:09:55,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://nameservice1/user/hdfs/.staging/job_1582677468069_0009/job.jar 395 2020-02-26 17:09:55,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hdfs/.staging/job_1582677468069_0009/job.xml 396 2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container 397 2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1 398 2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData 399 2020-02-26 17:09:55,308 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 400 2020-02-26 17:09:55,311 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 401 2020-02-26 17:09:55,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 402 2020-02-26 17:09:55,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 403 2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 404 2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 405 2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 406 2020-02-26 17:09:55,316 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 407 2020-02-26 17:09:55,318 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000002 taskAttempt attempt_1582677468069_0009_m_000000_0 408 2020-02-26 17:09:55,318 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000003 taskAttempt attempt_1582677468069_0009_m_000001_0 409 2020-02-26 17:09:55,318 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000004 taskAttempt attempt_1582677468069_0009_m_000002_0 410 2020-02-26 17:09:55,319 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000005 taskAttempt attempt_1582677468069_0009_m_000003_0 411 2020-02-26 17:09:55,321 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_0 412 2020-02-26 17:09:55,321 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_0 413 2020-02-26 17:09:55,321 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_0 414 2020-02-26 17:09:55,321 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_0 415 2020-02-26 17:09:55,404 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_0 : 13562 416 2020-02-26 17:09:55,405 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_0 : 13562 417 2020-02-26 17:09:55,405 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_0] using containerId: [container_e02_1582677468069_0009_01_000005 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041] 418 2020-02-26 17:09:55,408 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_0 : 13562 419 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 420 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_0] using containerId: [container_e02_1582677468069_0009_01_000003 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041] 421 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 422 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_0] using containerId: [container_e02_1582677468069_0009_01_000002 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041] 423 2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 424 2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from SCHEDULED to RUNNING 425 2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from SCHEDULED to RUNNING 426 2020-02-26 17:09:55,411 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from SCHEDULED to RUNNING 427 2020-02-26 17:09:55,412 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_0 : 13562 428 2020-02-26 17:09:55,412 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_0] using containerId: [container_e02_1582677468069_0009_01_000004 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041] 429 2020-02-26 17:09:55,413 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING 430 2020-02-26 17:09:55,413 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from SCHEDULED to RUNNING 431 2020-02-26 17:09:56,221 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=6 release= 0 newContainers=0 finishedContainers=4 resourcelimit=<memory:454656, vCores:63> knownNMs=4 432 2020-02-26 17:09:56,221 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000004 433 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000003 434 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000002 435 2020-02-26 17:09:56,222 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000002_0: [2020-02-26 17:09:55.814]Exception from container-launch. 436 Container id: container_e02_1582677468069_0009_01_000004 437 Exit code: 1 438 [2020-02-26 17:09:55.815]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 439 Last 4096 bytes of prelaunch.err : 440 Last 4096 bytes of stderr : 441 Error: Could not find or load main class NOT 442 [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 443 Last 4096 bytes of prelaunch.err : 444 Last 4096 bytes of stderr : 445 Error: Could not find or load main class NOT 446 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000005 447 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:454656, vCores:63> 448 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 449 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1 450 2020-02-26 17:09:56,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041 451 2020-02-26 17:09:56,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_0: [2020-02-26 17:09:55.826]Exception from container-launch. 452 Container id: container_e02_1582677468069_0009_01_000003 453 Exit code: 1 454 [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 455 Last 4096 bytes of prelaunch.err : 456 Last 4096 bytes of stderr : 457 Error: Could not find or load main class NOT 458 [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 459 Last 4096 bytes of prelaunch.err : 460 Last 4096 bytes of stderr : 461 Error: Could not find or load main class NOT 462 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041 463 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_0: [2020-02-26 17:09:55.813]Exception from container-launch. 464 Container id: container_e02_1582677468069_0009_01_000002 465 Exit code: 1 466 [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 467 Last 4096 bytes of prelaunch.err : 468 Last 4096 bytes of stderr : 469 Error: Could not find or load main class NOT 470 [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 471 Last 4096 bytes of prelaunch.err : 472 Last 4096 bytes of stderr : 473 Error: Could not find or load main class NOT 474 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041 475 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_0: [2020-02-26 17:09:55.819]Exception from container-launch. 476 Container id: container_e02_1582677468069_0009_01_000005 477 Exit code: 1 478 [2020-02-26 17:09:55.852]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 479 Last 4096 bytes of prelaunch.err : 480 Last 4096 bytes of stderr : 481 Error: Could not find or load main class NOT 482 [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 483 Last 4096 bytes of prelaunch.err : 484 Last 4096 bytes of stderr : 485 Error: Could not find or load main class NOT 486 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041 487 2020-02-26 17:09:56,228 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000004 taskAttempt attempt_1582677468069_0009_m_000002_0 488 2020-02-26 17:09:56,237 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000003 taskAttempt attempt_1582677468069_0009_m_000001_0 489 2020-02-26 17:09:56,240 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000002 taskAttempt attempt_1582677468069_0009_m_000000_0 490 2020-02-26 17:09:56,241 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000005 taskAttempt attempt_1582677468069_0009_m_000003_0 491 2020-02-26 17:09:56,243 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com 492 2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from NEW to UNASSIGNED 493 2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from NEW to UNASSIGNED 494 2020-02-26 17:09:56,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com 495 2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED 496 2020-02-26 17:09:56,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com 497 2020-02-26 17:09:56,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted host cdh-worker-1.private.cdhvcn.hiddenvcn.com 498 2020-02-26 17:09:56,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from NEW to UNASSIGNED 499 2020-02-26 17:09:56,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_1 to list of failed maps 500 2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000001_1 to list of failed maps 501 2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000000_1 to list of failed maps 502 2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000003_1 to list of failed maps 503 2020-02-26 17:09:57,223 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1 504 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:339968, vCores:47> knownNMs=4 505 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=1 blacklistRemovals=0 506 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:339968, vCores:47> 507 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 508 2020-02-26 17:09:58,231 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4 509 2020-02-26 17:09:58,232 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000006, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 510 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 511 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000006 to attempt_1582677468069_0009_m_000002_1 512 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000007, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 513 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 514 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000007 to attempt_1582677468069_0009_m_000001_1 515 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000008, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 516 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 517 2020-02-26 17:09:58,233 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 518 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000008 to attempt_1582677468069_0009_m_000000_1 519 2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000009, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 520 2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 521 2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000009 to attempt_1582677468069_0009_m_000003_1 522 2020-02-26 17:09:58,234 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 523 2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:335872, vCores:43> 524 2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 525 2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1 526 2020-02-26 17:09:58,235 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 527 2020-02-26 17:09:58,236 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 528 2020-02-26 17:09:58,236 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 529 2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 530 2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 531 2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 532 2020-02-26 17:09:58,238 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000006 taskAttempt attempt_1582677468069_0009_m_000002_1 533 2020-02-26 17:09:58,238 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000009 taskAttempt attempt_1582677468069_0009_m_000003_1 534 2020-02-26 17:09:58,238 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_1 535 2020-02-26 17:09:58,238 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000007 taskAttempt attempt_1582677468069_0009_m_000001_1 536 2020-02-26 17:09:58,238 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000008 taskAttempt attempt_1582677468069_0009_m_000000_1 537 2020-02-26 17:09:58,238 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_1 538 2020-02-26 17:09:58,238 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_1 539 2020-02-26 17:09:58,238 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_1 540 2020-02-26 17:09:58,265 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_1 : 13562 541 2020-02-26 17:09:58,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_1] using containerId: [container_e02_1582677468069_0009_01_000009 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041] 542 2020-02-26 17:09:58,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from ASSIGNED to RUNNING 543 2020-02-26 17:09:58,273 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_1 : 13562 544 2020-02-26 17:09:58,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_1] using containerId: [container_e02_1582677468069_0009_01_000007 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041] 545 2020-02-26 17:09:58,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from ASSIGNED to RUNNING 546 2020-02-26 17:09:58,280 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_1 : 13562 547 2020-02-26 17:09:58,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_1] using containerId: [container_e02_1582677468069_0009_01_000006 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041] 548 2020-02-26 17:09:58,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from ASSIGNED to RUNNING 549 2020-02-26 17:09:58,289 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_1 : 13562 550 2020-02-26 17:09:58,289 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_1] using containerId: [container_e02_1582677468069_0009_01_000008 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041] 551 2020-02-26 17:09:58,289 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from ASSIGNED to RUNNING 552 2020-02-26 17:09:59,238 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=3 resourcelimit=<memory:338944, vCores:46> knownNMs=4 553 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000008 554 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000009 555 2020-02-26 17:09:59,239 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_1: [2020-02-26 17:09:58.706]Exception from container-launch. 556 Container id: container_e02_1582677468069_0009_01_000008 557 Exit code: 1 558 [2020-02-26 17:09:58.708]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 559 Last 4096 bytes of prelaunch.err : 560 Last 4096 bytes of stderr : 561 Error: Could not find or load main class NOT 562 [2020-02-26 17:09:58.708]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 563 Last 4096 bytes of prelaunch.err : 564 Last 4096 bytes of stderr : 565 Error: Could not find or load main class NOT 566 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000007 567 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:338944, vCores:46> 568 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 569 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1 570 2020-02-26 17:09:59,239 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041 571 2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_1: [2020-02-26 17:09:58.721]Exception from container-launch. 572 Container id: container_e02_1582677468069_0009_01_000009 573 Exit code: 1 574 [2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 575 Last 4096 bytes of prelaunch.err : 576 Last 4096 bytes of stderr : 577 Error: Could not find or load main class NOT 578 [2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 579 Last 4096 bytes of prelaunch.err : 580 Last 4096 bytes of stderr : 581 Error: Could not find or load main class NOT 582 2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041 583 2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_1: [2020-02-26 17:09:58.712]Exception from container-launch. 584 Container id: container_e02_1582677468069_0009_01_000007 585 Exit code: 1 586 [2020-02-26 17:09:58.732]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 587 Last 4096 bytes of prelaunch.err : 588 Last 4096 bytes of stderr : 589 Error: Could not find or load main class NOT 590 [2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 591 Last 4096 bytes of prelaunch.err : 592 Last 4096 bytes of stderr : 593 Error: Could not find or load main class NOT 594 2020-02-26 17:09:59,241 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041 595 2020-02-26 17:09:59,241 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000008 taskAttempt attempt_1582677468069_0009_m_000000_1 596 2020-02-26 17:09:59,242 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000009 taskAttempt attempt_1582677468069_0009_m_000003_1 597 2020-02-26 17:09:59,244 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000007 taskAttempt attempt_1582677468069_0009_m_000001_1 598 2020-02-26 17:09:59,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com 599 2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from NEW to UNASSIGNED 600 2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com 601 2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from NEW to UNASSIGNED 602 2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com 603 2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted host cdh-worker-3.private.cdhvcn.hiddenvcn.com 604 2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from NEW to UNASSIGNED 605 2020-02-26 17:09:59,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000000_2 to list of failed maps 606 2020-02-26 17:09:59,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000003_2 to list of failed maps 607 2020-02-26 17:09:59,248 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000001_2 to list of failed maps 608 2020-02-26 17:10:00,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1 609 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:229376, vCores:32> knownNMs=4 610 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=1 blacklistRemovals=0 611 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Ignore blacklisting set to true. Known: 4, Blacklisted: 2, 50% 612 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000006 613 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:229376, vCores:32> 614 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 615 2020-02-26 17:10:00,243 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000002_1: [2020-02-26 17:09:58.725]Exception from container-launch. 616 Container id: container_e02_1582677468069_0009_01_000006 617 Exit code: 1 618 [2020-02-26 17:09:58.734]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 619 Last 4096 bytes of prelaunch.err : 620 Last 4096 bytes of stderr : 621 Error: Could not find or load main class NOT 622 [2020-02-26 17:09:58.734]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 623 Last 4096 bytes of prelaunch.err : 624 Last 4096 bytes of stderr : 625 Error: Could not find or load main class NOT 626 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1 627 2020-02-26 17:10:00,244 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041 628 2020-02-26 17:10:00,244 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000006 taskAttempt attempt_1582677468069_0009_m_000002_1 629 2020-02-26 17:10:00,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_2 TaskAttempt Transitioned from NEW to UNASSIGNED 630 2020-02-26 17:10:00,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_2 to list of failed maps 631 2020-02-26 17:10:01,244 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1 632 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=3 finishedContainers=0 resourcelimit=<memory:451584, vCores:60> knownNMs=4 633 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=0 blacklistRemovals=2 634 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 3 635 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000010, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 636 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 637 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000010 to attempt_1582677468069_0009_m_000000_2 638 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000011, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 639 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 640 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000011 to attempt_1582677468069_0009_m_000003_2 641 2020-02-26 17:10:01,251 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 642 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000012, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 643 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 644 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000012 to attempt_1582677468069_0009_m_000001_2 645 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:451584, vCores:60> 646 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 647 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:11 ContRel:0 HostLocal:3 RackLocal:1 648 2020-02-26 17:10:01,252 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 649 2020-02-26 17:10:01,254 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 650 2020-02-26 17:10:01,254 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 651 2020-02-26 17:10:01,255 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 652 2020-02-26 17:10:01,255 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 653 2020-02-26 17:10:01,255 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000010 taskAttempt attempt_1582677468069_0009_m_000000_2 654 2020-02-26 17:10:01,255 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000011 taskAttempt attempt_1582677468069_0009_m_000003_2 655 2020-02-26 17:10:01,255 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000012 taskAttempt attempt_1582677468069_0009_m_000001_2 656 2020-02-26 17:10:01,256 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_2 657 2020-02-26 17:10:01,255 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_2 658 2020-02-26 17:10:01,256 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_2 659 2020-02-26 17:10:01,267 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_2 : 13562 660 2020-02-26 17:10:01,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_2] using containerId: [container_e02_1582677468069_0009_01_000011 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 661 2020-02-26 17:10:01,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from ASSIGNED to RUNNING 662 2020-02-26 17:10:01,271 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_2 : 13562 663 2020-02-26 17:10:01,271 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_2] using containerId: [container_e02_1582677468069_0009_01_000012 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 664 2020-02-26 17:10:01,271 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from ASSIGNED to RUNNING 665 2020-02-26 17:10:01,273 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_2 : 13562 666 2020-02-26 17:10:01,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_2] using containerId: [container_e02_1582677468069_0009_01_000010 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 667 2020-02-26 17:10:01,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from ASSIGNED to RUNNING 668 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=4 finishedContainers=3 resourcelimit=<memory:450560, vCores:59> knownNMs=4 669 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000011 670 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000012 671 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000010 672 2020-02-26 17:10:02,257 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_2: [2020-02-26 17:10:01.685]Exception from container-launch. 673 Container id: container_e02_1582677468069_0009_01_000011 674 Exit code: 1 675 [2020-02-26 17:10:01.686]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 676 Last 4096 bytes of prelaunch.err : 677 Last 4096 bytes of stderr : 678 Error: Could not find or load main class NOT 679 [2020-02-26 17:10:01.691]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 680 Last 4096 bytes of prelaunch.err : 681 Last 4096 bytes of stderr : 682 Error: Could not find or load main class NOT 683 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4 684 2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041 685 2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_2: [2020-02-26 17:10:01.698]Exception from container-launch. 686 Container id: container_e02_1582677468069_0009_01_000012 687 Exit code: 1 688 [2020-02-26 17:10:01.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 689 Last 4096 bytes of prelaunch.err : 690 Last 4096 bytes of stderr : 691 Error: Could not find or load main class NOT 692 [2020-02-26 17:10:01.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 693 Last 4096 bytes of prelaunch.err : 694 Last 4096 bytes of stderr : 695 Error: Could not find or load main class NOT 696 2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000013, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 697 2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 698 2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041 699 2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000013 to attempt_1582677468069_0009_m_000002_2 700 2020-02-26 17:10:02,259 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_2: [2020-02-26 17:10:01.688]Exception from container-launch. 701 Container id: container_e02_1582677468069_0009_01_000010 702 Exit code: 1   ... View more

ALL hadoop-mapreduce-examples.jar fail cdh6

by Explorer in Support Questions
‎02-26-2020 09:14 AM
‎02-26-2020 09:14 AM
Please help I've spent 3 days and still can't get an example to run including pi, terasort or even wordcount New installation and have turned on HA for yarn and also hdfs, and installed the yarn MapReduce Framework JARs as well. Cluster looks health as seen below:   If I try and run any of the examples ...they fail. [hdfs@cdh-utility-1 opc]$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4 WARNING: Use "yarn jar" to launch YARN applications. Number of Maps = 4 Samples per Map example.com= 4 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Starting Job 20/02/26 17:09:47 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1582677468069_0009 20/02/26 17:09:47 INFO input.FileInputFormat: Total input files to process : 4 20/02/26 17:09:47 INFO mapreduce.JobSubmitter: number of splits:4 20/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.zk-address is deprecated. Instead, use hadoop.zk.address 20/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 20/02/26 17:09:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1582677468069_0009 20/02/26 17:09:48 INFO mapreduce.JobSubmitter: Executing with tokens: [] 20/02/26 17:09:48 INFO conf.Configuration: resource-types.xml not found 20/02/26 17:09:48 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 20/02/26 17:09:48 INFO impl.YarnClientImpl: Submitted application application_1582677468069_0009 20/02/26 17:09:48 INFO mapreduce.Job: The url to track the job: http://cdh-master-1.private.cdhvcn.examplevcn.com:8088/proxy/application_1582677468069_0009/ 20/02/26 17:09:48 INFO mapreduce.Job: Running job: job_1582677468069_0009 20/02/26 17:09:54 INFO mapreduce.Job: Job job_1582677468069_0009 running in uber mode : false 20/02/26 17:09:54 INFO mapreduce.Job: map 0% reduce 0% 20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000002_0, Status : FAILED [2020-02-26 17:09:55.814]Exception from container-launch. Container id: container_e02_1582677468069_0009_01_000004 Exit code: 1 [2020-02-26 17:09:55.815]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT   20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000001_0, Status : FAILED [2020-02-26 17:09:55.826]Exception from container-launch. Container id: container_e02_1582677468069_0009_01_000003 Exit code: 1 [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT   20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000000_0, Status : FAILED [2020-02-26 17:09:55.813]Exception from container-launch. Container id: container_e02_1582677468069_0009_01_000002 Exit code: 1 [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT   20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000003_0, Status : FAILED [2020-02-26 17:09:55.819]Exception from container-launch. Container id: container_e02_1582677468069_0009_01_000005 Exit code: 1 [2020-02-26 17:09:55.852]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT ... View more
Contact Me
Online
Offline
Last Visited
‎11-18-2020 12:54 AM
Community Statistics
Member Since ‎03-02-2019 11:11 AM
Last Visited ‎11-18-2020 12:54 AM
Posts 13