Member since
03-02-2019
13
Posts
0
Kudos Received
0
Solutions
10-14-2020
02:20 PM
We are trying to pull metrics into Promtheus and for the life of me I spent most of lastnight trying to get CDH 6.2 to work with a javaagent. I read a previous post where someone was have a heck of a time with jmx exporter and Ambari, so I'm assuming this is also an issue in CDH? If I goto: HBase Service Environment Advanced Configuration Snippet (Safety Valve) and add this: HBASE_MASTER_OPTS=-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=10110 -javaagent:/run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar=9214:/run/cloudera-scm-agent/hbasemaster.yml" HBASE_REGIONSERVER_OPTS=-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=10111 -javaagent:/run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar=9215:/run/cloudera-scm-agent/regionserver.yml" and then restart the service The service WILL not restart and stderr shows: Error opening zip file or JAR manifest missing : /run/cloudera-scm-agent/jmx_prometheus_javaagent-0.14.0.jar I've verified all files are there : -rw-rw-rw- 1 root root 405K Sep 4 12:20 jmx_prometheus_javaagent-0.14.0.jar -rwxr-xr-x 1 root root 283 Oct 14 02:04 namenode.jml -rwxr-xr-x 1 root root 283 Oct 14 02:04 datanode.jml -rwxr-xr-x 1 root root 283 Oct 14 02:04 regionserver.jml -rwxr-xr-x 1 root root 283 Oct 14 06:17 hbasemaster.jml -rwxr-xr-x 1 root root 283 Oct 14 06:57 yarn.jml I've tried to modify the hbase-env.sh directly, but of course there are so many and that I can't seem to get it to work. Has anyone got this working in CDH5 or CDH6 and if so, can point me in the right direction
... View more
Labels:
03-03-2020
12:01 PM
yep already did and even looked for a tuning parameter based on the cloud shape I'm using and so ended up setting both of them to: -Djava.net.preferIPv4Stack=true -Xms2368m -Xmx2368m and now pi works. Thanks a bunch.
... View more
03-02-2020
08:53 PM
@bgooley You might be onto something there. Since I was not running 6.3...I tore down the cluster and rebuilt from scratch this time using CDH 6.3.2, enabled HA for yarn and hdfs, deployed yarn framework, created job history directory and created nodemanager application log directory, then set Localized Dir Deletion Delay to 1800 and Enable Container Launch Debug Information.. Then ran this: sudo -u hdfs hadoop jar /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4 then pulled the logs and looked for the bash line and here is what I found: echo "ls -l:" 1>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" ls -l 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" find -L . -maxdepth 5 -ls 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/directory.info" echo "Launching container" exec /bin/bash -c "$JAVA_HOME/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN NOT FOUND -Xmx820m -Djava.io.tmpdir=$PWD/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 10.0.2.7 33307 attempt_1583199702433_0138_m_000002_3 3298534883342 1>/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/stdout 2>/var/log/cloudera/hadoop-yarn/container/application_1583199702433_0138/container_e03_1583199702433_0138_01_000014/stderr " End of LogType:launch_container.sh ************************************************************************************ so I then went into Yarn and looked for Map Task Java Opts Base and found: BINGO. You hit the nail on the head. Great find.
... View more
03-01-2020
10:33 AM
The snippets from the log that I uploaded were actually from the output of the yarn log command as both of you had requested. If there is a way I can upload the full log I'd be more than happy to. As for remove of the domain name I actually change them before I submit so they are not feasible anyway 🙂 🙂
... View more
02-28-2020
08:10 AM
I don't see anything here at all for doing an attachment, just links 😞 😞 . so I'll apologize now. But, here is more of the log. 753 [2020-02-26 17:10:02.569]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 754 Last 4096 bytes of prelaunch.err : 755 Last 4096 bytes of stderr : 756 Error: Could not find or load main class NOT 757 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000015 758 2020-02-26 17:10:03,267 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container container_e02_1582677468069_0009_01_000015 759 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000016 760 2020-02-26 17:10:03,267 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container complete event for unknown container container_e02_1582677468069_0009_01_000016 761 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 762 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000017, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 763 2020-02-26 17:10:03,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041 764 2020-02-26 17:10:03,267 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 765 2020-02-26 17:10:03,267 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000013 taskAttempt attempt_1582677468069_0009_m_000002_2 766 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000017 to attempt_1582677468069_0009_m_000003_3 767 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:453632, vCores:62> 768 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 769 2020-02-26 17:10:03,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:2 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:16 ContRel:3 HostLocal:3 RackLocal:1 770 2020-02-26 17:10:03,269 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 771 2020-02-26 17:10:03,270 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 772 2020-02-26 17:10:03,270 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from NEW to UNASSIGNED 773 2020-02-26 17:10:03,270 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000017 taskAttempt attempt_1582677468069_0009_m_000003_3 774 2020-02-26 17:10:03,270 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_3 to list of failed maps 775 2020-02-26 17:10:03,270 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_3 776 2020-02-26 17:10:03,281 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_3 : 13562 777 2020-02-26 17:10:03,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_3] using containerId: [container_e02_1582677468069_0009_01_000017 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 778 2020-02-26 17:10:03,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 TaskAttempt Transitioned from ASSIGNED to RUNNING 779 2020-02-26 17:10:04,268 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:16 ContRel:3 HostLocal:3 RackLocal:1 780 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=3 finishedContainers=1 resourcelimit=<memory:451584, vCores:60> knownNMs=4 781 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000017 782 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 3 783 2020-02-26 17:10:04,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_3: [2020-02-26 17:10:03.568]Exception from container-launch. 784 Container id: container_e02_1582677468069_0009_01_000017 785 Exit code: 1 786 [2020-02-26 17:10:03.570]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 787 Last 4096 bytes of prelaunch.err : 788 Last 4096 bytes of stderr : 789 Error: Could not find or load main class NOT 790 [2020-02-26 17:10:03.570]Container exited with a non-zero exit code 1. Error file: prelaunch.err. 791 Last 4096 bytes of prelaunch.err : 792 Last 4096 bytes of stderr : 793 Error: Could not find or load main class NOT 794 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000018, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 795 2020-02-26 17:10:04,273 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 796 2020-02-26 17:10:04,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_3 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041 797 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000018 to attempt_1582677468069_0009_m_000001_3 798 2020-02-26 17:10:04,274 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000017 taskAttempt attempt_1582677468069_0009_m_000003_3 799 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000019, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 800 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 801 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000019 to attempt_1582677468069_0009_m_000000_3 802 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000020, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map 803 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps 804 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000020 to attempt_1582677468069_0009_m_000002_3 805 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:451584, vCores:60> 806 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4 807 2020-02-26 17:10:04,274 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:19 ContRel:3 HostLocal:3 RackLocal:1 808 2020-02-26 17:10:04,276 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from RUNNING to FAILED 809 2020-02-26 17:10:04,277 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 810 2020-02-26 17:10:04,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 811 2020-02-26 17:10:04,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 812 2020-02-26 17:10:04,279 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 813 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m 814 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 815 2020-02-26 17:10:04,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1 816 2020-02-26 17:10:04,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 817 2020-02-26 17:10:04,282 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from RUNNING to FAIL_WAIT 818 2020-02-26 17:10:04,282 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000018 taskAttempt attempt_1582677468069_0009_m_000001_3 819 2020-02-26 17:10:04,282 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000019 taskAttempt attempt_1582677468069_0009_m_000000_3 820 2020-02-26 17:10:04,282 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_3 821 2020-02-26 17:10:04,283 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_3 822 2020-02-26 17:10:04,283 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000020 taskAttempt attempt_1582677468069_0009_m_000002_3 823 2020-02-26 17:10:04,283 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_3 824 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from RUNNING to KILL_WAIT 825 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from RUNNING to KILL_WAIT 826 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from RUNNING to KILL_WAIT 827 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from SCHEDULED to KILL_WAIT 828 2020-02-26 17:10:04,283 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP 829 2020-02-26 17:10:04,284 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP 830 2020-02-26 17:10:04,284 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from ASSIGNED to KILL_CONTAINER_CLEANUP 831 2020-02-26 17:10:04,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_r_000000_0 TaskAttempt Transitioned from UNASSIGNED to KILLED 832 2020-02-26 17:10:04,285 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Processing the event EventType: CONTAINER_DEALLOCATE 833 2020-02-26 17:10:04,285 ERROR [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Could not deallocate container for task attemptId attempt_1582677468069_0009_r_000000_0 834 2020-02-26 17:10:04,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from KILL_WAIT to KILLED 835 2020-02-26 17:10:04,286 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000019 taskAttempt attempt_1582677468069_0009_m_000000_3 836 2020-02-26 17:10:04,286 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000018 taskAttempt attempt_1582677468069_0009_m_000001_3 837 2020-02-26 17:10:04,286 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_e02_1582677468069_0009_01_000020 taskAttempt attempt_1582677468069_0009_m_000002_3 838 2020-02-26 17:10:04,297 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_3 : 13562 839 2020-02-26 17:10:04,298 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000001_3 840 2020-02-26 17:10:04,305 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_3 : 13562 841 2020-02-26 17:10:04,306 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000002_3 842 2020-02-26 17:10:04,310 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_3 : 13562 843 2020-02-26 17:10:04,310 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1582677468069_0009_m_000000_3 844 2020-02-26 17:10:04,324 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 845 2020-02-26 17:10:04,324 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 846 2020-02-26 17:10:04,325 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 847 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 848 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 849 2020-02-26 17:10:04,325 INFO [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 850 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #3] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000000_3 851 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000001_3 852 2020-02-26 17:10:04,331 WARN [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/user/hdfs/QuasiMonteCarlo_1582736986036_1214499353/out/_temporary/1/_temporary/attempt_1582677468069_0009_m_000002_3 853 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_3] using containerId: [container_e02_1582677468069_0009_01_000019 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 854 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 855 2020-02-26 17:10:04,331 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_3] using containerId: [container_e02_1582677468069_0009_01_000018 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 856 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 857 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_3] using containerId: [container_e02_1582677468069_0009_01_000020 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041] 858 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_3 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 859 2020-02-26 17:10:04,332 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from KILL_WAIT to KILLED 860 2020-02-26 17:10:04,333 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from KILL_WAIT to KILLED 861 2020-02-26 17:10:04,333 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from KILL_WAIT to KILLED 862 2020-02-26 17:10:04,334 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from FAIL_WAIT to FAIL_ABORT 863 2020-02-26 17:10:04,334 INFO [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_ABORT 864 2020-02-26 17:10:04,344 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from FAIL_ABORT to FAILED 865 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job finished cleanly, recording last MRAppMaster retry 866 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true 867 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: RMCommunicator notified that shouldUnregistered is: true 868 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true 869 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true 870 2020-02-26 17:10:04,345 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services 871 2020-02-26 17:10:04,346 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 1 872 2020-02-26 17:10:04,347 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: In stop, writing event JOB_FAILED 873 2020-02-26 17:10:04,389 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp 874 2020-02-26 17:10:04,424 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied from: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist to done location: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp 875 2020-02-26 17:10:04,427 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Set historyUrl to http://cdh-utility-1.public.cdhvcn.hiddenvcn.com:19888/jobhistory/job/job_1582677468069_0009 876 2020-02-26 17:10:04,428 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1_conf.xml to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp 877 2020-02-26 17:10:04,460 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied from: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1_conf.xml to done location: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp 878 2020-02-26 17:10:04,466 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009.summary_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009.summary 879 2020-02-26 17:10:04,469 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009_conf.xml 880 2020-02-26 17:10:04,471 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist_tmp to hdfs://nameservice1:8020/user/history/done_intermediate/hdfs/job_1582677468069_0009-1582736988269-hdfs-QuasiMonteCarlo-1582737004281-0-0-FAILED-root.users.hdfs-1582736993164.jhist 881 2020-02-26 17:10:04,471 INFO [Thread-90] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 882 2020-02-26 17:10:04,476 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Setting job diagnostics to Task failed task_1582677468069_0009_m_000003 883 Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0 884 2020-02-26 17:10:04,476 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: History url is http://cdh-utility-1.public.cdhvcn.hiddenvcn.com:19888/jobhistory/job/job_1582677468069_0009 885 2020-02-26 17:10:04,481 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: Waiting for application to be successfully unregistered. 886 2020-02-26 17:10:05,482 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:19 ContRel:3 HostLocal:3 RackLocal:1 887 2020-02-26 17:10:05,483 INFO [Thread-90] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://nameservice1 /user/hdfs/.staging/job_1582677468069_0009 888 2020-02-26 17:10:05,487 INFO [Thread-90] org.apache.hadoop.ipc.Server: Stopping server on 34533 889 2020-02-26 17:10:05,488 INFO [IPC Server listener on 34533] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 34533 890 2020-02-26 17:10:05,488 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted 891 2020-02-26 17:10:05,490 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 892 2020-02-26 17:10:05,490 INFO [Ping Checker] org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: TaskAttemptFinishingMonitor thread interrupted 893 2020-02-26 17:10:10,491 INFO [Thread-90] org.apache.hadoop.ipc.Server: Stopping server on 46625 894 2020-02-26 17:10:10,491 INFO [IPC Server listener on 46625] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 46625 895 2020-02-26 17:10:10,491 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 896 2020-02-26 17:10:10,494 INFO [Thread-90] org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.w.WebAppContext@1f939a0f{/,null,UNAVAILABLE}{/mapreduce} 897 2020-02-26 17:10:10,497 INFO [Thread-90] org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@7112ce6{HTTP/1.1,[http/1.1]}{0.0.0.0:0} 898 2020-02-26 17:10:10,498 INFO [Thread-90] org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@39109136{/static,jar:file:/data4/yarn/nm/filecache/12/3.0.0-cdh6.1.1-mr-framework.tar.gz/hadoop-yarn-common-3.0.0-cdh6.1.1.jar!/webapps/static,UNAVAILABLE} 899 End of LogType:syslog 900 ***********************************************************************
... View more
02-27-2020
12:26 PM
I went ahead and tried on the RM server as well and same darn error. I just don't get it. As you had asked for the log . Here it is: 296 java.io.tmpdir: /data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001/tmp
297 user.dir: /data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001
298 user.name: yarn
299 ************************************************************/
300 2020-02-26 17:09:49,953 INFO [main] org.apache.hadoop.security.SecurityUtil: Updating Configuration
301 2020-02-26 17:09:50,097 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens: [Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id { id: 9 cluster_timestamp: 1582677468069 } attemptId: 1 } keyId: -1009427054)]
302 2020-02-26 17:09:50,128 INFO [main] org.apache.hadoop.conf.Configuration: resource-types.xml not found
303 2020-02-26 17:09:50,128 INFO [main] org.apache.hadoop.yarn.util.resource.ResourceUtils: Unable to find 'resource-types.xml'.
304 2020-02-26 17:09:50,136 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
305 2020-02-26 17:09:50,138 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
306 2020-02-26 17:09:50,173 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output Committer Algorithm version is 2
307 2020-02-26 17:09:50,173 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
308 2020-02-26 17:09:50,627 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
309 2020-02-26 17:09:50,662 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
310 2020-02-26 17:09:50,830 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
311 2020-02-26 17:09:50,831 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
312 2020-02-26 17:09:50,831 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
313 2020-02-26 17:09:50,832 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
314 2020-02-26 17:09:50,832 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
315 2020-02-26 17:09:50,833 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
316 2020-02-26 17:09:50,834 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
317 2020-02-26 17:09:50,834 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
318 2020-02-26 17:09:50,864 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
319 2020-02-26 17:09:50,881 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
320 2020-02-26 17:09:50,895 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system [hdfs://nameservice1:8020]
321 2020-02-26 17:09:50,909 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
322 2020-02-26 17:09:50,946 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
323 2020-02-26 17:09:51,187 WARN [main] org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-mrappmaster.properties,hadoop-metrics2.properties
324 2020-02-26 17:09:51,244 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
325 2020-02-26 17:09:51,244 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
326 2020-02-26 17:09:51,252 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1582677468069_0009 to jobTokenSecretManager
327 2020-02-26 17:09:51,358 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1582677468069_0009 because: not enabled;
328 2020-02-26 17:09:51,374 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1582677468069_0009 = 472. Number of splits = 4
329 2020-02-26 17:09:51,375 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1582677468069_0009 = 1
330 2020-02-26 17:09:51,375 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from NEW to INITED
331 2020-02-26 17:09:51,376 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1582677468069_0009.
332 2020-02-26 17:09:51,399 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 100 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
333 2020-02-26 17:09:51,408 INFO [Socket Reader #1 for port 46625] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 46625
334 2020-02-26 17:09:51,577 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
335 2020-02-26 17:09:51,578 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
336 2020-02-26 17:09:51,578 INFO [IPC Server listener on 46625] org.apache.hadoop.ipc.Server: IPC Server listener on 46625: starting
337 2020-02-26 17:09:51,579 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at cdh-worker-3.private.cdhvcn.hiddenvcn.com/10.161.2.5:46625
338 2020-02-26 17:09:51,609 INFO [main] org.eclipse.jetty.util.log: Logging initialized @2635ms
339 2020-02-26 17:09:51,697 INFO [main] org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
340 2020-02-26 17:09:51,700 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
341 2020-02-26 17:09:51,705 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
342 2020-02-26 17:09:51,747 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
343 2020-02-26 17:09:51,747 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
344 2020-02-26 17:09:51,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
345 2020-02-26 17:09:51,749 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
346 2020-02-26 17:09:52,083 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
347 2020-02-26 17:09:52,084 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 45277
348 2020-02-26 17:09:52,085 INFO [main] org.eclipse.jetty.server.Server: jetty-9.3.20.v20170531
349 2020-02-26 17:09:52,122 INFO [main] org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@39109136{/static,jar:file:/data4/yarn/nm/filecache/12/3.0.0-cdh6.1.1-mr-framework.tar.gz/hadoop-yarn-common-3.0.0-cdh6.1.1.jar!/webapps/static,AVAILABLE}
350 2020-02-26 17:09:53,009 INFO [main] org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@1f939a0f{/,file:///data5/yarn/nm/usercache/hdfs/appcache/application_1582677468069_0009/container_e02_1582677468069_0009_01_000001/tmp/jetty-0.0.0.0-45277-mapreduce-_-any-7944828710115519068.dir/webapp/,AVAILABLE}{/mapreduce}
351 2020-02-26 17:09:53,017 INFO [main] org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@7112ce6{HTTP/1.1,[http/1.1]}{0.0.0.0:45277}
352 2020-02-26 17:09:53,017 INFO [main] org.eclipse.jetty.server.Server: Started @4043ms
353 2020-02-26 17:09:53,017 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app mapreduce started at 45277
354 2020-02-26 17:09:53,022 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue queueCapacity: 3000 scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler
355 2020-02-26 17:09:53,023 INFO [Socket Reader #1 for port 34533] org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 34533
356 2020-02-26 17:09:53,028 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
357 2020-02-26 17:09:53,028 INFO [IPC Server listener on 34533] org.apache.hadoop.ipc.Server: IPC Server listener on 34533: starting
358 2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
359 2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
360 2020-02-26 17:09:53,055 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
361 2020-02-26 17:09:53,060 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: 0% of the mappers will be scheduled using OPPORTUNISTIC containers
362 2020-02-26 17:09:53,154 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: maxContainerCapability: <memory:8192, vCores:2>
363 2020-02-26 17:09:53,154 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: queue: root.users.hdfs
364 2020-02-26 17:09:53,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
365 2020-02-26 17:09:53,159 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: The thread pool initial size is 10
366 2020-02-26 17:09:53,169 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from INITED to SETUP
367 2020-02-26 17:09:53,171 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
368 2020-02-26 17:09:53,182 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1582677468069_0009Job Transitioned from SETUP to RUNNING
369 2020-02-26 17:09:53,274 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1582677468069_0009, File: hdfs://nameservice1:8020/user/hdfs/.staging/job_1582677468069_0009/job_1582677468069_0009_1.jhist
370 2020-02-26 17:09:53,288 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from NEW to SCHEDULED
371 2020-02-26 17:09:53,311 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from NEW to SCHEDULED
372 2020-02-26 17:09:53,313 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from NEW to SCHEDULED
373 2020-02-26 17:09:53,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from NEW to SCHEDULED
374 2020-02-26 17:09:53,316 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_r_000000 Task Transitioned from NEW to SCHEDULED
375 2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
376 2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from NEW to UNASSIGNED
377 2020-02-26 17:09:53,317 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from NEW to UNASSIGNED
378 2020-02-26 17:09:53,318 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from NEW to UNASSIGNED
379 2020-02-26 17:09:53,318 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
380 2020-02-26 17:09:53,319 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>
381 2020-02-26 17:09:53,332 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:1024, vCores:1>
382 2020-02-26 17:09:54,158 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
383 2020-02-26 17:09:54,189 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=6 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:454656, vCores:63> knownNMs=4
384 2020-02-26 17:09:54,192 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:454656, vCores:63>
385 2020-02-26 17:09:54,192 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
386 2020-02-26 17:09:55,206 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
387 2020-02-26 17:09:55,209 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000002 to attempt_1582677468069_0009_m_000000_0
388 2020-02-26 17:09:55,211 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000003 to attempt_1582677468069_0009_m_000001_0
389 2020-02-26 17:09:55,212 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000004 to attempt_1582677468069_0009_m_000002_0
390 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000005 to attempt_1582677468069_0009_m_000003_0
391 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:450560, vCores:59>
392 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
393 2020-02-26 17:09:55,213 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1
394 2020-02-26 17:09:55,278 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-jar file on the remote FS is hdfs://nameservice1/user/hdfs/.staging/job_1582677468069_0009/job.jar
395 2020-02-26 17:09:55,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: The job-conf file on the remote FS is /user/hdfs/.staging/job_1582677468069_0009/job.xml
396 2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Adding #0 tokens and #1 secret keys for NM use for launching container
397 2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Size of containertokens_dob is 1
398 2020-02-26 17:09:55,285 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Putting shuffle token in serviceData
399 2020-02-26 17:09:55,308 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
400 2020-02-26 17:09:55,311 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
401 2020-02-26 17:09:55,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
402 2020-02-26 17:09:55,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
403 2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
404 2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
405 2020-02-26 17:09:55,315 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
406 2020-02-26 17:09:55,316 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
407 2020-02-26 17:09:55,318 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000002 taskAttempt attempt_1582677468069_0009_m_000000_0
408 2020-02-26 17:09:55,318 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000003 taskAttempt attempt_1582677468069_0009_m_000001_0
409 2020-02-26 17:09:55,318 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000004 taskAttempt attempt_1582677468069_0009_m_000002_0
410 2020-02-26 17:09:55,319 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000005 taskAttempt attempt_1582677468069_0009_m_000003_0
411 2020-02-26 17:09:55,321 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_0
412 2020-02-26 17:09:55,321 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_0
413 2020-02-26 17:09:55,321 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_0
414 2020-02-26 17:09:55,321 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_0
415 2020-02-26 17:09:55,404 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_0 : 13562
416 2020-02-26 17:09:55,405 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_0 : 13562
417 2020-02-26 17:09:55,405 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_0] using containerId: [container_e02_1582677468069_0009_01_000005 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
418 2020-02-26 17:09:55,408 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_0 : 13562
419 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
420 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_0] using containerId: [container_e02_1582677468069_0009_01_000003 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
421 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
422 2020-02-26 17:09:55,409 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_0] using containerId: [container_e02_1582677468069_0009_01_000002 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
423 2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
424 2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000003 Task Transitioned from SCHEDULED to RUNNING
425 2020-02-26 17:09:55,410 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000001 Task Transitioned from SCHEDULED to RUNNING
426 2020-02-26 17:09:55,411 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000000 Task Transitioned from SCHEDULED to RUNNING
427 2020-02-26 17:09:55,412 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_0 : 13562
428 2020-02-26 17:09:55,412 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_0] using containerId: [container_e02_1582677468069_0009_01_000004 on NM: [cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041]
429 2020-02-26 17:09:55,413 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 TaskAttempt Transitioned from ASSIGNED to RUNNING
430 2020-02-26 17:09:55,413 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1582677468069_0009_m_000002 Task Transitioned from SCHEDULED to RUNNING
431 2020-02-26 17:09:56,221 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=6 release= 0 newContainers=0 finishedContainers=4 resourcelimit=<memory:454656, vCores:63> knownNMs=4
432 2020-02-26 17:09:56,221 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000004
433 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000003
434 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000002
435 2020-02-26 17:09:56,222 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000002_0: [2020-02-26 17:09:55.814]Exception from container-launch.
436 Container id: container_e02_1582677468069_0009_01_000004
437 Exit code: 1
438 [2020-02-26 17:09:55.815]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
439 Last 4096 bytes of prelaunch.err :
440 Last 4096 bytes of stderr :
441 Error: Could not find or load main class NOT
442 [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
443 Last 4096 bytes of prelaunch.err :
444 Last 4096 bytes of stderr :
445 Error: Could not find or load main class NOT
446 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000005
447 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:454656, vCores:63>
448 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
449 2020-02-26 17:09:56,222 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1
450 2020-02-26 17:09:56,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
451 2020-02-26 17:09:56,226 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_0: [2020-02-26 17:09:55.826]Exception from container-launch.
452 Container id: container_e02_1582677468069_0009_01_000003
453 Exit code: 1
454 [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
455 Last 4096 bytes of prelaunch.err :
456 Last 4096 bytes of stderr :
457 Error: Could not find or load main class NOT
458 [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
459 Last 4096 bytes of prelaunch.err :
460 Last 4096 bytes of stderr :
461 Error: Could not find or load main class NOT
462 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
463 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_0: [2020-02-26 17:09:55.813]Exception from container-launch.
464 Container id: container_e02_1582677468069_0009_01_000002
465 Exit code: 1
466 [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
467 Last 4096 bytes of prelaunch.err :
468 Last 4096 bytes of stderr :
469 Error: Could not find or load main class NOT
470 [2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
471 Last 4096 bytes of prelaunch.err :
472 Last 4096 bytes of stderr :
473 Error: Could not find or load main class NOT
474 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
475 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_0: [2020-02-26 17:09:55.819]Exception from container-launch.
476 Container id: container_e02_1582677468069_0009_01_000005
477 Exit code: 1
478 [2020-02-26 17:09:55.852]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
479 Last 4096 bytes of prelaunch.err :
480 Last 4096 bytes of stderr :
481 Error: Could not find or load main class NOT
482 [2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
483 Last 4096 bytes of prelaunch.err :
484 Last 4096 bytes of stderr :
485 Error: Could not find or load main class NOT
486 2020-02-26 17:09:56,227 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_0 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-1.private.cdhvcn.hiddenvcn.com:8041
487 2020-02-26 17:09:56,228 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000004 taskAttempt attempt_1582677468069_0009_m_000002_0
488 2020-02-26 17:09:56,237 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000003 taskAttempt attempt_1582677468069_0009_m_000001_0
489 2020-02-26 17:09:56,240 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000002 taskAttempt attempt_1582677468069_0009_m_000000_0
490 2020-02-26 17:09:56,241 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000005 taskAttempt attempt_1582677468069_0009_m_000003_0
491 2020-02-26 17:09:56,243 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com
492 2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from NEW to UNASSIGNED
493 2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from NEW to UNASSIGNED
494 2020-02-26 17:09:56,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com
495 2020-02-26 17:09:56,245 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
496 2020-02-26 17:09:56,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on node cdh-worker-1.private.cdhvcn.hiddenvcn.com
497 2020-02-26 17:09:56,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted host cdh-worker-1.private.cdhvcn.hiddenvcn.com
498 2020-02-26 17:09:56,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from NEW to UNASSIGNED
499 2020-02-26 17:09:56,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_1 to list of failed maps
500 2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000001_1 to list of failed maps
501 2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000000_1 to list of failed maps
502 2020-02-26 17:09:56,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000003_1 to list of failed maps
503 2020-02-26 17:09:57,223 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0 HostLocal:3 RackLocal:1
504 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:339968, vCores:47> knownNMs=4
505 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=1 blacklistRemovals=0
506 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:339968, vCores:47>
507 2020-02-26 17:09:57,226 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
508 2020-02-26 17:09:58,231 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
509 2020-02-26 17:09:58,232 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000006, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
510 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
511 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000006 to attempt_1582677468069_0009_m_000002_1
512 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000007, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
513 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
514 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000007 to attempt_1582677468069_0009_m_000001_1
515 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000008, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
516 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
517 2020-02-26 17:09:58,233 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
518 2020-02-26 17:09:58,233 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000008 to attempt_1582677468069_0009_m_000000_1
519 2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000009, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-3.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.5:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
520 2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
521 2020-02-26 17:09:58,234 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000009 to attempt_1582677468069_0009_m_000003_1
522 2020-02-26 17:09:58,234 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
523 2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:335872, vCores:43>
524 2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
525 2020-02-26 17:09:58,235 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:4 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
526 2020-02-26 17:09:58,235 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
527 2020-02-26 17:09:58,236 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
528 2020-02-26 17:09:58,236 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
529 2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
530 2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
531 2020-02-26 17:09:58,237 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
532 2020-02-26 17:09:58,238 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000006 taskAttempt attempt_1582677468069_0009_m_000002_1
533 2020-02-26 17:09:58,238 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000009 taskAttempt attempt_1582677468069_0009_m_000003_1
534 2020-02-26 17:09:58,238 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000002_1
535 2020-02-26 17:09:58,238 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000007 taskAttempt attempt_1582677468069_0009_m_000001_1
536 2020-02-26 17:09:58,238 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000008 taskAttempt attempt_1582677468069_0009_m_000000_1
537 2020-02-26 17:09:58,238 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_1
538 2020-02-26 17:09:58,238 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_1
539 2020-02-26 17:09:58,238 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_1
540 2020-02-26 17:09:58,265 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_1 : 13562
541 2020-02-26 17:09:58,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_1] using containerId: [container_e02_1582677468069_0009_01_000009 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
542 2020-02-26 17:09:58,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
543 2020-02-26 17:09:58,273 INFO [ContainerLauncher #9] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_1 : 13562
544 2020-02-26 17:09:58,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_1] using containerId: [container_e02_1582677468069_0009_01_000007 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
545 2020-02-26 17:09:58,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
546 2020-02-26 17:09:58,280 INFO [ContainerLauncher #8] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000002_1 : 13562
547 2020-02-26 17:09:58,280 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000002_1] using containerId: [container_e02_1582677468069_0009_01_000006 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
548 2020-02-26 17:09:58,281 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
549 2020-02-26 17:09:58,289 INFO [ContainerLauncher #3] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_1 : 13562
550 2020-02-26 17:09:58,289 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_1] using containerId: [container_e02_1582677468069_0009_01_000008 on NM: [cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041]
551 2020-02-26 17:09:58,289 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 TaskAttempt Transitioned from ASSIGNED to RUNNING
552 2020-02-26 17:09:59,238 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=3 resourcelimit=<memory:338944, vCores:46> knownNMs=4
553 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000008
554 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000009
555 2020-02-26 17:09:59,239 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_1: [2020-02-26 17:09:58.706]Exception from container-launch.
556 Container id: container_e02_1582677468069_0009_01_000008
557 Exit code: 1
558 [2020-02-26 17:09:58.708]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
559 Last 4096 bytes of prelaunch.err :
560 Last 4096 bytes of stderr :
561 Error: Could not find or load main class NOT
562 [2020-02-26 17:09:58.708]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
563 Last 4096 bytes of prelaunch.err :
564 Last 4096 bytes of stderr :
565 Error: Could not find or load main class NOT
566 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000007
567 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:338944, vCores:46>
568 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
569 2020-02-26 17:09:59,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
570 2020-02-26 17:09:59,239 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
571 2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_1: [2020-02-26 17:09:58.721]Exception from container-launch.
572 Container id: container_e02_1582677468069_0009_01_000009
573 Exit code: 1
574 [2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
575 Last 4096 bytes of prelaunch.err :
576 Last 4096 bytes of stderr :
577 Error: Could not find or load main class NOT
578 [2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
579 Last 4096 bytes of prelaunch.err :
580 Last 4096 bytes of stderr :
581 Error: Could not find or load main class NOT
582 2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
583 2020-02-26 17:09:59,240 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_1: [2020-02-26 17:09:58.712]Exception from container-launch.
584 Container id: container_e02_1582677468069_0009_01_000007
585 Exit code: 1
586 [2020-02-26 17:09:58.732]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
587 Last 4096 bytes of prelaunch.err :
588 Last 4096 bytes of stderr :
589 Error: Could not find or load main class NOT
590 [2020-02-26 17:09:58.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
591 Last 4096 bytes of prelaunch.err :
592 Last 4096 bytes of stderr :
593 Error: Could not find or load main class NOT
594 2020-02-26 17:09:59,241 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
595 2020-02-26 17:09:59,241 INFO [ContainerLauncher #0] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000008 taskAttempt attempt_1582677468069_0009_m_000000_1
596 2020-02-26 17:09:59,242 INFO [ContainerLauncher #2] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000009 taskAttempt attempt_1582677468069_0009_m_000003_1
597 2020-02-26 17:09:59,244 INFO [ContainerLauncher #4] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000007 taskAttempt attempt_1582677468069_0009_m_000001_1
598 2020-02-26 17:09:59,245 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com
599 2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from NEW to UNASSIGNED
600 2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com
601 2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from NEW to UNASSIGNED
602 2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 3 failures on node cdh-worker-3.private.cdhvcn.hiddenvcn.com
603 2020-02-26 17:09:59,246 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Blacklisted host cdh-worker-3.private.cdhvcn.hiddenvcn.com
604 2020-02-26 17:09:59,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from NEW to UNASSIGNED
605 2020-02-26 17:09:59,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000000_2 to list of failed maps
606 2020-02-26 17:09:59,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000003_2 to list of failed maps
607 2020-02-26 17:09:59,248 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000001_2 to list of failed maps
608 2020-02-26 17:10:00,239 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
609 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:229376, vCores:32> knownNMs=4
610 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=1 blacklistRemovals=0
611 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Ignore blacklisting set to true. Known: 4, Blacklisted: 2, 50%
612 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000006
613 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:229376, vCores:32>
614 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
615 2020-02-26 17:10:00,243 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000002_1: [2020-02-26 17:09:58.725]Exception from container-launch.
616 Container id: container_e02_1582677468069_0009_01_000006
617 Exit code: 1
618 [2020-02-26 17:09:58.734]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
619 Last 4096 bytes of prelaunch.err :
620 Last 4096 bytes of stderr :
621 Error: Could not find or load main class NOT
622 [2020-02-26 17:09:58.734]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
623 Last 4096 bytes of prelaunch.err :
624 Last 4096 bytes of stderr :
625 Error: Could not find or load main class NOT
626 2020-02-26 17:10:00,243 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
627 2020-02-26 17:10:00,244 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_1 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-3.private.cdhvcn.hiddenvcn.com:8041
628 2020-02-26 17:10:00,244 INFO [ContainerLauncher #5] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_COMPLETED for container container_e02_1582677468069_0009_01_000006 taskAttempt attempt_1582677468069_0009_m_000002_1
629 2020-02-26 17:10:00,246 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000002_2 TaskAttempt Transitioned from NEW to UNASSIGNED
630 2020-02-26 17:10:00,247 INFO [Thread-57] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1582677468069_0009_m_000002_2 to list of failed maps
631 2020-02-26 17:10:01,244 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:4 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:8 ContRel:0 HostLocal:3 RackLocal:1
632 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=3 finishedContainers=0 resourcelimit=<memory:451584, vCores:60> knownNMs=4
633 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: Update the blacklist for application_1582677468069_0009: blacklistAdditions=0 blacklistRemovals=2
634 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 3
635 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000010, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
636 2020-02-26 17:10:01,250 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
637 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000010 to attempt_1582677468069_0009_m_000000_2
638 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000011, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
639 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
640 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000011 to attempt_1582677468069_0009_m_000003_2
641 2020-02-26 17:10:01,251 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
642 2020-02-26 17:10:01,251 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000012, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
643 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
644 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000012 to attempt_1582677468069_0009_m_000001_2
645 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating schedule, headroom=<memory:451584, vCores:60>
646 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow start threshold not met. completedMapsForReduceSlowstart 4
647 2020-02-26 17:10:01,252 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:1 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:11 ContRel:0 HostLocal:3 RackLocal:1
648 2020-02-26 17:10:01,252 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
649 2020-02-26 17:10:01,254 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
650 2020-02-26 17:10:01,254 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
651 2020-02-26 17:10:01,255 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapred.JobConf: Task java-opts do not specify heap size. Setting task attempt jvm max heap size to -Xmx820m
652 2020-02-26 17:10:01,255 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED
653 2020-02-26 17:10:01,255 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000010 taskAttempt attempt_1582677468069_0009_m_000000_2
654 2020-02-26 17:10:01,255 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000011 taskAttempt attempt_1582677468069_0009_m_000003_2
655 2020-02-26 17:10:01,255 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_e02_1582677468069_0009_01_000012 taskAttempt attempt_1582677468069_0009_m_000001_2
656 2020-02-26 17:10:01,256 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000003_2
657 2020-02-26 17:10:01,255 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000000_2
658 2020-02-26 17:10:01,256 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1582677468069_0009_m_000001_2
659 2020-02-26 17:10:01,267 INFO [ContainerLauncher #7] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000003_2 : 13562
660 2020-02-26 17:10:01,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000003_2] using containerId: [container_e02_1582677468069_0009_01_000011 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
661 2020-02-26 17:10:01,267 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
662 2020-02-26 17:10:01,271 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000001_2 : 13562
663 2020-02-26 17:10:01,271 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000001_2] using containerId: [container_e02_1582677468069_0009_01_000012 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
664 2020-02-26 17:10:01,271 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
665 2020-02-26 17:10:01,273 INFO [ContainerLauncher #6] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1582677468069_0009_m_000000_2 : 13562
666 2020-02-26 17:10:01,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1582677468069_0009_m_000000_2] using containerId: [container_e02_1582677468069_0009_01_000010 on NM: [cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041]
667 2020-02-26 17:10:01,273 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000000_2 TaskAttempt Transitioned from ASSIGNED to RUNNING
668 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1582677468069_0009: ask=1 release= 0 newContainers=4 finishedContainers=3 resourcelimit=<memory:450560, vCores:59> knownNMs=4
669 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000011
670 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000012
671 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_e02_1582677468069_0009_01_000010
672 2020-02-26 17:10:02,257 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000003_2: [2020-02-26 17:10:01.685]Exception from container-launch.
673 Container id: container_e02_1582677468069_0009_01_000011
674 Exit code: 1
675 [2020-02-26 17:10:01.686]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
676 Last 4096 bytes of prelaunch.err :
677 Last 4096 bytes of stderr :
678 Error: Could not find or load main class NOT
679 [2020-02-26 17:10:01.691]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
680 Last 4096 bytes of prelaunch.err :
681 Last 4096 bytes of stderr :
682 Error: Could not find or load main class NOT
683 2020-02-26 17:10:02,257 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 4
684 2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000003_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041
685 2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000001_2: [2020-02-26 17:10:01.698]Exception from container-launch.
686 Container id: container_e02_1582677468069_0009_01_000012
687 Exit code: 1
688 [2020-02-26 17:10:01.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
689 Last 4096 bytes of prelaunch.err :
690 Last 4096 bytes of stderr :
691 Error: Could not find or load main class NOT
692 [2020-02-26 17:10:01.733]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
693 Last 4096 bytes of prelaunch.err :
694 Last 4096 bytes of stderr :
695 Error: Could not find or load main class NOT
696 2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning container Container: [ContainerId: container_e02_1582677468069_0009_01_000013, AllocationRequestId: -1, Version: 0, NodeId: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041, NodeHttpAddress: cdh-worker-2.private.cdhvcn.hiddenvcn.com:8042, Resource: <memory:1024, vCores:1>, Priority: 5, Token: Token { kind: ContainerToken, service: 10.161.2.3:8041 }, ExecutionType: GUARANTEED, ] to fast fail map
697 2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from earlierFailedMaps
698 2020-02-26 17:10:02,258 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1582677468069_0009_m_000001_2 transitioned from state RUNNING to FAILED, event type is TA_CONTAINER_COMPLETED and nodeId=cdh-worker-2.private.cdhvcn.hiddenvcn.com:8041
699 2020-02-26 17:10:02,258 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_e02_1582677468069_0009_01_000013 to attempt_1582677468069_0009_m_000002_2
700 2020-02-26 17:10:02,259 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1582677468069_0009_m_000000_2: [2020-02-26 17:10:01.688]Exception from container-launch.
701 Container id: container_e02_1582677468069_0009_01_000010
702 Exit code: 1
... View more
02-26-2020
09:14 AM
Please help I've spent 3 days and still can't get an example to run including pi, terasort or even wordcount
New installation and have turned on HA for yarn and also hdfs, and installed the yarn MapReduce Framework JARs as well. Cluster looks health as seen below:
If I try and run any of the examples ...they fail.
[hdfs@cdh-utility-1 opc]$ hadoop jar /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 4 4 WARNING: Use "yarn jar" to launch YARN applications. Number of Maps = 4 Samples per Map example.com= 4 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Starting Job 20/02/26 17:09:47 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1582677468069_0009 20/02/26 17:09:47 INFO input.FileInputFormat: Total input files to process : 4 20/02/26 17:09:47 INFO mapreduce.JobSubmitter: number of splits:4 20/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.zk-address is deprecated. Instead, use hadoop.zk.address 20/02/26 17:09:47 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 20/02/26 17:09:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1582677468069_0009 20/02/26 17:09:48 INFO mapreduce.JobSubmitter: Executing with tokens: [] 20/02/26 17:09:48 INFO conf.Configuration: resource-types.xml not found 20/02/26 17:09:48 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 20/02/26 17:09:48 INFO impl.YarnClientImpl: Submitted application application_1582677468069_0009 20/02/26 17:09:48 INFO mapreduce.Job: The url to track the job: http://cdh-master-1.private.cdhvcn.examplevcn.com:8088/proxy/application_1582677468069_0009/ 20/02/26 17:09:48 INFO mapreduce.Job: Running job: job_1582677468069_0009 20/02/26 17:09:54 INFO mapreduce.Job: Job job_1582677468069_0009 running in uber mode : false 20/02/26 17:09:54 INFO mapreduce.Job: map 0% reduce 0% 20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000002_0, Status : FAILED [2020-02-26 17:09:55.814]Exception from container-launch. Container id: container_e02_1582677468069_0009_01_000004 Exit code: 1
[2020-02-26 17:09:55.815]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT
[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT
20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000001_0, Status : FAILED [2020-02-26 17:09:55.826]Exception from container-launch. Container id: container_e02_1582677468069_0009_01_000003 Exit code: 1
[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT
[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT
20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000000_0, Status : FAILED [2020-02-26 17:09:55.813]Exception from container-launch. Container id: container_e02_1582677468069_0009_01_000002 Exit code: 1
[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT
[2020-02-26 17:09:55.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT
20/02/26 17:09:56 INFO mapreduce.Job: Task Id : attempt_1582677468069_0009_m_000003_0, Status : FAILED [2020-02-26 17:09:55.819]Exception from container-launch. Container id: container_e02_1582677468069_0009_01_000005 Exit code: 1
[2020-02-26 17:09:55.852]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT
[2020-02-26 17:09:55.853]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : Error: Could not find or load main class NOT
... View more
Labels:
- Labels:
-
Apache YARN
-
Cloudera Manager
-
MapReduce