Reply
New Contributor
Posts: 5
Registered: ‎08-08-2014

mapreduce success ,but appear --process information unavailable,

Hi

I have study CDH for over a year from cdh4.2 to cdh5.0.3.

Now I Meet a problem.

Each time,I run mapreduce(V2), which is successful,but it always appears one unavailable process,as follow

28236 DataNode
25262 QuorumPeerMain
28793 Jps
28638 -- process information unavailable
28399 NodeManager

I kill it under /tmp/hsperfdata_yarn/,but it still appears after runing mapreduce for each time.

I found the YarnChild didn't shut down automatic after mapreduce.

but it don't appear deploymenting by tarball with the same configuration file.

I have studied this problem for two weeks, but it doesn't work.

 

Expert Contributor
Posts: 101
Registered: ‎01-24-2014

Re: mapreduce success ,but appear --process information unavailable,

I can see you are using Jps to check for the process,

 

jps is informed by java's own registry of processes[1][2]

 

if you use linux built in commands like ps you could confirm whether this was a problem with the jvm not unregistering itself with jstatd and the rmiregistry (which will happen when you kill -9 a jvm) or if it is a zombie jvm process not responding.

 

ps -ef | grep <pid>

 

for you example below that would be:

 

ps -ef | grep 28638

 

if the process is there with this command, then it's a zombie process

if the process is not there with this command, then it's just that the jvm didn't unregister itself.

 

 

[1]http://docs.oracle.com/javase/7/docs/technotes/tools/solaris/rmiregistry.html

[2]http://docs.oracle.com/javase/7/docs/technotes/tools/share/jstatd.html

New Contributor
Posts: 5
Registered: ‎08-08-2014

Re: mapreduce success ,but appear --process information unavailable,

Sorry

 

I used the  ps -ef | grep  ** command ,found that the process is not there.

then it's just that the jvm didn't unregister itself.

 

but how to solve it.

Each virtual machine memory is 4G in my hadoop environment.

I set:

HADOOP_HEAPSIZE=3000

HADOOP_OPTS  = 3g

mapreduce.map.java.opts       -Xmx1024M

mapreduce.reduce.java.opts   -Xmx1024M

mapred.child.java.opts           -Xmx1024M

 

 

 

Expert Contributor
Posts: 101
Registered: ‎01-24-2014

Re: mapreduce success ,but appear --process information unavailable,

Okay we know then that likely the task is getting killed by something else. I would suspect the node manager in this case as a first place to look. maybe look at the logs and see if there is a corresponding kill command executed?  unfortunately i don't have much experience with YARN, so I would defer to anyone that has MRv2 experience. 

New Contributor
Posts: 5
Registered: ‎08-08-2014

Re: mapreduce success ,but appear --process information unavailable,

Hi

I viewed the log,found no ERROR log.

But  some logs caught my eyes,as follow:

org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 17716 for container-id container_1408962713994_0001_01_000002: 89.9 MB of 2 GB physical memory used; 1.5 GB of 4.2 GB virtual memory used
2014-08-26 09:19:16,010 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 17716 for container-id container_1408962713994_0001_01_000002: 515.1 MB of 2 GB physical memory used; 1.5 GB of 4.2 GB virtual memory used
2014-08-26 09:19:16,599 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Stopping container with container Id: container_1408962713994_0001_01_000002
2014-08-26 09:19:16,600 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=root IP=192.168.200.38       OPERATION=Stop Container Request        TARGET=ContainerManageImpl      RESULT=SUCCESS  APPID=application_1408962713994_0001    CONTAINERID=container_1408962713994_0001_01_000002
2014-08-26 09:19:16,601 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1408962713994_0001_01_000002 transitioned from RUNNING to KILLING
2014-08-26 09:19:16,601 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1408962713994_0001_01_000002
2014-08-26 09:19:16,617 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1408962713994_0001_01_000002 is : 1432014-08-26 09:19:16,644 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1408962713994_0001_01_000002 transitioned from KILLING to CONTAINER_CLEANEDUP_AFTER_KILL
2014-08-26 09:19:16,645 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /hadoop/yarn/local/usercache/root/appcache/application_1408962713994_0001/container_1408962713994_0001_01_000002
2014-08-26 09:19:16,647 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=root OPERATION=Container Finished - Killed   TARGET=ContainerImpl    RESULT=SUCCESS  APPID=application_1408962713994_0001    CONTAINERID=container_1408962713994_0001_01_000002
2014-08-26 09:19:16,650 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1408962713994_0001_01_000002 transitioned from CONTAINER_CLEANEDUP_AFTER_KILL to DONE
2014-08-26 09:19:16,650 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1408962713994_0001_01_000002 from application application_1408962713994_0001
2014-08-26 09:19:16,650 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_1408962713994_0001_01_000002 for log-aggregation
2014-08-26 09:19:16,650 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1408962713994_0001
2014-08-26 09:19:17,603 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed container container_1408962713994_0001_01_000002
2014-08-26 09:19:19,011 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1408962713994_0001_01_000002
2014-08-26 09:19:29,629 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1408962713994_0001 transitioned from RUNNING to APPLICATION_RESOURCES_CLEANINGUP
2014-08-26 09:19:29,630 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_STOP for appId application_1408962713994_0001
2014-08-26 09:19:29,630 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /hadoop/yarn/local/usercache/root/appcache/application_1408962713994_0001
2014-08-26 09:19:29,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Application application_1408962713994_0001 transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2014-08-26 09:19:29,631 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Application just finished : application_1408962713994_0001
2014-08-26 09:19:29,632 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Starting aggregate log-file for app application_1408962713994_0001 at /var/log/hadoop-yarn/apps/root/logs/application_1408962713994_0001/datanode3_40778.tmp
2014-08-26 09:19:29,734 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Uploading logs for container container_1408962713994_0001_01_000002. Current good log dirs are /hadoop/yarn/logs
2014-08-26 09:19:29,737 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting path : /hadoop/yarn/logs/application_1408962713994_0001
2014-08-26 09:19:29,901 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Finished aggregate log-file for app application_1408962713994_0001

 

I  searched with google, someone said it's yarn oom problem.

I modified the jvm parameters for a lot of time according the method,but all didn't work still.

 

Highlighted
New Contributor
Posts: 5
Registered: ‎08-08-2014

Re: mapreduce success ,but appear --process information unavailable,

it has not been solved so far.

 

 

Announcements