Why is CPU_MILLISECONDS,PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES = 0?

[#1]

Dear Experts,

for my MR-Jobs aren't get Values for the Counters: CPU_MILLISECONDS,PHYSICAL_MEMORY_BYTES,VIRTUAL_MEMORY_BYTES

Can sameone explain me, why this Counters are allways 0? Is my configuration incorrectly?

The Test Job is the WordCount v1 example from Apache.

I use Java JDK1.8 and Hadoop 2.8.0 and a small network from 4 computers for my study work.

juniper is the Master on him there run 5 daemons.

NameNode SecondaryNameNode ResourceManager ApplicationHistoryServer JobHistoryServer

And on the three Workers run:

DataNode NodeManager

core-site.xml 

<configuration>
  <property>
  <name>fs.defaultFS</name>
  <value>hdfs://juniper:44001</value>
  </property>
</configuration>

mapred-site.xml

<configuration>
   <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
   </property>
</configuration>

yarn-site.xml

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>juniper</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>    
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.class</name>            
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.app.mapreduce.am.staging-dir</name>
        <value>/tmp/hadoop-yarn/staging</value>
    </property>
    <!-- YARN Timeline Server -->
    <property>
        <name>yarn.timeline-service.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.timeline-service.hostname</name>
        <value>juniper</value>
    </property>
    <property>
        <name>yarn.resourcemanager.system-metrics-publisher.enabled</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.timeline-service.generic-application-history.enabled</name>
        <value>true</value>
    </property>
</configuration>

I hope you can help me. Thank you!!!

[#2]

@Dennis Hude

What operating system are you using? CPU_MILLISECONDS, PHYSICAL_MEMORY_BYTES and VIRTUAL_MEMORY_BYTES are collected by ResourceCalculatorProcessTree which has two implementations: ProcfsBasedProcessTree (for Linux that uses /proc) and WindowsBasedProcessTree. You can check the syslogs from your application containers to see which one is used, grep for this message: "Using ResourceCalculatorProcessTree :". Not having an initiated ResourceCalculatorProcessTree instance explains your 0 values. If you have one, there might be a problem with that which requires further investigation (the container logs can help with that, too).

[#3]

Thank you gnovak.
The uni computers run with mac os 10.6.8 and I found that mac don't have a /proc folder. That will be the problem. 😞
Futhermore the same configuration run on my Lubuntu singleNode PC and in the VirtaulBox (4x Lubuntu). There is all fine.

Is there a solution without write a own ResourceCalculatorProcessTree implementation? Certainly not, right?

[#4]

@Dennis Hude

Unfortunately I don't think there is, Hadoop is mainly designed for Linux clusters. But if you're interested, you can definitely try to write a ResourceCalculatorProcessTree implementation for Mac or just open a Jira ticket for it and see if someone else is interested.

[#5]

thanks 🙂