NODEMANAGERs are going to unknown state

[#1]

Hello @rblough @Shelton 

I have added 5 slaves to cloudera manager and deployed slave services and started them.

But for NodeManager's we are seeing unknown state even after restarting multiple times.

I could see below logs,

2020-10-02 08:52:43,149 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: RECEIVED SIGNAL 15: SIGTERM
2020-10-02 08:56:16,234 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config.

Could you please help me if i am missing anything ?

Regards,

Vinod

[#2]

@kvinod 

If you have installed Yarn and MRV2 can you check the value of the below parameter in the yarn-site.xml 

yarn.nodemanager.aux-services

stop the services and change it to look like below

<property>
 <name>yarn.nodemanager.aux-services</name>
 <value>mapreduce_shuffle</value>
</property>

Restart and let me know

[#3]

Hello @Shelton 

Yes we are using YARN and other Nodemanagers are up and running with same configurations but for newly added nodes we are facing above logs and they are going to unknown state.

Yes i verified the above parameter but i dont see the above parameter but in yarn configurations page,

I can see "Enable Shuffle Auxiliary Service" was enabled.

Please suggest me your comments.

Thanks,

Vinod

[#4]

Hello @Shelton 

Any suggestions ?

My only doubt is, As other nodemanagers are running fine in the cluster why these newly added nodemanagers were going to unknown state?

Regards,

Vinod

[#5]

Hello @Shelton 

I have added above property in yarn-site.xml for nodemanager and restarted, Still i see same issue and same logs in nodemanager and it is in unknown state,

WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config

Please give me your valuable response.

Thanks,

Vinod

[#6]

Hello Team,

Can anyone please help me with your comments.

Thanks,

Vinod