question Re: Yarn node manager not starting. in Archives of Support Questions (Read Only)

Yarn node manager not starting.

maruthu_jm — Tue, 15 Nov 2016 22:33:52 GMT

Re: Yarn node manager not starting.

KuldeepK — Tue, 15 Nov 2016 22:39:32 GMT

Can you please share logs?

Re: Yarn node manager not starting.

maruthu_jm — Tue, 15 Nov 2016 23:24:35 GMT

https://drive.google.com/file/d/0B8mfisW8zEfLdmxqSXktZWtWcjA/view?usp=sharing https://drive.google.com/file/d/0B8mfisW8zEfLMzNkQ2Q0SE4zS0U/view?usp=sharing

Shared yarn resource manager logs. expecting any other logs ?

Re: Yarn node manager not starting.

hamid_zorgani — Wed, 16 Nov 2016 01:03:52 GMT

through you log you have some corrupted files

 hdfs fsck /   double check with this command

Please find here how to check that : http://stackoverflow.com/questions/19205057/how-to-fix-corrupt-hdfs-files/19216037#19216037

Re: Yarn node manager not starting.

maruthu_jm — Wed, 16 Nov 2016 22:14:55 GMT

No corrupt blocks.

Status: HEALTHY Total size: 2286120271 B Total dirs: 1826 Total files: 3378 Total symlinks: 0 (Files currently being written: 9) Total blocks (validated): 2905 (avg. block size 786960 B) (Total open file blocks (not validated): 3) Minimally replicated blocks: 2905 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 2905 (100.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 13902 (82.71554 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Wed Nov 16 14:10:34 UTC 2016 in 2531 milliseconds

Re: Yarn node manager not starting.

hamid_zorgani — Thu, 17 Nov 2016 00:17:30 GMT

Try to restart ambari-agent , or provide logs for yarn-nodemanager /var/log/hadoop-yarn/yarn/yarn-yarn-nodemanager-xxx.com.log

Re: Yarn node manager not starting.

maruthu_jm — Thu, 17 Nov 2016 08:10:03 GMT

Log :

https://drive.google.com/open?id=0B8mfisW8zEfLT2lLUkZ6M0Y5Vzg

Re: Yarn node manager not starting.

mugdha — Thu, 17 Nov 2016 11:32:17 GMT

@MARUTHU PANDIYAN JAYARAMAN

If nodemanager.recovery.enabled is set to true, please set it to false and try starting the nodemanager. This should get you unblocked. NM is failing during recovery:

2016-11-16 23:49:26,076 INFO  recovery.NMLeveldbStateStoreService (NMLeveldbStateStoreService.java:initStorage(927)) - Using state database at /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state for recovery
2016-11-16 23:49:26,299 INFO  recovery.NMLeveldbStateStoreService$LeveldbLogger (NMLeveldbStateStoreService.java:log(969)) - Recovering log #362
2016-11-16 23:49:26,306 INFO  recovery.NMLeveldbStateStoreService$LeveldbLogger (NMLeveldbStateStoreService.java:log(969)) - Delete type=3 #361


2016-11-16 23:49:26,306 INFO  recovery.NMLeveldbStateStoreService$LeveldbLogger (NMLeveldbStateStoreService.java:log(969)) - Delete type=0 #362


2016-11-16 23:49:26,681 INFO  service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService failed in state INITED; cause: org.iq80.leveldb.DBException: Invalid argument: not an sstable (bad magic number)
org.iq80.leveldb.DBException: Invalid argument: not an sstable (bad magic number)

Re: Yarn node manager not starting.

maruthu_jm — Thu, 17 Nov 2016 22:28:23 GMT

Problem solved. Node manager is up after setting this flag as false. Thank you Mugdha.

Re: Yarn node manager not starting.

Sak25agr — Tue, 29 Aug 2023 19:56:34 GMT

I am also Getting the error " localhost: ERROR: Cannot set priority of nodemanager process 41819". Can someone provide me answer

Re: Yarn node manager not starting.

Sak25agr — Tue, 29 Aug 2023 20:01:09 GMT

https://drive.google.com/file/d/1ApuElSIlU0zmDmI12pQ0PfJN8B1-du2k/view?usp=share_link

Here is the log file