Support Questions
Find answers, ask questions, and share your expertise

Nodemanager not starting after expanding volume groups

Nodemanager not starting after expanding volume groups

Explorer

I have 4 datanodes, in which I had to increase the space in vg /tmp and vg /var/log. I renamed the vg, created the new one, moved everything the new one.

So I have the same vg, but after starting the cluster, the nodemanager became unreachable (the other services works well).

The error log:

 

Thread Thread[Timer-2,5,main] threw an Exception.
java.lang.IllegalArgumentException: Wrong FS: hdfs://nameservice1:8020/user/history/done_intermediate/hive/job_1557996286771_33621_conf.xml, expected: hdfs://nameservice1
	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:662)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:222)
	at org.apache.hadoop.hdfs.DistributedFileSystem.access$000(DistributedFileSystem.java:114)
	at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)
	at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1262)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1262)
	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1418)
	at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:499)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:351)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:341)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292)
	at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.copy(KilledHistoryService.java:210)
	at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.access$300(KilledHistoryService.java:85)
	at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:138)
	at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler$1.run(KilledHistoryService.java:125)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
	at org.apache.hadoop.mapreduce.v2.hs.KilledHistoryService$FlagFileHandler.run(KilledHistoryService.java:125)
	at java.util.TimerThread.mainLoop(Timer.java:555)
	at java.util.TimerThread.run(Timer.java:505)
View Log File
host4	ERROR	October 15, 2019 11:40 PM	NodeManager	
RECEIVED SIGNAL 15: SIGTERM
View Log File
master3	ERROR	October 15, 2019 11:40 PM	JobHistoryServer	
RECEIVED SIGNAL 15: SIGTERM
View Log File
master1	ERROR	October 15, 2019 11:40 PM	ResourceManager	
RECEIVED SIGNAL 15: SIGTERM
View Log File
host3	ERROR	October 15, 2019 11:40 PM	NodeManager	
RECEIVED SIGNAL 15: SIGTERM
View Log File
master1	ERROR	October 15, 2019 11:40 PM	AbstractDelegationTokenSecretManager	
ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
View Log File
master1	ERROR	October 15, 2019 11:40 PM	AbstractDelegationTokenSecretManager	
ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
View Log File
master1	ERROR	October 15, 2019 11:40 PM	AbstractDelegationTokenSecretManager	
ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
View Log File
host2	ERROR	October 15, 2019 11:40 PM	NodeManager	
RECEIVED SIGNAL 15: SIGTERM
View Log File

 

I already tried to restart the cluster, decomissioning and recomissionig but nothing works.

Please some help to get started?