question Re: JournalNode ( HDFS ) restarting all the time in Archives of Support Questions (Read Only)

JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Sun, 18 Aug 2019 04:46:55 GMT

in our ambari cluster , we have a very strange problem

we restart all servers - master01-03

and on each master server we start the services from beginning according to the right order

first on all masters we start the zookeeper server

then on all masters we start the JournalNode

but we notice that on the last master machine - JournalNode restarting evry 10-20 seconds

and on all other machines - JournalNode is stable

please advice why this happend ?

Re: JournalNode ( HDFS ) restarting all the time

sankar_tumuluru — Thu, 25 Jan 2018 04:21:15 GMT

@Michael Bronson can you share the log file? It may tell you the root cause.

Re: JournalNode ( HDFS ) restarting all the time

jsensharma — Thu, 25 Jan 2018 05:18:34 GMT

@Michael Bronson

Can you please share the following log file content so that we can see the cause of failure? The latest log of the JournalNode process where it is keep restarting.

# ls -lart /var/log/hadoop/hdfs/hadoop-hdfs-journalnode*.log
# ls -lart /var/log/hadoop/hdfs/hadoop-hdfs-journalnode*.out

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 05:22:55 GMT

 grep -i error /var/log/hadoop/hdfs/hadoop-hdfs-journalnode-master03.sys57.com.log
2018-01-24 19:05:38,295 ERROR server.JournalNode (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM
2018-01-24 19:53:09,054 ERROR server.JournalNode (LogAdapter.java:error(69)) - RECEIVED SIGNAL 15: SIGTERM
2018-01-24 21:03:05,331 WARN  ipc.Server (Server.java:processResponse(1273)) - IPC Server handler 1 on 8485, call org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol.format from :49400 Call#4 Retry#0: output erro

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 05:25:14 GMT

grep -i war  /var/log/hadoop/hdfs/hadoop-hdfs-journalnode-master03.sys573.com.log
2018-01-24 19:03:41,115 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://master02.sys573.com:6188/ws/v1/timeline/metrics
2018-01-24 19:28:08,819 WARN  mortbay.log (Slf4jLog.java:warn(76)) - Can't reuse /tmp/Jetty_0_0_0_0_8480_journal____.8g4awa, using /tmp/Jetty_0_0_0_0_8480_journal____.8g4awa_661614759039131704
2018-01-24 19:29:18,310 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics
2018-01-24 19:30:28,393 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics
2018-01-24 19:56:39,690 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics
2018-01-24 19:59:38,233 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics
2018-01-24 20:29:58,228 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics
2018-01-24 20:37:29,599 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics
2018-01-24 21:00:28,236 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics
2018-01-24 21:03:05,331 WARN  ipc.Server (Server.java:processResponse(1273)) - IPC Server handler 1 on 8485, call org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocol.format from :49400 Call#4 Retry#0: output error
2018-01-24 21:16:59,654 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics
2018-01-24 21:21:08,278 WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics

Re: JournalNode ( HDFS ) restarting all the time

sankar_tumuluru — Thu, 25 Jan 2018 05:34:05 GMT

try to restart the ambari metrics service and check the error goes off

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 05:38:45 GMT

OK I will , need to wait couple min to see if it restarted as in the previos one

Re: JournalNode ( HDFS ) restarting all the time

jsensharma — Thu, 25 Jan 2018 05:42:29 GMT

@Michael Bronson

As we see "null:6188" hostname for the AMS collector in the Journal node output.

WARN  timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:putMetrics(349)) - Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics

So i guess somehow the following file does not have the correct AMS collector host reflected there.

# grep 'journal' /etc/hadoop/conf/hadoop-metrics2.properties
journalnode.sink.timeline.collector.hosts=amshost.example.co

Do you have any issue in starting AMS collector as well?

Can you please check the following path on ambari UI

Ambari UI --> HDFS --> Configs --> Advanced --> "Advanced hadoop-metrics2.properties"  --> "hadoop-metrics2.properties template"

Does it has the following section or missing somehow:

datanode.sink.timeline.collector.hosts={{ams_collector_hosts}}
resourcemanager.sink.timeline.collector.hosts={{ams_collector_hosts}}
nodemanager.sink.timeline.collector.hosts={{ams_collector_hosts}}
jobhistoryserver.sink.timeline.collector.hosts={{ams_collector_hosts}}
journalnode.sink.timeline.collector.hosts={{ams_collector_hosts}}
maptask.sink.timeline.collector.hosts={{ams_collector_hosts}}
reducetask.sink.timeline.collector.hosts={{ams_collector_hosts}}
applicationhistoryserver.sink.timeline.collector.hosts={{ams_collector_hosts}}

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 05:46:14 GMT

now the journalnode stop for restarting when I start the matrix , bur when now I start the name-node we get

- ERROR namenode.NameNode (NameNode.java:main(1774)) - Failed to start namenode.
java.io.FileNotFoundException: No valid image files found

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 05:50:36 GMT

in the GUI - I search the

Custom hadoop-metrics2.properties , and I see only Add Property ... thats all !

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 05:51:26 GMT

@Jay but as you know the main problem now is that we cant start the namenode on both machines ,

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 05:52:18 GMT

@JAY what we see from the log when we start the name node on master01/03 is that

ERROR namenode.NameNode (NameNode.java:main(1774)) - Failed to start namenode.
java.io.FileNotFoundException: No valid image files found

Re: JournalNode ( HDFS ) restarting all the time

jsensharma — Thu, 25 Jan 2018 05:58:05 GMT

@Michael Bronson

As you are getting the error:

ERROR namenode.NameNode (NameNode.java:main(1774)) - Failed to start namenode.
java.io.FileNotFoundException: No valid image files found

So can you please check of the following directory has any fsimage in it or not? Also if the fsimage file has proper read permission as following or not?

Example:

# ls -l /hadoop/hdfs/namenode/current/fsimage*

-rw-r--r--. 1 hdfs hadoop 195873 Jan 22 20:05 /hadoop/hdfs/namenode/current/fsimage_0000000000002711213
-rw-r--r--. 1 hdfs hadoop     62 Jan 22 20:05 /hadoop/hdfs/namenode/current/fsimage_0000000000002711213.md5
-rw-r--r--. 1 hdfs hadoop 195873 Jan 23 02:05 /hadoop/hdfs/namenode/current/fsimage_0000000000002718519
-rw-r--r--. 1 hdfs hadoop     62 Jan 23 02:05 /hadoop/hdfs/namenode/current/fsimage_0000000000002718519.md5

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 06:01:15 GMT

 I get that:



ls -l /hadoop/hdfs/journal/hdfsha/current/fsimage*

ls: cannot access /hadoop/hdfs/journal/hdfsha/current/fsimage*: No such file or directory

Re: JournalNode ( HDFS ) restarting all the time

sankar_tumuluru — Thu, 25 Jan 2018 06:02:42 GMT

Can you redeploy the HA and see if there were any steps that you missed during the HA enabling process. Please follow the steps suggested by Hortonworks.

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.4/bk_hadoop-high-availability/content/ch_HA-NameNode.html

Re: JournalNode ( HDFS ) restarting all the time

jsensharma — Thu, 25 Jan 2018 06:03:52 GMT

@Michael Bronson

It looks like some one deleted the "fsimage" file from the NameNode host. I am not aware of any hadoop bug which will cause deletion of this file.

So please check the Operating System Audit log to find out who has deleted the file and when?


#  less /var/log/audit/audit.log

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 06:08:47 GMT

Ho NO , any option for backup files or how to restore this ?

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 06:11:46 GMT

this huge file what need to search with grep ?

Re: JournalNode ( HDFS ) restarting all the time

jsensharma — Thu, 25 Jan 2018 06:21:59 GMT

@Michael Bronson

You can grep for 'fsimage" or the "current" word so that if there is any entry for file deletion then it might be logged in there.

Also as it looks like your "fsimage" file is missing so i do not have any option right now to fix this issue until we have fsimage backup stored somewhere which we can restore. (Else we are in Data Loss Situation)

Is there any Mount point Or Filesystem issue which is causing the particular fsimage partition to go away (disappear) may be you can check with some storage guy to findout what happened to that file?

Re: JournalNode ( HDFS ) restarting all the time

mike_bronson7 — Thu, 25 Jan 2018 06:22:20 GMT

@Jay is it possible to find other file fsimage from other cluster and use it on the problematic cluser ?