fsimage.ckpt_001122334455 files question

[#1]

In our main /dfs/nn/current directory I have a lot of old fsimage.ckpt_0000000001122334455 which are taking a lot of space local disk space.  Couple of questions.

1. Is it safe to delete these old files?  Files range from 6months old til a day old.

2. Do I need to stop the namenode service to deleted them or can I just delete them on the fly.

Thanks in advance!!!!!

[#2 · Accepted solution]

Yes it is safe to delete them while the NameNode is running but leave the most recent file alone as that may be actually in progress. The past ones are leftover files from failed checkpoint operations.

Its concerning though that you are observing this though, as it would also mean you may not have a fully done checkpoint yet. What is your CDH version for this HDFS?

[#3]

Thanks for the response Harsh.  We using cdh5.5.1,  for the past week I have at least one fsimage.ckpt file per day, but I do see newly created fsimage files after the fsimage.ckpt files has been created.  Our fsimage file is a few GBs in size.

[#4]

I'd recommend looking for WARN or higher logs with the reference "Checkpoint" in them, to find why it aborts mid-way frequently.

There were some timeout associated issues in very early CDH4 period, but I've not seen this issue repeat with CDH5, even for very large fsimages.