Support Questions
Find answers, ask questions, and share your expertise

Ambari Hbase

In my Ambari cluster setup, in HBASE I see the Files Local percentage is fluctuating. Though the cluster is not busy at all and am not doing any bulk load or any such heavy operation.

What are the things I need to verify and make changes to keep it stable and above 70%?


Can you be more specific about how the "Files Local" metric is fluctuating? Are you performing any operations on HBase?

I have created the cluster, it's brand new cluster and no hbase operations done so far. Though the cluster has gone through several restarts.

Super Collaborator

did you perform any region server restarts or experienced crashes? In this case regions from those servers are redistributed across the available RSes and locality may decrease. Check region server logs for any suspicious exceptions. Perform major compaction (the locality will 100% after that) and check master log for the reason for region reassigning when the locality is decreased.

I have mentioned below the log outputs from RS log

Super Guru

A major compaction will bring full file locality. You will time to time have to run major compaction when locality decreases. Many factor contribute to file locality decrease, memstore flush, region merge, split, failures, etc.

Can you please tell me what exactly I need to verify in region server's log's so that I can find out the specific reason, either it could be any of memstore flush, region merge, split, failures or a major compaction. I can configure accordingly, so it doesn't start by itself.

Super Guru

You can check logso for thexample activities I have identified and view many of these on hbase UI. major compaction is kicked off manually. major compaction is good, brings data locality to 100%. but this needs to be performed off-peak hours.

Got it.

I want to first figure out what caused the file locality to go down then take an action

Super Guru

right and this is completly normal activity. You will have data locality loss over time, exactly what major compaction is for. You will have to view logs for when memstore flushed and other activity I have identified which will point to how locality was lost. again perfectly normal.