We are using phoenix(4.8-Version) to perform read operation on our Hbase(1.2.3 - Version) table but suddenly after 7 months from deployment we are stuck in an serious issue where suddenly phoenix is showing the table count as '16232555' but when we perform count using Hbase Shell it is reflecting correctly as '1985950'. We had tried to drop the table & reload the data but again after sometime phoenix count is reflecting incorrectly.
1. We are unable to understand what is triggering this issue
2. How can we solve this issue
3. Is there any issue related to Phoenix metadata which is going into inconsistent mode
4. What measures to be implemented to avoid this issue in future.
We tried updating the guidepost value from default(100MB) to 5GB after which we are getting correct count but still unable to understand why this count suddenly started showing incorrectly and is the value of 5GB for guidepost would impact other things. Still need a confirm solution for this problem. Let us know if there is any solution/suggesstion. Any help would greatly appreciated. Thanks