Zookeeper Heap Space issue

[#1]

Zookeeper heap configured to 1GB and snapshot size suddenly reached to 1GB(950M ) and Zookeeper service got stopped. Once we increase heap to 4GB, ZK is working fine. The snapshot size is now back to few MBs once we restart the Cluster.

How to get the cause for this problem? What is stored as part of zookeeper snapshot? Why there is a sudden increase in snapshot size?

[#2]

Lots of Hadoop components are using ZK to store information. One example I have seen is that when someone tries to query a table in Hive that has many many partitions, Hive will try to lock those partitions by storing this locking info in ZK. So depends on how many partitions, ZK will increase its usage.

However, again, many things can happen, you need to capture what was happening in your cluster when issue occurred and do proper analysis.

Cheers
Eric

[#3]

@vinodnerella 
It depends based on the scenario that how much Heap you should be allocating for the Zookeeper.

In your case if you are keep noticing that the Zookeeper heap is reaching to its max 1GB then it is better to increase the Zookeeper heap to a larger value and if needed then enable GC logging for zookeeper to monitor the gc usages in a period of time to findout the approximate heap that you need to setup for your zookeeper based on the environment requirement.

As you have already set the Zookeeper heap to 4GB it should be good for now. We can monitor it for some time.

The common cause of Zookeeper OutOfMemory can be when clients submit requests faster than ZooKeeper can process them, especially if there are a lot of clients.  The it can lead to OOM errors.

You can also take a look into parameters like "zookeeper.snapCount"  but better to monitor Zookeeper with 4GB heap for some time before tuning such parameters.