Member since
03-23-2015
7
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5634 | 03-24-2015 09:35 PM |
06-19-2015
10:15 AM
Thanks Ben. I did not get OOM in the logs of (RegionServers), (HMaster), (Zookeeper), Yarn Container logs. So I don't think it was starving from memory. However clouderaManager graphs for Regionserver reported frequent Garbage Collection, mean time for GC ~(23m(ms)) I went adhead with your recommendations on increasing regionserver memory. Since I had the liberty of a node with 122GB of memory, I bumped the (Java Heap Size of HBase RegionServer) from it default 4GB to 16GB. After the switch I no loger have ReagionServer crashes and applications are performing great. Later I checked the heap utilization of RegionServer on ClouderaManager Graphs, it uses (12-14 GB) of the acclocated heap. Also my GC graphs show (86s(ms)) which is 100 times lower that ~(23m(ms)). I wish I can share both these graphs.
... View more
06-16-2015
06:14 PM
Using the class path precedence is not the correct solution for all cases. A solution that will work in all cases is to use shading for the classes that you have modified versions of (use maven or gradle to do that). In your case you need to shade the parquet classes that you have modified when you package the jar. Be careful if you change classes like parquet: you could ed up with files that are only readable with your code and force you to keep packaging it with all jobs. That could cause problems later if you decide to use a different method to access the files. Wilfred
... View more