Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st. We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here. Need help or have questions? Drop us a line at [email protected]
Created on 08-23-201811:11 PM - edited 09-16-202201:44 AM
For HDFS analysis, an Activity Analyzer needs to be deployed to each NameNode in the cluster. These instances will automatically begin processing the fsimage on startup and will reprocess the latest fsimage data once every 24 hours. By default, when deployed on a NameNode, these Activity Analyzers do not process YARN, or MapReduce & Tez utilization data; This is to reduce the amount of processing done on servers hosting critical services like the NameNode.
Unless we install analyzer on namenode, it will not be able to find fsimage in directory and hence fail with below error
2018-08-23 14:28:04,401 ERROR [pool-3-thread-1] ActivityManager:78 - Failed to process activity id null of type HDFS
java.lang.NullPointerException
at java.io.File.<init>(File.java:277)
at com.hortonworks.smartsense.activity.hdfs.HDFSImageProcessor.processActivity(HDFSImageProcessor.java:71)
at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:69)
at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:62)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-08-23 14:28:04,403 INFO [Thread-3] ActivityManager:172 - Shutting down activity manager...
2018-08-23 14:28:04,425 WARN [Thread-6] HeapMemorySizeUtil:55 - hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.sizectivityManager:78 - Failed to process activity id null of type HDFSShutting down activity
Solution - Install activity analyzer on both the namenodes