Activity Analyzer is not reporting any data for HDFS Dashboard

[#1]

Affected versions:

All SmartSense versions

Symptoms:

YARN and MR dashboards work fine but there is no data for HDFS dashboard. Activity Analyzer logs are available at /var/log/smartsense-activity/activity-analyzer.log. These logs will shows an error stack trace similar to something as provided below:

ERROR [pool-14-thread-1] ActivityManager:89 - Failed to process
activity id /hadoop/hdfs/namenode/current/fsimage_0009281 of type HDFScom.hortonworks.smartsense.activity.ActivityException: Failed to process activity. Activity Type: HDFS; ActivityId:
/hadoop/hdfs/namenode/current/fsimage_0009281; ActivityData: null  
at com.hortonworks.smartsense.activity.hdfs.HDFSImageProcessor.processActivity(HDFSImageProcessor.java:104)  
at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:80)  
at com.hortonworks.smartsense.activity.ActivityManager$1.call(ActivityManager.java:73)  
at java.util.concurrent.FutureTask.run(FutureTask.java:262)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:745)Caused by: com.hortonworks.smartsense.activity.ActivityException: Error while getting fsImage iterator from path /hadoop/hdfs/namenode/current/fsimage_0009281  
at org.apache.hadoop.hdfs.tools.offlineImageViewer.PBImageParser.getIterator(PBImageParser.java:118) 
at com.hortonworks.smartsense.activity.hdfs.HDFSImageProcessor.processActivity(HDFSImageProcessor.java:88)  ... 6 more

Caused by: java.io.IOException: Unsupported layout version -63  at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75)

Reason:

Activity Analyzer maintains all the dashboard in a new schema created in the same database as setup by Ambari Metrics Collector. It also utilizes the clients for HDFS available in the AMS libraries. This issue can happen if Activity Analyzer is used with older Hadoop/MapReduce/YARN libraries from AMS.

Solution:

Verify all the jars available under /usr/lib/ambari-metrics-collector and remove the older versions. Restart the Activity Analyzer after removing the older jars.

Examples of older version jars:

/usr/lib/ambari-metrics-collector/hadoop-annotations-2.6.0.2.2.0.0-2041.jar
/usr/lib/ambari-metrics-collector/hadoop-auth-2.6.0.2.2.0.0-2041.jar 
/usr/lib/ambari-metrics-collector/hadoop-client-2.6.0.2.2.0.0-2041.jar 
/usr/lib/ambari-metrics-collector/hadoop-common-2.6.0.2.2.0.0-2041.jar 
/usr/lib/ambari-metrics-collector/hadoop-hdfs-2.6.0.2.2.0.0-2041.jar 
/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-app-2.6.0.2.2.0.0-2041.jar
/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-common-2.6.0.2.2.0.0-2041.jar
/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-core-2.6.0.2.2.1.0-2340.jar
/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-jobclient-2.6.0.2.2.0.0-2041.jar
/usr/lib/ambari-metrics-collector/hadoop-mapreduce-client-shuffle-2.6.0.2.2.0.0-2041.jar
/usr/lib/ambari-metrics-collector/hadoop-yarn-api-2.6.0.2.2.0.0-2041.jar
/usr/lib/ambari-metrics-collector/hadoop-yarn-client-2.6.0.2.2.0.0-2041.jar
/usr/lib/ambari-metrics-collector/hadoop-yarn-common-2.6.0.2.2.0.0-2041.jar
/usr/lib/ambari-metrics-collector/hadoop-yarn-registry-2.6.0.2.2.0.0-2041.jar