Reports Manager - Aggregate Data has not been initialized

[#1]

I was trying to access some of the HDFS reports (through the Reports tab) and I ran into the following error

The Reports Manager (reportsmanager (cmhost)) could not complete the request: Aggregate data has not been initialized. Please retry later. 
If problem persists for several hours, please check the Reports Manager log for errors.

When I check for the Report Manager's logs I see the following: 

2015-11-05 18:01:27,865 INFO com.cloudera.headlamp.LuceneImageVisitor: Starting index build
2015-11-05 18:01:27,877 ERROR com.cloudera.headlamp.AbstractIndexBuilder: Failed to build index from FSImage at /var/lib/cloudera-scm-headlamp/hdfs/nameservice1/fsimage.tmp
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
	at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:77)
	at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:256)
	at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:324)
	at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:238)
	at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:58)
	at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:492)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
	at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:71)
	at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:225)
	at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:215)
	at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:240)
	at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
	at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:76)
	at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:69)
	... 9 more
Caused by: java.io.IOException: Unsupported layout version -60
	at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:106)
	at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:69)
	... 10 more
2015-11-05 18:01:27,880 ERROR com.cloudera.headlamp.HeadlampIndex: Failed to build index
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
	at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:77)
	at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:256)
	at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:324)
	at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:238)
	at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:58)
	at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:492)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
	at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:71)
	at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:225)
	at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:215)
	at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:240)
	at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
	at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:76)
	at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:69)
	... 9 more
Caused by: java.io.IOException: Unsupported layout version -60
	at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:106)
	at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:69)
	... 10 more
2015-11-05 18:01:27,882 ERROR com.cloudera.headlamp.HeadlampIndexManager: Index build failed for service hdfs
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
	at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
	at java.util.concurrent.FutureTask.get(FutureTask.java:83)
	at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:77)
	at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:256)
	at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:324)
	at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:238)
	at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:58)
	at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:492)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
	at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:71)
	at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:225)
	at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:215)
	at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:240)
	at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -60
	at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:76)
	at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:69)
	... 9 more
Caused by: java.io.IOException: Unsupported layout version -60
	at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75)
	at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:106)
	at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:69)
	... 10 more

Seems that it is failing to build index from FSImage? Can anyone highlight what this is?

[#2]

Is your CM version higher than 5.4? What is the CDH version, btw?

Dice.

[#3]

CM Version: Cloudera Enterprise 5.4.3

CDH Version: 5.4.4

[#4]

Is the issue still occurring?

Just in case, can you run 'rpm -qa | grep cloudera' in the CM server host?

[#5]

Yes, this issue is still prevalent for me.

[user@cmhost ~]$ rpm -qa | grep cloudera
cloudera-manager-daemons-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-server-db-2-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-server-5.4.3-1.cm543.p0.258.el6.x86_64
cloudera-manager-agent-5.4.3-1.cm543.p0.258.el6.x86_64

[#6]

Run 'hadoop version' as well.

[#7]

This is the CM host. We do not have hadoop installed on this node. 

I ran the Host Inspector which alerts me that hdfs,hadoop and other such users are not present on this node and also this returns CDH version as "None". Could this be causing the Reports Manager to fail? We have a similar cluster (where CM host reports CDH version as NONE) but we do not see the same issue there. That cluster runs CM v5.4.7 and CDH 5.4.3.

[#8]

If you are still hitting the same stack trace, the indications won't be
relevant. Can you upload the Reports Manager log please?



--
Sent from my mobile

[#9]

You can find the Reports Manager logs here. Changed the hostnames for security reasons :)

[#10]

Hi Dice, 

Have you had the chance to look at those logs?