Indexing Issue in HDFS

[#1]

Hello,
I was trying to take report from HDFS Report and get the below message

Upon checking /var/log/cloudera-scm-headlamp and see the following error 

021-04-15 18:09:44,934 ERROR com.cloudera.headlamp.HeadlampIndexManager: Index build failed for service hdfs
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:192)
        at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:80)
        at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:257)
        at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:325)
        at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:240)
        at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:57)
        at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:494)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64
        at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:74)
        at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:221)
        at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:211)
        at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:236)
        at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64
        at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:77)
        at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:72)
        ... 8 more
Caused by: java.io.IOException: Unsupported layout version -64
        at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75)
        at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:113)
        at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:70)
        ... 9 more

CM / CDH - 6.3.3

Appreciate any help / guidance in fixing this issue .

Thanks 

Amn

[#2]

Any assistance on this please ?

[#3]

Hello Amn,

Thanks for using Cloudera Community. I believe this issue is happening due to index failure. Can you please follow the below steps?

1. Stop the RMAN service from CM.

2. Navigate to CM > Cloudera Management Service > Configuration and search for Reports Manager Working Directory

By default it would be /var/lib/cloudera-scm-headlamp

Once identified we can move contents of /var/lib/cloudera-scm-headlamp/HDFS-1/ns1/* to some other tmp directory. Post that start RMAN service. This should help us in rebuilding the index once again.

Please let me know if it helps.

[#4]

Hi @amk 

Hi,

When I go to /var/lib/cloudera-scm-headlamp/hdfs/ I see the below listed :

[nm1@myserver hdfs]# ls -ltr
-rw-r----- 1 cloudera-scm cloudera-scm 710967007 Jan 29 23:35 fsimage
-rw-r----- 1 cloudera-scm cloudera-scm 25680 Jan 29 23:44 aggregates
drwxr-x--- 2 cloudera-scm cloudera-scm 8192 Jan 29 23:44 index
-rw-r----- 1 cloudera-scm cloudera-scm 468901714 May 3 14:26 fsimage.tmp

Are you suggesting that I move everything that is inside hdfs folder to tmp location that then start Rman server? Kindly confirm.

Thanks 

Amn

[#5 · Accepted solution]

Yes, right. Please move all the contents and start RMAN service once again. 

[#6]

You can find the error message 'Index build failed for service hdfs' in the RM log. This issue is caused by exceptional ' /var/lib/cloudera-scm-headlamp/' where the Reports Manager role is configured to run.

step 1: stop CMS Reports Manager role

step 2: RUN THIS ON THE HOST WHERE RM ROLE RUNS
sudo rm -rf /var/lib/cloudera-scm-headlamp/*

step 3: start CMS Reports Manager role (will rebuild the headlamp dir)

Done.
You can now use CM> HDFS> File Browser.