question Indexing Issue in HDFS in Support Questions

Indexing Issue in HDFS

Amn_468 — Fri, 16 Sep 2022 14:41:25 GMT

Hello,
I was trying to take report from HDFS Report and get the below message

Upon checking /var/log/cloudera-scm-headlamp and see the following error

021-04-15 18:09:44,934 ERROR com.cloudera.headlamp.HeadlampIndexManager: Index build failed for service hdfs java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:80) at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:257) at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:325) at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:240) at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java:57) at com.cloudera.headlamp.HeadlampIndexManager$1.run(HeadlampIndexManager.java:494) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64 at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:74) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:221) at com.cloudera.cmf.cdhclient.CdhExecutor$RunnableWrapper.call(CdhExecutor.java:211) at com.cloudera.cmf.cdhclient.CdhExecutor$CallableWrapper.doWork(CdhExecutor.java:236) at com.cloudera.cmf.cdhclient.CdhExecutor$1.call(CdhExecutor.java:125) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64 at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:77) at com.cloudera.headlamp.AbstractIndexBuilder$1.run(AbstractIndexBuilder.java:72) ... 8 more Caused by: java.io.IOException: Unsupported layout version -64 at org.apache.hadoop.hdfs.server.namenode.FSImageUtil.loadSummary(FSImageUtil.java:75) at org.apache.hadoop.hdfs.tools.offlineImageViewer.CdhClientPBImageViewer.go(CdhClientPBImageViewer.java:113) at com.cloudera.headlamp.IndexBuilderCDH.buildIndexImpl(IndexBuilderCDH.java:70) ... 9 more

CM / CDH - 6.3.3

Appreciate any help / guidance in fixing this issue .

Thanks

Amn

Re: Indexing Issue in HDFS

Amn_468 — Wed, 28 Apr 2021 01:31:22 GMT

Any assistance on this please ?

Re: Indexing Issue in HDFS

amk — Fri, 30 Apr 2021 12:14:31 GMT

Hello Amn,

Thanks for using Cloudera Community. I believe this issue is happening due to index failure. Can you please follow the below steps?

1. Stop the RMAN service from CM.

2. Navigate to CM > Cloudera Management Service > Configuration and search for Reports Manager Working Directory

By default it would be /var/lib/cloudera-scm-headlamp

Once identified we can move contents of /var/lib/cloudera-scm-headlamp/HDFS-1/ns1/* to some other tmp directory. Post that start RMAN service. This should help us in rebuilding the index once again.

Please let me know if it helps.

Re: Indexing Issue in HDFS

Amn_468 — Mon, 03 May 2021 02:45:38 GMT

Hi @amk

Hi,

When I go to /var/lib/cloudera-scm-headlamp/hdfs/ I see the below listed :

[nm1@myserver hdfs]# ls -ltr
-rw-r----- 1 cloudera-scm cloudera-scm 710967007 Jan 29 23:35 fsimage
-rw-r----- 1 cloudera-scm cloudera-scm 25680 Jan 29 23:44 aggregates
drwxr-x--- 2 cloudera-scm cloudera-scm 8192 Jan 29 23:44 index
-rw-r----- 1 cloudera-scm cloudera-scm 468901714 May 3 14:26 fsimage.tmp

Are you suggesting that I move everything that is inside hdfs folder to tmp location that then start Rman server? Kindly confirm.

Thanks

Amn

Re: Indexing Issue in HDFS

amk — Mon, 03 May 2021 03:43:50 GMT

Yes, right. Please move all the contents and start RMAN service once again.

Re: Indexing Issue in HDFS

JuliaC — Wed, 25 Aug 2021 15:21:45 GMT

You can find the error message 'Index build failed for service hdfs' in the RM log. This issue is caused by exceptional ' /var/lib/cloudera-scm-headlamp/' where the Reports Manager role is configured to run.

step 1: stop CMS Reports Manager role

step 2: RUN THIS ON THE HOST WHERE RM ROLE RUNS
sudo rm -rf /var/lib/cloudera-scm-headlamp/*

step 3: start CMS Reports Manager role (will rebuild the headlamp dir)

Done.
You can now use CM> HDFS> File Browser.