Skip to main content
MantuDeka
MantuDeka
Silver ⭐️⭐️⭐️
March 15, 2019
Solved

Event Server is failed to start.

In my cluster Event server is faild to start.When I restart it it shows the below error message :

 

---------

Process Status:


This role's process failed to start.


Cloudera Manager Descriptor Age


Not enough data to test: Test of whether the Cloudera Manager descriptor is up to date


Unknown : Not enough data to test: Test of whether the Cloudera Manager descriptor is up to date.
---------------

 

Someone sugesated to increase Java heap size of Event server 1 GB. I can see its already 1 GB.

 

Can you guys help me why its failed to start , what other things I can check??

 

lwang
lwangAnswer
Platinum ⭐️
March 21, 2019

Hi @MantuDeka ,

 

Thanks for providing the log file. The error message indicates that the Event Server index has an corruption. However, the log snippet does not tell what caused it. 

 

For resolution, there is an option to attempt to fix the index which involves some steps, however the index may not be recoverable.

Alternatively a quick fix would be to start the Event Server with a fresh new index. This will not cause any impact to cluster operation, but you will not be able to search for previous events in Cloudera Manager UI, CM -> Diagnostics -> Events page.

 

Here are the steps for quick fix:

1. Stop the Event Server role instance in CM
2. Backup the current data directory (you can find the value from CM UI and search for -> "Event Server Index Directory"), default value usually is /var/lib/cloudera-scm-eventserver/ (just for safety reasons, this backup is likely not needed afterwards)
3. Empty the data directory: # rm -rf /var/lib/cloudera-scm-eventserver/*
4. Start the Event Server role instance in CM
5. Monitor the Event Server role logs in /var/log/cloudera-scm-eventserver/ directory, it should confirm the process is able to start up and operate

 

Thanks,

Li

Li Wang, Technical Solution Manager
View in thread

9 replies

manuroman
manuroman
Gold ⭐️
March 19, 2019

Hi @MantuDeka,

 

Try to stop some unnecesary services that are running.

 

 

Regards,

Manu.

Translate
gzigldrum
Gold ⭐️
gzigldrum
Gold ⭐️
March 19, 2019

Please check for errors in the Event Server logs and stderr/stdout logs. Also check /var/log/messages if it may have been killed by the kernel oom killer.

Translate
    MantuDeka
    MantuDekaAuthor
    Silver ⭐️⭐️⭐️
    March 20, 2019
    Spoiler
     

    Hi @gzigldrum@lwang

    Below are the Event Server error logs-

     

    Error starting EventServer
    java.io.IOException: read past EOF
    at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:207)
    at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
    at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:40)
    at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:71)
    at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:260)
    at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:168)
    at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1155)
    at com.cloudera.cmf.eventcatcher.server.SingleIndexManager.makeIndexWriter(SingleIndexManager.java:139)
    at com.cloudera.cmf.eventcatcher.server.SingleIndexManager.<init>(SingleIndexManager.java:112)
    at com.cloudera.cmf.eventcatcher.server.EventCatcherService.<init>(EventCatcherService.java:282)
    at com.cloudera.cmf.eventcatcher.server.EventCatcherService.main(EventCatcherService.java:148)

    Translate
    lwang
    Platinum ⭐️
    lwangAnswer
    Platinum ⭐️
    March 21, 2019

    Hi @MantuDeka ,

     

    Thanks for providing the log file. The error message indicates that the Event Server index has an corruption. However, the log snippet does not tell what caused it. 

     

    For resolution, there is an option to attempt to fix the index which involves some steps, however the index may not be recoverable.

    Alternatively a quick fix would be to start the Event Server with a fresh new index. This will not cause any impact to cluster operation, but you will not be able to search for previous events in Cloudera Manager UI, CM -> Diagnostics -> Events page.

     

    Here are the steps for quick fix:

    1. Stop the Event Server role instance in CM
    2. Backup the current data directory (you can find the value from CM UI and search for -> "Event Server Index Directory"), default value usually is /var/lib/cloudera-scm-eventserver/ (just for safety reasons, this backup is likely not needed afterwards)
    3. Empty the data directory: # rm -rf /var/lib/cloudera-scm-eventserver/*
    4. Start the Event Server role instance in CM
    5. Monitor the Event Server role logs in /var/log/cloudera-scm-eventserver/ directory, it should confirm the process is able to start up and operate

     

    Thanks,

    Li

    Translate
    Li Wang, Technical Solution Manager
      lwang
      Platinum ⭐️
      lwang
      Platinum ⭐️
      March 19, 2019

      Hi @MantuDeka ,

       

      To second @gzigldrum 's feedback, you can find out more information from some logs. By default:

       

      1) On the Event Server host, find out the process logs:

      /var/run/cloudera-scm-agent/process/<process-ID>-cloudera-mgmt-EVENTSERVER/logs/stderr and stdout.log

       

      2) On the same host, find out the role log:

      /var/log/cloudera-scm-eventserver/mgmt-cmf-mgmt-EVENTSERVER-<hostname>.log.out

       

      Thanks and hope it helps,

      Li

      Translate
      Li Wang, Technical Solution Manager
        Terms & ConditionsAccessibility statement