Reply
Explorer
Posts: 12
Registered: ‎03-14-2019
Accepted Solution

Event Server is failed to start.

[ Edited ]

In my cluster Event server is faild to start.When I restart it it shows the below error message :

 

---------

Process Status:


This role's process failed to start.


Cloudera Manager Descriptor Age


Not enough data to test: Test of whether the Cloudera Manager descriptor is up to date


Unknown : Not enough data to test: Test of whether the Cloudera Manager descriptor is up to date.
---------------

 

Someone sugesated to increase Java heap size of Event server 1 GB. I can see its already 1 GB.

 

Can you guys help me why its failed to start , what other things I can check??

 

Expert Contributor
Posts: 104
Registered: ‎02-23-2018

Re: Event Server is failed to start.

Hi @MantuDeka,

 

Try to stop some unnecesary services that are running.

 

 

Regards,

Manu.

Cloudera Employee
Posts: 225
Registered: ‎01-15-2015

Re: Event Server is failed to start.

Please check for errors in the Event Server logs and stderr/stdout logs. Also check /var/log/messages if it may have been killed by the kernel oom killer.

Posts: 939
Kudos: 28
Solutions: 17
Registered: ‎05-27-2014

Re: Event Server is failed to start.

[ Edited ]

Hi @MantuDeka ,

 

To second @gzigldrum 's feedback, you can find out more information from some logs. By default:

 

1) On the Event Server host, find out the process logs:

/var/run/cloudera-scm-agent/process/<process-ID>-cloudera-mgmt-EVENTSERVER/logs/stderr and stdout.log

 

2) On the same host, find out the role log:

/var/log/cloudera-scm-eventserver/mgmt-cmf-mgmt-EVENTSERVER-<hostname>.log.out

 

Thanks and hope it helps,

Li

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Explorer
Posts: 12
Registered: ‎03-14-2019

Re: Event Server is failed to start.

[ Edited ]
Spoiler
 

Hi gzigldrum ,  lwang

Below are the Event Server error logs-

 

Error starting EventServer
java.io.IOException: read past EOF
	at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:207)
	at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
	at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:40)
	at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:71)
	at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:260)
	at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:168)
	at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1155)
	at com.cloudera.cmf.eventcatcher.server.SingleIndexManager.makeIndexWriter(SingleIndexManager.java:139)
	at com.cloudera.cmf.eventcatcher.server.SingleIndexManager.<init>(SingleIndexManager.java:112)
	at com.cloudera.cmf.eventcatcher.server.EventCatcherService.<init>(EventCatcherService.java:282)
	at com.cloudera.cmf.eventcatcher.server.EventCatcherService.main(EventCatcherService.java:148)
Posts: 939
Kudos: 28
Solutions: 17
Registered: ‎05-27-2014

Re: Event Server is failed to start.

Hi @MantuDeka ,

 

Thanks for providing the log file. The error message indicates that the Event Server index has an corruption. However, the log snippet does not tell what caused it. 

 

For resolution, there is an option to attempt to fix the index which involves some steps, however the index may not be recoverable.

Alternatively a quick fix would be to start the Event Server with a fresh new index. This will not cause any impact to cluster operation, but you will not be able to search for previous events in Cloudera Manager UI, CM -> Diagnostics -> Events page.

 

Here are the steps for quick fix:

1. Stop the Event Server role instance in CM
2. Backup the current data directory (you can find the value from CM UI and search for -> "Event Server Index Directory"), default value usually is /var/lib/cloudera-scm-eventserver/ (just for safety reasons, this backup is likely not needed afterwards)
3. Empty the data directory: # rm -rf /var/lib/cloudera-scm-eventserver/*
4. Start the Event Server role instance in CM
5. Monitor the Event Server role logs in /var/log/cloudera-scm-eventserver/ directory, it should confirm the process is able to start up and operate

 

Thanks,

Li

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Explorer
Posts: 12
Registered: ‎03-14-2019

Re: Event Server is failed to start.

[ Edited ]

Thanks lwang for your response .

Is this the reason we are not getting health check alert over email?? Even Report manager is also failed to start. Same issue.

 

Below is the Health Test for both..

 

  • Cloudera Manager Descriptor Age
    Not enough data to test: Test of whether the Cloudera Manager descriptor is up to date.

 

WE are getting the eror while tring to restart report manager :

 

 

 

 

 

Highlighted
Posts: 939
Kudos: 28
Solutions: 17
Registered: ‎05-27-2014

Re: Event Server is failed to start.

Hi @MantuDeka ,

 

Yes, if event server is not working, you won't be able to receive alerts. The alerts basically are the events which are marked as alerts based on CM configuration checking.

 

About Report Manager, we need to check the RM process logs and role log for more details about what went wrong. By default:

 

1) On the Report Manager host, find out the process logs:

/var/run/cloudera-scm-agent/process/<process-ID>-cloudera-mgmt-REPORTSMANAGER/logs/stderr and stdout.log

 

2) On the same host, find out the role log:

/var/log/cloudera-scm-headlamp/mgmt-cmf-mgmt-REPORTSMANAGER-<hostname>.log.out

 

Thanks and hope it helps,

Li

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Explorer
Posts: 12
Registered: ‎03-14-2019

Re: Event Server is failed to start.

Hi lwang ,

 

Thanks lots.It worked.

Posts: 939
Kudos: 28
Solutions: 17
Registered: ‎05-27-2014

Re: Event Server is failed to start.

Great to hear and thanks for letting us know the result!

Li Wang, Technical Resolution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Announcements