question Re: Event Server is failed to start. in Support Questions

Event Server is failed to start.

MantuDeka — Fri, 15 Mar 2019 21:39:28 GMT

In my cluster Event server is faild to start.When I restart it it shows the below error message :

---------

Process Status:

This role's process failed to start.

Cloudera Manager Descriptor Age

Not enough data to test: Test of whether the Cloudera Manager descriptor is up to date

Unknown : Not enough data to test: Test of whether the Cloudera Manager descriptor is up to date.
---------------

Someone sugesated to increase Java heap size of Event server 1 GB. I can see its already 1 GB.

Can you guys help me why its failed to start , what other things I can check??

Re: Event Server is failed to start.

manuroman — Tue, 19 Mar 2019 11:12:32 GMT

Hi @MantuDeka,

Try to stop some unnecesary services that are running.

Regards,

Manu.

Re: Event Server is failed to start.

gzigldrum — Tue, 19 Mar 2019 15:51:15 GMT

Please check for errors in the Event Server logs and stderr/stdout logs. Also check /var/log/messages if it may have been killed by the kernel oom killer.

Re: Event Server is failed to start.

lwang — Tue, 19 Mar 2019 22:43:04 GMT

Hi @MantuDeka ,

To second @gzigldrum 's feedback, you can find out more information from some logs. By default:

1) On the Event Server host, find out the process logs:

/var/run/cloudera-scm-agent/process/<process-ID>-cloudera-mgmt-EVENTSERVER/logs/stderr and stdout.log

2) On the same host, find out the role log:

/var/log/cloudera-scm-eventserver/mgmt-cmf-mgmt-EVENTSERVER-<hostname>.log.out

Thanks and hope it helps,

Re: Event Server is failed to start.

MantuDeka — Wed, 20 Mar 2019 23:57:50 GMT

Hi gzigldrum , lwang

Below are the Event Server error logs-

Error starting EventServer
java.io.IOException: read past EOF
	at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:207)
	at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)
	at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:40)
	at org.apache.lucene.store.IndexInput.readInt(IndexInput.java:71)
	at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:260)
	at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:168)
	at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1155)
	at com.cloudera.cmf.eventcatcher.server.SingleIndexManager.makeIndexWriter(SingleIndexManager.java:139)
	at com.cloudera.cmf.eventcatcher.server.SingleIndexManager.<init>(SingleIndexManager.java:112)
	at com.cloudera.cmf.eventcatcher.server.EventCatcherService.<init>(EventCatcherService.java:282)
	at com.cloudera.cmf.eventcatcher.server.EventCatcherService.main(EventCatcherService.java:148)

Re: Event Server is failed to start.

lwang — Thu, 21 Mar 2019 01:35:51 GMT

Hi @MantuDeka ,

Thanks for providing the log file. The error message indicates that the Event Server index has an corruption. However, the log snippet does not tell what caused it.

For resolution, there is an option to attempt to fix the index which involves some steps, however the index may not be recoverable.

Alternatively a quick fix would be to start the Event Server with a fresh new index. This will not cause any impact to cluster operation, but you will not be able to search for previous events in Cloudera Manager UI, CM -> Diagnostics -> Events page.

Here are the steps for quick fix:

1. Stop the Event Server role instance in CM
2. Backup the current data directory (you can find the value from CM UI and search for -> "Event Server Index Directory"), default value usually is /var/lib/cloudera-scm-eventserver/ (just for safety reasons, this backup is likely not needed afterwards)
3. Empty the data directory: # rm -rf /var/lib/cloudera-scm-eventserver/*
4. Start the Event Server role instance in CM
5. Monitor the Event Server role logs in /var/log/cloudera-scm-eventserver/ directory, it should confirm the process is able to start up and operate

Thanks,

Re: Event Server is failed to start.

MantuDeka — Fri, 22 Mar 2019 18:36:49 GMT

Thanks lwang for your response .

Is this the reason we are not getting health check alert over email?? Even Report manager is also failed to start. Same issue.

Below is the Health Test for both..

Cloudera Manager Descriptor Age
.
Not enough data to test: Test of whether the Cloudera Manager descriptor is up to date.

WE are getting the eror while tring to restart report manager :

Re: Event Server is failed to start.

lwang — Mon, 25 Mar 2019 18:10:50 GMT

Hi @MantuDeka ,

Yes, if event server is not working, you won't be able to receive alerts. The alerts basically are the events which are marked as alerts based on CM configuration checking.

About Report Manager, we need to check the RM process logs and role log for more details about what went wrong. By default:

1) On the Report Manager host, find out the process logs:

/var/run/cloudera-scm-agent/process/<process-ID>-cloudera-mgmt-REPORTSMANAGER/logs/stderr and stdout.log

2) On the same host, find out the role log:

/var/log/cloudera-scm-headlamp/mgmt-cmf-mgmt-REPORTSMANAGER-<hostname>.log.out

Thanks and hope it helps,

Re: Event Server is failed to start.

MantuDeka — Fri, 14 Jun 2019 11:39:19 GMT

Hi lwang ,

Thanks lots.It worked.

Re: Event Server is failed to start.

lwang — Fri, 14 Jun 2019 22:34:40 GMT

Great to hear and thanks for letting us know the result!