Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Charts error - Internal error while querying the Host Monitor

Charts error - Internal error while querying the Host Monitor

Hi, 

 

Charts are not showing up properly in our Clouder manager after we had shutdown  and restarted our CM machine on jan 15th for upgrade. I see that firehouse hostmonitor is still looking for older corrupt files. 

How can I deal with this ?

 

And I see that 

 

2015-05-12 09:41:19,232 INFO com.cloudera.cmon.tstore.leveldb.LDBPartitionManager: Opening partition LDBPartitionMetadataWrapper{tableName=ts_stream_rollup_PT600S, partitionName=ts_stream_rollup_PT600S_2015-01-15T12:58:02.039Z, startTime=2015-01-15T12:58:02.039Z, endTime=null, version=2, state=CLOSED}
2015-05-12 09:41:19,232 ERROR com.cloudera.cmon.firehose.tsquery.TimeSeriesQueryRequestHandler: Query select dfs_capacity_across_hdfss failed to retrieve time series.
java.lang.RuntimeException: com.cloudera.cmon.tstore.leveldb.LDBPartitionManager$LDBPartitionException: Unable to open DB in directory /var/lib/cloudera-host-monitor/ts/ts_stream_rollup_PT600S/partitions/ts_stream_rollup_PT600S_2015-01-15T12:58:02.039Z for partition LDBPartitionMetadataWrapper{tableName=ts_stream_rollup_PT600S, partitionName=ts_stream_rollup_PT600S_2015-01-15T12:58:02.039Z, startTime=2015-01-15T12:58:02.039Z, endTime=null, version=2, state=CLOSED}
at com.cloudera.cmon.tstore.leveldb.LDBPartitionManager.getPartitionRange(LDBPartitionManager.java:857)
at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesStreamTable.read(LDBTimeSeriesStreamTable.java:218)
at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesStreamTable.read(LDBTimeSeriesStreamTable.java:408)
at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesStore.readFromStreamTable(LDBTimeSeriesStore.java:566)
at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesStore.read(LDBTimeSeriesStore.java:497)
at com.cloudera.cmon.tstore.AggregatingTimeSeriesStore.read(AggregatingTimeSeriesStore.java:426)
at com.cloudera.cmon.tstore.EntityDataTableHandler.getTimeSeriesResults(EntityDataTableHandler.java:212)
at com.cloudera.cmon.firehose.tsquery.TimeSeriesQueryHandler.doGetTimeSeries(TimeSeriesQueryHandler.java:319)
at com.cloudera.cmon.firehose.tsquery.TimeSeriesQueryHandler.getTimeSeries(TimeSeriesQueryHandler.java:200)
at com.cloudera.cmon.firehose.tsquery.TimeSeriesQueryRequestHandler.getResponse(TimeSeriesQueryRequestHandler.java:183)
at com.cloudera.cmon.firehose.TimeSeriesNozzleImpl.queryTimeSeries(TimeSeriesNozzleImpl.java:230)
at com.cloudera.cmon.firehose.NozzleImpl.queryTimeSeriesHelper(NozzleImpl.java:477)
at com.cloudera.cmon.firehose.NozzleImpl.queryMultiTimeSeries(NozzleImpl.java:518)
at sun.reflect.GeneratedMethodAccessor118.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.cloudera.cmon.firehose.InstrumentedNozzle$InstrumentedCallable$1.call(InstrumentedNozzle.java:149)
at com.cloudera.cmon.firehose.InstrumentedNozzle$InstrumentedCallable.call(InstrumentedNozzle.java:164)
at com.cloudera.cmon.firehose.InstrumentedNozzle.queryMultiTimeSeries(InstrumentedNozzle.java:444)
at sun.reflect.GeneratedMethodAccessor117.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.apache.avro.ipc.specific.SpecificResponder.respond(SpecificResponder.java:88)
at org.apache.avro.ipc.Responder.respond(Responder.java:149)
at org.apache.avro.ipc.Responder.respond(Responder.java:99)
at org.apache.avro.ipc.ResponderServlet.doPost(ResponderServlet.java:48)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)

..............

 

Caused by: com.cloudera.cmon.tstore.leveldb.LDBPartitionManager$LDBPartitionException: Unable to open DB in directory /var/lib/cloudera-host-monitor/ts/ts_stream_rollup_PT600S/partitions/ts_stream_rollup_PT600S_2015-01-15T12:58:02.039Z for partition LDBPartitionMetadataWrapper{tableName=ts_stream_rollup_PT600S, partitionName=ts_stream_rollup_PT600S_2015-01-15T12:58:02.039Z, startTime=2015-01-15T12:58:02.039Z, endTime=null, version=2, state=CLOSED}
at com.cloudera.cmon.tstore.leveldb.LDBUtils.openOrCreatePartitionDB(LDBUtils.java:181)
at com.cloudera.cmon.tstore.leveldb.LDBPartitionManager.getOrOpenInternal(LDBPartitionManager.java:616)
at com.cloudera.cmon.tstore.leveldb.LDBPartitionManager.openOrCreatePartitionLDB(LDBPartitionManager.java:557)
at com.cloudera.cmon.tstore.leveldb.LDBPartitionManager.getPartition(LDBPartitionManager.java:451)
at com.cloudera.cmon.tstore.leveldb.LDBPartitionManager.getPartitionRange(LDBPartitionManager.java:844)
... 40 more
Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: Corruption: 1 missing files; e.g.: /var/lib/cloudera-host-monitor/ts/ts_stream_rollup_PT600S/partitions/ts_stream_rollup_PT600S_2015-01-15T12:58:02.039Z/000010.sst
at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:194)
at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:212)
at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
at com.cloudera.cmon.tstore.leveldb.LDBUtils.openOrCreatePartitionDB(LDBUtils.java:172)
... 44 more
2015-05-12 09:41:20,034 INFO com.cloudera.cmon.tstore.leveldb.LDBPartitionManager: Opening partition LDBPartitionMetadataWrapper{tableName=subject_ts, partitionName=subject_ts_2014-07-21T17:08:50.273Z, startTime=2014-07-21T17:08:50.273Z, endTime=null, version=3, state=CLOSED}

1 REPLY 1
Highlighted

Re: Charts error - Internal error while querying the Host Monitor

New Contributor

Hello,

How did you replace your failed node? Is the CM agent running? Can you look at config.ini of the new host? Does it have the right `server_host'?