Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cannot start Cloudera Management Services regarding as no space left on device

Cannot start Cloudera Management Services regarding as no space left on device

New Contributor

 

Regarding as no space left on device, I want to help to solve this issue.

 

This is on Redhat linux running CDH 6.0 beta

 

There are still failure to start service.

 

I'm trying to find related issue on community, I cloudn't find any more.

 

Any idea to solve thos issue? Any help is appreciated.

 

 

 

2018-09-29 21:04:10,536 ERROR com.cloudera.cmon.firehose.Main: Error creating LevelDB timeseries store in directory /var/lib/cloudera-service-monitor/ts
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/lib/cloudera-service-monitor/ts/ts_entity_metadata/000259.log: No space left on device

at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)

at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)

at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)

at com.cloudera.cmon.tstore.leveldb.LDBUtils.openVersionedDB(LDBUtils.java:264)

at com.cloudera.cmon.tstore.leveldb.LDBUtils.openVersionedDB(LDBUtils.java:212)

at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesMetadataStore.openMetadataDB(LDBTimeSeriesMetadataStore.java:147)
at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesMetadataStore.<init>(LDBTimeSeriesMetadataStore.java:137)

at com.cloudera.cmon.firehose.Main.main(Main.java:473)

4 REPLIES 4

Re: Cannot start Cloudera Management Services regarding as no space left on device

New Contributor
Looks like no space left in the following directory '/var/lib/cloudera-service-monitor/ts/ts_entity_metadata/000259.log' IO error: /var/lib/cloudera-service-monitor/ts/ts_entity_metadata/000259.log: No space left on device Try to delete existing log files and allocate more space.

Re: Cannot start Cloudera Management Services regarding as no space left on device

New Contributor

 

Thanks,

 

I have already delete the existing files but the services can't start it.

Is there a cleanup procedure? (delete files, restart Host Monitor and Service Monitor)

Let me know the detail procedure.

 

Re: Cannot start Cloudera Management Services regarding as no space left on device

New Contributor

Did you tried restarting all services on the cluster?

Re: Cannot start Cloudera Management Services regarding as no space left on device

Super Guru

@yt,

 

What does df -k show for space left on your devices?

 

By default, Cloudera Manager and Management service as well as all roles store log and application data in /var.

Make sure that /tmp also has free space available.

 

You can clean up log files in /var/log/

and

If you do not care about your timeseries data in host and service monitor, you can stop the servers, delete /var/lib/cloudera-service-monitor and /var/lib/cloudera-host-monitor.

 

NOTE:

Other roles store data on disk in /var/lib:

 

Event Server: /var/lib/cloudera-scm-eventserver

Reports Manager: /var/lib/cloudera-scm-headlamp

Navigator Metadata Server: /var/lib/cloudera-scm-navigator

 

There are other servers that store data there, too.  What you remove really depends on your needs and what is usurping the most space.  If you can, move the data directories to another volume with more space.  Cloudera Manager allows this configuration in the Management Service configuration for each role.