Reply
yt
New Contributor
Posts: 2
Registered: ‎10-01-2018

Cannot start Cloudera Management Services regarding as no space left on device

 

Regarding as no space left on device, I want to help to solve this issue.

 

This is on Redhat linux running CDH 6.0 beta

 

There are still failure to start service.

 

I'm trying to find related issue on community, I cloudn't find any more.

 

Any idea to solve thos issue? Any help is appreciated.

 

 

 

2018-09-29 21:04:10,536 ERROR com.cloudera.cmon.firehose.Main: Error creating LevelDB timeseries store in directory /var/lib/cloudera-service-monitor/ts
org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: /var/lib/cloudera-service-monitor/ts/ts_entity_metadata/000259.log: No space left on device

at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)

at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)

at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)

at com.cloudera.cmon.tstore.leveldb.LDBUtils.openVersionedDB(LDBUtils.java:264)

at com.cloudera.cmon.tstore.leveldb.LDBUtils.openVersionedDB(LDBUtils.java:212)

at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesMetadataStore.openMetadataDB(LDBTimeSeriesMetadataStore.java:147)
at com.cloudera.cmon.tstore.leveldb.LDBTimeSeriesMetadataStore.<init>(LDBTimeSeriesMetadataStore.java:137)

at com.cloudera.cmon.firehose.Main.main(Main.java:473)

New Contributor
Posts: 5
Registered: ‎10-01-2018

Re: Cannot start Cloudera Management Services regarding as no space left on device

Looks like no space left in the following directory '/var/lib/cloudera-service-monitor/ts/ts_entity_metadata/000259.log' IO error: /var/lib/cloudera-service-monitor/ts/ts_entity_metadata/000259.log: No space left on device Try to delete existing log files and allocate more space.
yt
New Contributor
Posts: 2
Registered: ‎10-01-2018

Re: Cannot start Cloudera Management Services regarding as no space left on device

 

Thanks,

 

I have already delete the existing files but the services can't start it.

Is there a cleanup procedure? (delete files, restart Host Monitor and Service Monitor)

Let me know the detail procedure.

 

Highlighted
New Contributor
Posts: 5
Registered: ‎10-01-2018

Re: Cannot start Cloudera Management Services regarding as no space left on device

Did you tried restarting all services on the cluster?

Posts: 1,108
Topics: 1
Kudos: 285
Solutions: 134
Registered: ‎04-22-2014

Re: Cannot start Cloudera Management Services regarding as no space left on device

@yt,

 

What does df -k show for space left on your devices?

 

By default, Cloudera Manager and Management service as well as all roles store log and application data in /var.

Make sure that /tmp also has free space available.

 

You can clean up log files in /var/log/

and

If you do not care about your timeseries data in host and service monitor, you can stop the servers, delete /var/lib/cloudera-service-monitor and /var/lib/cloudera-host-monitor.

 

NOTE:

Other roles store data on disk in /var/lib:

 

Event Server: /var/lib/cloudera-scm-eventserver

Reports Manager: /var/lib/cloudera-scm-headlamp

Navigator Metadata Server: /var/lib/cloudera-scm-navigator

 

There are other servers that store data there, too.  What you remove really depends on your needs and what is usurping the most space.  If you can, move the data directories to another volume with more space.  Cloudera Manager allows this configuration in the Management Service configuration for each role.