Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

org.postgresql.util.PSQLException:ERROR:could not extend relation base/17381:No space left on device

Highlighted

org.postgresql.util.PSQLException:ERROR:could not extend relation base/17381:No space left on device

New Contributor

The CM went down and I cannot see any services from Cm interface.

 

I see the following in CM log:

[root@ash-r101-14l ~]# vi /var/log/cloudera-scm-server/cloudera-scm-server.log

        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)

        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)

        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)

        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)

        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)

        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:133)

        ... 19 more

2015-05-14 19:52:02,914 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.

javax.persistence.RollbackException: Error while committing the transaction

        at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:92)

        at com.cloudera.enterprise.AbstractWrappedEntityManager.commit(AbstractWrappedEntityManager.java:110)

        at com.cloudera.cmf.persist.CmfEntityManager.commit(CmfEntityManager.java:366)

        at com.cloudera.server.cmf.Main.setUpClusterStatsListeners(Main.java:371)

        at com.cloudera.server.cmf.Main.<init>(Main.java:271)

        at com.cloudera.server.cmf.Main.main(Main.java:196)

Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: could not execute statement

        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)

        at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)

        at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:80)

        ... 5 more

Caused by: org.hibernate.exception.GenericJDBCException: could not execute statement

        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54)

        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)

        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)

        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:136)

        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3228)

        at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:3126)

        at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:3456)

        at org.hibernate.action.internal.EntityUpdateAction.execute(EntityUpdateAction.java:140)

        at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:377)

        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:369)

        at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:287)

        at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:339)

        at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52)

        at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1234)

        at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404)

        at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.beforeTransactionCommit(JdbcTransaction.java:101)

        at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:175)

        at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:75)

        ... 5 more

Caused by: org.postgresql.util.PSQLException: ERROR: could not extend relation base/16385/17381: No space left on device

  Hint: Check free disk space.

        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)

        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)

        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)

        at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)

        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)

        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)

        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)

        at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:133)

        ... 19 more

 

When I run a df –h command, I see this:

 

[root@ash-r101-14l ~]# df -h

Filesystem                         Size  Used Avail Use% Mounted on

/dev/mapper/vg_primenode-rootvol    50G  7.4G   40G  16% /

tmpfs                              127G     0  127G   0% /dev/shm

/dev/mapper/vg_primenode-datavol   815G   80G  695G  11% /MSTR

/dev/sda1                           97M   35M   58M  38% /boot

/dev/mapper/vg_primenode-homevol    20G  172M   19G   1% /home

/dev/mapper/vg_primenode-varvol    9.9G  9.4G   16M 100% /var

/dev/mapper/vg_primenode-logvol    9.9G  1.7G  7.8G  18% /var/log

/dev/mapper/vg_primenode-auditvol  9.9G  151M  9.2G   2% /var/log/audit

10.242.105.1:/MSTR/share           815G  113G  661G  15% /MSTR/shared

cm_processes                       127G   56M  127G   1% /var/run/cloudera-scm-agent/process

 

The above tells me that the postgres database files are probably writing to /var directory and hence running short of disk space.

 

How can I fix this? Thanks.

 

 

1 REPLY 1

Re: org.postgresql.util.PSQLException:ERROR:could not extend relation base/17381:No space left on de

Expert Contributor

Hi,

 

Yes, you've zeroed in on the apparent issue which is that your device providing /var

 

/dev/mapper/vg_primenode-varvol    9.9G  9.4G   16M 100% /var

 

is full.

 

You need to remove data from this location in some manner to free some space.

It would be up to you to determine what data you can delete from this location, however 
you may find it possible to remove something old from /var/log. 

 

Do not attempt to manually remove files under

 

/var/lib/cloudera-scm-server-db/ 

 

as these are datafiles for the embedded postgres database. Removing the individual files 

from the filesystem level will most likely just corrupt or fully break the database(s) hosted
therein. 

 

If this is only a test cluster, or you can endure losing metrics and monitoring data, it may

be possible to remove some larger datastores in /var/lib. Ping this topic back if that is the

case and we can discuss what locations those are, but it should not be your first resort.

 

Once you've freed enough space to successfully start - or attempt to start - the postgres database,

you may find there has been corruption to the postgres databases. An ungraceful out-of-space

event can have that unfortunate impact. Hopefully yours starts fine, however. If it does not, 

please post back with any error logs noted.

 

Finally, and once you are in the clear, expand /var space or provide a new volume or mount on which to host /var or /var/lib that has 
more space where you will not run this risk again in the future.

 

Best,

Don't have an account?
Coming from Hortonworks? Activate your account here