Reply
Highlighted
Explorer
Posts: 9
Registered: ‎09-10-2013

Activity monitor bad health issue due to Ident authentication failed

Activity Monitor Log trace:

 

5:01:59.848 PM	WARN	com.mchange.v2.async.ThreadPoolAsynchronousRunner	
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@a1871df -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
5:01:59.853 PM	WARN	com.mchange.v2.async.ThreadPoolAsynchronousRunner	
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@a1871df -- APPARENT DEADLOCK!!! Complete Status: 
	Managed Threads: 3
	Active Threads: 3
	Active Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@106d6599 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7e29283b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@40107a32 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
	Pending Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@79c68979
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@596ba564
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2a377619
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1f2a78f6
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d6b6e76
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@78211410
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@43ac640d
Pool thread stack traces:
	Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
		java.lang.Thread.sleep(Native Method)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
	Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
		java.lang.Thread.sleep(Native Method)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
	Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
		java.lang.Thread.sleep(Native Method)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)


5:02:08.969 PM	WARN	com.mchange.v2.resourcepool.BasicResourcePool	
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@40107a32 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "amon"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:393)
	at org.postgresql.Driver.connect(Driver.java:267)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
5:02:08.970 PM	WARN	com.mchange.v2.resourcepool.BasicResourcePool	
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7e29283b -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "amon"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:393)
	at org.postgresql.Driver.connect(Driver.java:267)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
5:02:08.969 PM	WARN	com.mchange.v2.resourcepool.BasicResourcePool	
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@106d6599 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "amon"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:393)
	at org.postgresql.Driver.connect(Driver.java:267)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
5:02:08.996 PM	WARN	org.hibernate.engine.jdbc.internal.JdbcServicesImpl	
HHH000342: Could not obtain connection to query metadata &colon; Connections could not be acquired from the underlying database!
5:02:09.017 PM	INFO	org.hibernate.dialect.Dialect	
HHH000400: Using dialect: org.hibernate.dialect.PostgreSQLDialect
5:02:09.037 PM	INFO	org.hibernate.engine.jdbc.internal.LobCreatorBuilder	
HHH000422: Disabling contextual LOB creation as connection was null
5:02:09.225 PM	INFO	org.hibernate.engine.transaction.internal.TransactionFactoryInitiator	
HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory
5:02:09.236 PM	INFO	org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory	
HHH000397: Using ASTQueryTranslatorFactory
5:02:09.488 PM	INFO	org.hibernate.cache.spi.UpdateTimestampsCache	
HHH000250: Starting update timestamps cache at region: org.hibernate.cache.spi.UpdateTimestampsCache
5:02:09.493 PM	WARN	org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory	
HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.spi.UpdateTimestampsCache]; using defaults.
5:02:09.498 PM	INFO	org.hibernate.cache.internal.StandardQueryCache	
HHH000248: Starting query cache at region: org.hibernate.cache.internal.StandardQueryCache
5:02:09.499 PM	WARN	org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory	
HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.internal.StandardQueryCache]; using defaults.
5:02:09.564 PM	INFO	org.hibernate.validator.internal.util.Version	
HV000001: Hibernate Validator 5.0.1.Final

 

I have check status of postgresql and db.properties:
service postgresql status
postmaster (pid  14837) is running...
---------------------------------------------------------------------
# These are database settings for CM Manager
#
com.cloudera.cmf.db.type=postgresql
com.cloudera.cmf.db.host=localhost:7432
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=xmW97cV1nO
 
 
Followings are the configuration of Activity Monitor in Cloudera Manager:
 
Activity Monitor Database : localhost
 
Activity Monitor Database Name: amon
 
I have changed the IP-configuration for Cloudera-Manager and after restarting, activity monitor moved to bad health. 

 

Posts: 1,894
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: Activity monitor bad health issue due to Ident authentication failed

Are you using the embedded postgres DB? If so, can you check instead the status of service 'cloudera-scm-server-db'?
Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Re: Activity monitor bad health issue due to Ident authentication failed

[ Edited ]
service cloudera-scm-server-db status
pg_ctl: server is running (PID: 1009)
/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"

 

However, I have the same error:

com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@70db0a99 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "amon"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:393)
	at org.postgresql.Driver.connect(Driver.java:267)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPerTaskAsynchronousRunner$TaskThread.run(ThreadPerTaskAsynchronousRunner.java:255)

 

 

And:

 

cat /var/log/cloudera-scm-server/db.log
FATAL:  could not create shared memory segment: Cannot allocate memory
DETAIL:  Failed system call was shmget(key=7432001, size=7848697856, 03600).
HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space. To reduce the request size (currently 7848697856 bytes), reduce PostgreSQL's shared_buffers parameter (currently 935296) and/or its max_connections parameter (currently 503).
	The PostgreSQL documentation contains more information about shared memory configuration.

 

GHERMAN Alina
Champion Alumni
Posts: 196
Registered: ‎11-18-2014

Re: Activity monitor bad health issue due to Ident authentication failed

In order to make it work, I changed the password for amon. 

 

http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_ig_embed_pstgrs....

 

I didn't understand why this error happend. (No changes on the configuration since yesterday)

GHERMAN Alina
Announcements