Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Activity monitor bad health issue due to Ident authentication failed

Highlighted

Activity monitor bad health issue due to Ident authentication failed

Explorer

Activity Monitor Log trace:

 

5:01:59.848 PM	WARN	com.mchange.v2.async.ThreadPoolAsynchronousRunner	
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@a1871df -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
5:01:59.853 PM	WARN	com.mchange.v2.async.ThreadPoolAsynchronousRunner	
com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@a1871df -- APPARENT DEADLOCK!!! Complete Status: 
	Managed Threads: 3
	Active Threads: 3
	Active Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@106d6599 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7e29283b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@40107a32 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
	Pending Tasks: 
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@79c68979
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@596ba564
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2a377619
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1f2a78f6
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1d6b6e76
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@78211410
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@43ac640d
Pool thread stack traces:
	Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
		java.lang.Thread.sleep(Native Method)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
	Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
		java.lang.Thread.sleep(Native Method)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
	Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
		java.lang.Thread.sleep(Native Method)
		com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1805)
		com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)


5:02:08.969 PM	WARN	com.mchange.v2.resourcepool.BasicResourcePool	
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@40107a32 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "amon"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:393)
	at org.postgresql.Driver.connect(Driver.java:267)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
5:02:08.970 PM	WARN	com.mchange.v2.resourcepool.BasicResourcePool	
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@7e29283b -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "amon"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:393)
	at org.postgresql.Driver.connect(Driver.java:267)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
5:02:08.969 PM	WARN	com.mchange.v2.resourcepool.BasicResourcePool	
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@106d6599 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "amon"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:393)
	at org.postgresql.Driver.connect(Driver.java:267)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
5:02:08.996 PM	WARN	org.hibernate.engine.jdbc.internal.JdbcServicesImpl	
HHH000342: Could not obtain connection to query metadata &colon; Connections could not be acquired from the underlying database!
5:02:09.017 PM	INFO	org.hibernate.dialect.Dialect	
HHH000400: Using dialect: org.hibernate.dialect.PostgreSQLDialect
5:02:09.037 PM	INFO	org.hibernate.engine.jdbc.internal.LobCreatorBuilder	
HHH000422: Disabling contextual LOB creation as connection was null
5:02:09.225 PM	INFO	org.hibernate.engine.transaction.internal.TransactionFactoryInitiator	
HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory
5:02:09.236 PM	INFO	org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory	
HHH000397: Using ASTQueryTranslatorFactory
5:02:09.488 PM	INFO	org.hibernate.cache.spi.UpdateTimestampsCache	
HHH000250: Starting update timestamps cache at region: org.hibernate.cache.spi.UpdateTimestampsCache
5:02:09.493 PM	WARN	org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory	
HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.spi.UpdateTimestampsCache]; using defaults.
5:02:09.498 PM	INFO	org.hibernate.cache.internal.StandardQueryCache	
HHH000248: Starting query cache at region: org.hibernate.cache.internal.StandardQueryCache
5:02:09.499 PM	WARN	org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory	
HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.internal.StandardQueryCache]; using defaults.
5:02:09.564 PM	INFO	org.hibernate.validator.internal.util.Version	
HV000001: Hibernate Validator 5.0.1.Final

 

I have check status of postgresql and db.properties:
service postgresql status
postmaster (pid  14837) is running...
---------------------------------------------------------------------
# These are database settings for CM Manager
#
com.cloudera.cmf.db.type=postgresql
com.cloudera.cmf.db.host=localhost:7432
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=xmW97cV1nO
 
 
Followings are the configuration of Activity Monitor in Cloudera Manager:
 
Activity Monitor Database : localhost
 
Activity Monitor Database Name: amon
 
I have changed the IP-configuration for Cloudera-Manager and after restarting, activity monitor moved to bad health. 

 

3 REPLIES 3

Re: Activity monitor bad health issue due to Ident authentication failed

Master Guru
Are you using the embedded postgres DB? If so, can you check instead the status of service 'cloudera-scm-server-db'?

Re: Activity monitor bad health issue due to Ident authentication failed

Champion Alumni
service cloudera-scm-server-db status
pg_ctl: server is running (PID: 1009)
/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"

 

However, I have the same error:

com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@70db0a99 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "amon"
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)
	at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
	at org.postgresql.Driver.makeConnection(Driver.java:393)
	at org.postgresql.Driver.connect(Driver.java:267)
	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:135)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPerTaskAsynchronousRunner$TaskThread.run(ThreadPerTaskAsynchronousRunner.java:255)

 

 

And:

 

cat /var/log/cloudera-scm-server/db.log
FATAL:  could not create shared memory segment: Cannot allocate memory
DETAIL:  Failed system call was shmget(key=7432001, size=7848697856, 03600).
HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space. To reduce the request size (currently 7848697856 bytes), reduce PostgreSQL's shared_buffers parameter (currently 935296) and/or its max_connections parameter (currently 503).
	The PostgreSQL documentation contains more information about shared memory configuration.

 

GHERMAN Alina

Re: Activity monitor bad health issue due to Ident authentication failed

Champion Alumni

In order to make it work, I changed the password for amon. 

 

http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_ig_embed_pstgrs....

 

I didn't understand why this error happend. (No changes on the configuration since yesterday)

GHERMAN Alina
Don't have an account?
Coming from Hortonworks? Activate your account here