Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Navigator Audit Server keeps restarting (without error logs)

Navigator Audit Server keeps restarting (without error logs)

New Contributor

2015-03-10 15:33:33,748 INFO com.cloudera.navigator.NavigatorMain: Starting Cloudera Navigator server...
2015-03-10 15:33:33,913 INFO com.cloudera.enterprise.CommonMain: Reading database properties from /var/run/cloudera-scm-agent/process/135-cloudera-mgmt-NAVIGATOR/db.navigator.properties
2015-03-10 15:33:33,919 INFO com.cloudera.enterprise.CommonMain: Statistics not enabled, c3p0 JMX disabled
2015-03-10 15:33:34,070 INFO org.hibernate.annotations.common.Version: HCANN000001: Hibernate Commons Annotations {4.0.2.Final}
2015-03-10 15:33:34,075 INFO org.hibernate.Version: HHH000412: Hibernate Core {4.2.2.Final}
2015-03-10 15:33:34,077 INFO org.hibernate.cfg.Environment: HHH000206: hibernate.properties not found
2015-03-10 15:33:34,079 INFO org.hibernate.cfg.Environment: HHH000021: Bytecode provider name : javassist
2015-03-10 15:33:34,312 INFO org.hibernate.cfg.Configuration: HHH000221: Reading mappings from resource: META-INF/DbPartitionInfo.hbm.xml
2015-03-10 15:33:34,695 INFO org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator: HHH000130: Instantiating explicit connection provider: org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider
2015-03-10 15:33:34,696 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH010002: C3P0 using driver: org.postgresql.Driver at URL: jdbc:postgresql://vm10-140-22-125.ksc.com:7432/nav
2015-03-10 15:33:34,697 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH000046: Connection properties: {user=nav, password=****, autocommit=true, release_mode=auto}
2015-03-10 15:33:34,697 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH000006: Autocommit mode: true
2015-03-10 15:33:34,713 INFO com.mchange.v2.log.MLog: MLog clients using log4j logging.
2015-03-10 15:33:34,834 INFO com.mchange.v2.c3p0.C3P0Registry: Initializing c3p0-0.9.1.2 [built 21-May-2007 15:04:56; debug? true; trace: 10]
2015-03-10 15:33:34,886 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH000149: JDBC isolation level: READ_COMMITTED
2015-03-10 15:33:34,922 INFO com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@cff032bb [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@7bd02b7a [ acquireIncrement -> 3, acquireRetryAttempts -> 5, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 20000, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2xcprx98q01we78duro|4343dd02, idleConnectionTestPeriod -> 300, initialPoolSize -> 3, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 2500, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@bbbc2d11 [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 2xcprx98q01we78duro|6c759865, jdbcUrl -> jdbc:postgresql://vm10-140-22-125.ksc.com:7432/nav, properties -> {user=******, password=******, autocommit=true, release_mode=auto} ], preferredTestQuery -> null, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 2xcprx98q01we78duro|389f6c33, numHelperThreads -> 3 ]
2015-03-10 15:33:35,206 INFO org.hibernate.dialect.Dialect: HHH000400: Using dialect: org.hibernate.dialect.PostgreSQLDialect
2015-03-10 15:33:35,216 INFO org.hibernate.engine.jdbc.internal.LobCreatorBuilder: HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException
2015-03-10 15:33:35,300 INFO org.hibernate.engine.transaction.internal.TransactionFactoryInitiator: HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory
2015-03-10 15:33:35,304 INFO org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory: HHH000397: Using ASTQueryTranslatorFactory
2015-03-10 15:33:35,401 INFO org.hibernate.cache.spi.UpdateTimestampsCache: HHH000250: Starting update timestamps cache at region: org.hibernate.cache.spi.UpdateTimestampsCache
2015-03-10 15:33:35,403 WARN org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory: HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.spi.UpdateTimestampsCache]; using defaults.
2015-03-10 15:33:35,416 INFO org.hibernate.cache.internal.StandardQueryCache: HHH000248: Starting query cache at region: org.hibernate.cache.internal.StandardQueryCache
2015-03-10 15:33:35,417 WARN org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory: HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.internal.StandardQueryCache]; using defaults.
2015-03-10 15:33:35,446 INFO org.hibernate.validator.internal.util.Version: HV000001: Hibernate Validator 5.0.1.Final
2015-03-10 15:33:36,215 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2015-03-10 15:33:36,219 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.
2015-03-10 15:33:36,220 INFO com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 5003.
2015-03-10 15:33:36,221 INFO com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 5003
2015-03-10 15:33:36,237 INFO com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlEmptyFileRunner: Processed empty file: /usr/share/cmf/cloudera-navigator-audit-server/schema/postgresql/05100_navigator_schema.postgresql.ddl
2015-03-10 15:33:36,446 INFO com.cloudera.navigator.utility.JsonModelLoader: Added audit event model for SENTRY
2015-03-10 15:33:36,472 INFO com.cloudera.navigator.NavigatorEntityManager: Table SENTRY_AUDIT_EVENTS already exists.
2015-03-10 15:33:36,673 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_10
2015-03-10 15:33:36,683 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_11
2015-03-10 15:33:36,687 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:33:36,690 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:33:36,694 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:33:36,697 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:33:36,700 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_10
2015-03-10 15:33:36,704 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_11
2015-03-10 15:33:36,708 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_10
2015-03-10 15:33:36,711 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_11
2015-03-10 15:33:36,966 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-03-10 15:33:37,032 INFO com.cloudera.enterprise.DebugServer: Running debug HTTP server on 0.0.0.0:8089
2015-03-10 15:33:37,033 INFO com.cloudera.enterprise.EnterpriseService: Starting com.cloudera.enterprise.dbpartition.PartitionService
2015-03-10 15:33:37,033 INFO com.cloudera.enterprise.EnterpriseService: Starting com.cloudera.enterprise.dbpartition.PartitionExpirationService
2015-03-10 15:33:37,036 INFO com.cloudera.enterprise.EnterpriseService: Starting ActivityPollingService
2015-03-10 15:33:37,036 INFO com.cloudera.enterprise.EnterpriseService: Starting ActivityAuditPurgeService
2015-03-10 15:33:37,037 INFO com.cloudera.enterprise.EnterpriseService: Starting Navigator Debug Server
2015-03-10 15:33:37,037 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:33:37.037Z for table HDFS_AUDIT_EVENTS
2015-03-10 15:33:37,043 INFO org.mortbay.log: jetty-6.1.26.cloudera.4
2015-03-10 15:33:37,045 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_10
2015-03-10 15:33:37,052 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_11
2015-03-10 15:33:37,058 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:33:37,064 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:33:37,070 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:33:37.070Z for table HBASE_AUDIT_EVENTS
2015-03-10 15:33:37,071 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:33:37.071Z for table HIVE_AUDIT_EVENTS
2015-03-10 15:33:37,071 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:33:37.071Z for table IMPALA_AUDIT_EVENTS
2015-03-10 15:33:37,072 INFO org.mortbay.log: Started SocketConnector@0.0.0.0:8089
2015-03-10 15:33:37,072 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:33:37.072Z for table SENTRY_AUDIT_EVENTS
2015-03-10 15:33:37,076 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:33:37,078 INFO com.cloudera.enterprise.EnterpriseService: Starting NavigatorServer
2015-03-10 15:33:37,083 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:33:37,086 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_10
2015-03-10 15:33:37,089 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_11
2015-03-10 15:33:37,092 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_10
2015-03-10 15:33:37,095 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_11
2015-03-10 15:33:37,187 INFO org.mortbay.log: jetty-6.1.26.cloudera.4
2015-03-10 15:33:37,192 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7186
2015-03-10 15:33:37,193 INFO com.cloudera.navigator.NavigatorServer: Running NavigatorIPC on port 7186





































2015-03-10 15:34:00,077 INFO com.cloudera.navigator.NavigatorMain: Starting Cloudera Navigator server...
2015-03-10 15:34:00,283 INFO com.cloudera.enterprise.CommonMain: Reading database properties from /var/run/cloudera-scm-agent/process/135-cloudera-mgmt-NAVIGATOR/db.navigator.properties
2015-03-10 15:34:00,293 INFO com.cloudera.enterprise.CommonMain: Statistics not enabled, c3p0 JMX disabled
2015-03-10 15:34:00,475 INFO org.hibernate.annotations.common.Version: HCANN000001: Hibernate Commons Annotations {4.0.2.Final}
2015-03-10 15:34:00,481 INFO org.hibernate.Version: HHH000412: Hibernate Core {4.2.2.Final}
2015-03-10 15:34:00,483 INFO org.hibernate.cfg.Environment: HHH000206: hibernate.properties not found
2015-03-10 15:34:00,485 INFO org.hibernate.cfg.Environment: HHH000021: Bytecode provider name : javassist
2015-03-10 15:34:00,712 INFO org.hibernate.cfg.Configuration: HHH000221: Reading mappings from resource: META-INF/DbPartitionInfo.hbm.xml
2015-03-10 15:34:01,111 INFO org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator: HHH000130: Instantiating explicit connection provider: org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider
2015-03-10 15:34:01,112 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH010002: C3P0 using driver: org.postgresql.Driver at URL: jdbc:postgresql://vm10-140-22-125.ksc.com:7432/nav
2015-03-10 15:34:01,113 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH000046: Connection properties: {user=nav, password=****, autocommit=true, release_mode=auto}
2015-03-10 15:34:01,113 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH000006: Autocommit mode: true
2015-03-10 15:34:01,132 INFO com.mchange.v2.log.MLog: MLog clients using log4j logging.
2015-03-10 15:34:01,262 INFO com.mchange.v2.c3p0.C3P0Registry: Initializing c3p0-0.9.1.2 [built 21-May-2007 15:04:56; debug? true; trace: 10]
2015-03-10 15:34:01,319 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH000149: JDBC isolation level: READ_COMMITTED
2015-03-10 15:34:01,355 INFO com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@6f5f9214 [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@b64848af [ acquireIncrement -> 3, acquireRetryAttempts -> 5, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 20000, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2xcprx98q0ma71ykii13|6cb0642f, idleConnectionTestPeriod -> 300, initialPoolSize -> 3, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 2500, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@b93e4ac6 [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 2xcprx98q0ma71ykii13|2addbc17, jdbcUrl -> jdbc:postgresql://vm10-140-22-125.ksc.com:7432/nav, properties -> {user=******, password=******, autocommit=true, release_mode=auto} ], preferredTestQuery -> null, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 2xcprx98q0ma71ykii13|4e1900e4, numHelperThreads -> 3 ]
2015-03-10 15:34:01,619 INFO org.hibernate.dialect.Dialect: HHH000400: Using dialect: org.hibernate.dialect.PostgreSQLDialect
2015-03-10 15:34:01,630 INFO org.hibernate.engine.jdbc.internal.LobCreatorBuilder: HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException
2015-03-10 15:34:01,724 INFO org.hibernate.engine.transaction.internal.TransactionFactoryInitiator: HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory
2015-03-10 15:34:01,729 INFO org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory: HHH000397: Using ASTQueryTranslatorFactory
2015-03-10 15:34:01,831 INFO org.hibernate.cache.spi.UpdateTimestampsCache: HHH000250: Starting update timestamps cache at region: org.hibernate.cache.spi.UpdateTimestampsCache
2015-03-10 15:34:01,834 WARN org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory: HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.spi.UpdateTimestampsCache]; using defaults.
2015-03-10 15:34:01,849 INFO org.hibernate.cache.internal.StandardQueryCache: HHH000248: Starting query cache at region: org.hibernate.cache.internal.StandardQueryCache
2015-03-10 15:34:01,850 WARN org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory: HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.internal.StandardQueryCache]; using defaults.
2015-03-10 15:34:01,882 INFO org.hibernate.validator.internal.util.Version: HV000001: Hibernate Validator 5.0.1.Final
2015-03-10 15:34:02,719 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2015-03-10 15:34:02,723 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.
2015-03-10 15:34:02,724 INFO com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 5003.
2015-03-10 15:34:02,725 INFO com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 5003
2015-03-10 15:34:02,741 INFO com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlEmptyFileRunner: Processed empty file: /usr/share/cmf/cloudera-navigator-audit-server/schema/postgresql/05100_navigator_schema.postgresql.ddl
2015-03-10 15:34:02,964 INFO com.cloudera.navigator.utility.JsonModelLoader: Added audit event model for SENTRY
2015-03-10 15:34:02,990 INFO com.cloudera.navigator.NavigatorEntityManager: Table SENTRY_AUDIT_EVENTS already exists.
2015-03-10 15:34:03,216 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:03,225 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:03,229 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:03,232 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:03,236 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:03,239 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:03,242 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:03,245 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:03,250 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:03,253 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:03,492 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-03-10 15:34:03,561 INFO com.cloudera.enterprise.DebugServer: Running debug HTTP server on 0.0.0.0:8089
2015-03-10 15:34:03,561 INFO com.cloudera.enterprise.EnterpriseService: Starting com.cloudera.enterprise.dbpartition.PartitionService
2015-03-10 15:34:03,562 INFO com.cloudera.enterprise.EnterpriseService: Starting com.cloudera.enterprise.dbpartition.PartitionExpirationService
2015-03-10 15:34:03,562 INFO com.cloudera.enterprise.EnterpriseService: Starting ActivityPollingService
2015-03-10 15:34:03,562 INFO com.cloudera.enterprise.EnterpriseService: Starting ActivityAuditPurgeService
2015-03-10 15:34:03,562 INFO com.cloudera.enterprise.EnterpriseService: Starting Navigator Debug Server
2015-03-10 15:34:03,563 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:34:03.563Z for table HDFS_AUDIT_EVENTS
2015-03-10 15:34:03,566 INFO org.mortbay.log: jetty-6.1.26.cloudera.4
2015-03-10 15:34:03,572 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:03,576 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:03,583 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:03,586 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:03,590 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:34:03.590Z for table HBASE_AUDIT_EVENTS
2015-03-10 15:34:03,597 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:34:03.596Z for table HIVE_AUDIT_EVENTS
2015-03-10 15:34:03,598 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:34:03.598Z for table IMPALA_AUDIT_EVENTS
2015-03-10 15:34:03,599 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:03,600 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:34:03.600Z for table SENTRY_AUDIT_EVENTS
2015-03-10 15:34:03,607 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:03,622 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:03,626 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:03,628 INFO org.mortbay.log: Started SocketConnector@0.0.0.0:8089
2015-03-10 15:34:03,629 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:03,633 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:03,633 INFO com.cloudera.enterprise.EnterpriseService: Starting NavigatorServer
2015-03-10 15:34:03,747 INFO org.mortbay.log: jetty-6.1.26.cloudera.4
2015-03-10 15:34:03,753 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7186
2015-03-10 15:34:03,753 INFO com.cloudera.navigator.NavigatorServer: Running NavigatorIPC on port 7186











2015-03-10 15:34:24,412 INFO com.cloudera.navigator.NavigatorMain: Starting Cloudera Navigator server...
2015-03-10 15:34:24,588 INFO com.cloudera.enterprise.CommonMain: Reading database properties from /var/run/cloudera-scm-agent/process/135-cloudera-mgmt-NAVIGATOR/db.navigator.properties
2015-03-10 15:34:24,594 INFO com.cloudera.enterprise.CommonMain: Statistics not enabled, c3p0 JMX disabled
2015-03-10 15:34:24,757 INFO org.hibernate.annotations.common.Version: HCANN000001: Hibernate Commons Annotations {4.0.2.Final}
2015-03-10 15:34:24,763 INFO org.hibernate.Version: HHH000412: Hibernate Core {4.2.2.Final}
2015-03-10 15:34:24,765 INFO org.hibernate.cfg.Environment: HHH000206: hibernate.properties not found
2015-03-10 15:34:24,767 INFO org.hibernate.cfg.Environment: HHH000021: Bytecode provider name : javassist
2015-03-10 15:34:24,997 INFO org.hibernate.cfg.Configuration: HHH000221: Reading mappings from resource: META-INF/DbPartitionInfo.hbm.xml
2015-03-10 15:34:25,383 INFO org.hibernate.service.jdbc.connections.internal.ConnectionProviderInitiator: HHH000130: Instantiating explicit connection provider: org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider
2015-03-10 15:34:25,385 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH010002: C3P0 using driver: org.postgresql.Driver at URL: jdbc:postgresql://vm10-140-22-125.ksc.com:7432/nav
2015-03-10 15:34:25,385 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH000046: Connection properties: {user=nav, password=****, autocommit=true, release_mode=auto}
2015-03-10 15:34:25,385 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH000006: Autocommit mode: true
2015-03-10 15:34:25,403 INFO com.mchange.v2.log.MLog: MLog clients using log4j logging.
2015-03-10 15:34:25,519 INFO com.mchange.v2.c3p0.C3P0Registry: Initializing c3p0-0.9.1.2 [built 21-May-2007 15:04:56; debug? true; trace: 10]
2015-03-10 15:34:25,572 INFO org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider: HHH000149: JDBC isolation level: READ_COMMITTED
2015-03-10 15:34:25,607 INFO com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource: Initializing c3p0 pool... com.mchange.v2.c3p0.PoolBackedDataSource@2ee52ca1 [ connectionPoolDataSource -> com.mchange.v2.c3p0.WrapperConnectionPoolDataSource@42f17f94 [ acquireIncrement -> 3, acquireRetryAttempts -> 5, acquireRetryDelay -> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 20000, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, debugUnreturnedConnectionStackTraces -> false, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 2xcprx98q150dfyp6mf|7b9880b2, idleConnectionTestPeriod -> 300, initialPoolSize -> 3, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 0, maxIdleTime -> 0, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 50, maxStatements -> 2500, maxStatementsPerConnection -> 0, minPoolSize -> 5, nestedDataSource -> com.mchange.v2.c3p0.DriverManagerDataSource@fa5c35a9 [ description -> null, driverClass -> null, factoryClassLocation -> null, identityToken -> 2xcprx98q150dfyp6mf|36928c0f, jdbcUrl -> jdbc:postgresql://vm10-140-22-125.ksc.com:7432/nav, properties -> {user=******, password=******, autocommit=true, release_mode=auto} ], preferredTestQuery -> null, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false; userOverrides: {} ], dataSourceName -> null, factoryClassLocation -> null, identityToken -> 2xcprx98q150dfyp6mf|646d5273, numHelperThreads -> 3 ]
2015-03-10 15:34:25,864 INFO org.hibernate.dialect.Dialect: HHH000400: Using dialect: org.hibernate.dialect.PostgreSQLDialect
2015-03-10 15:34:25,873 INFO org.hibernate.engine.jdbc.internal.LobCreatorBuilder: HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException
2015-03-10 15:34:25,969 INFO org.hibernate.engine.transaction.internal.TransactionFactoryInitiator: HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory
2015-03-10 15:34:25,974 INFO org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory: HHH000397: Using ASTQueryTranslatorFactory
2015-03-10 15:34:26,081 INFO org.hibernate.cache.spi.UpdateTimestampsCache: HHH000250: Starting update timestamps cache at region: org.hibernate.cache.spi.UpdateTimestampsCache
2015-03-10 15:34:26,084 WARN org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory: HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.spi.UpdateTimestampsCache]; using defaults.
2015-03-10 15:34:26,098 INFO org.hibernate.cache.internal.StandardQueryCache: HHH000248: Starting query cache at region: org.hibernate.cache.internal.StandardQueryCache
2015-03-10 15:34:26,099 WARN org.hibernate.cache.ehcache.AbstractEhcacheRegionFactory: HHH020003: Could not find a specific ehcache configuration for cache named [org.hibernate.cache.internal.StandardQueryCache]; using defaults.
2015-03-10 15:34:26,132 INFO org.hibernate.validator.internal.util.Version: HV000001: Hibernate Validator 5.0.1.Final
2015-03-10 15:34:26,978 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table doesn't exist.
2015-03-10 15:34:26,981 INFO com.cloudera.enterprise.dbutil.DbUtil: Schema version table already exists.
2015-03-10 15:34:26,983 INFO com.cloudera.enterprise.dbutil.DbUtil: DB Schema version 5003.
2015-03-10 15:34:26,983 INFO com.cloudera.enterprise.dbutil.DbUtil: Current database schema version: 5003
2015-03-10 15:34:26,998 INFO com.cloudera.enterprise.dbutil.SqlScriptRunnerFactory$SqlEmptyFileRunner: Processed empty file: /usr/share/cmf/cloudera-navigator-audit-server/schema/postgresql/05100_navigator_schema.postgresql.ddl
2015-03-10 15:34:27,227 INFO com.cloudera.navigator.utility.JsonModelLoader: Added audit event model for SENTRY
2015-03-10 15:34:27,253 INFO com.cloudera.navigator.NavigatorEntityManager: Table SENTRY_AUDIT_EVENTS already exists.
2015-03-10 15:34:27,471 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:27,482 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:27,486 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:27,490 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:27,494 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:27,497 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:27,501 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:27,504 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:27,509 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:27,512 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:27,760 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-03-10 15:34:27,830 INFO com.cloudera.enterprise.DebugServer: Running debug HTTP server on 0.0.0.0:8089
2015-03-10 15:34:27,830 INFO com.cloudera.enterprise.EnterpriseService: Starting com.cloudera.enterprise.dbpartition.PartitionService
2015-03-10 15:34:27,831 INFO com.cloudera.enterprise.EnterpriseService: Starting com.cloudera.enterprise.dbpartition.PartitionExpirationService
2015-03-10 15:34:27,831 INFO com.cloudera.enterprise.EnterpriseService: Starting ActivityPollingService
2015-03-10 15:34:27,832 INFO com.cloudera.enterprise.EnterpriseService: Starting ActivityAuditPurgeService
2015-03-10 15:34:27,832 INFO com.cloudera.enterprise.EnterpriseService: Starting Navigator Debug Server
2015-03-10 15:34:27,832 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:34:27.832Z for table HDFS_AUDIT_EVENTS
2015-03-10 15:34:27,833 INFO org.mortbay.log: jetty-6.1.26.cloudera.4
2015-03-10 15:34:27,841 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:27,845 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HDFS_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:27,858 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:27,862 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HBASE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:27,870 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:34:27.869Z for table HBASE_AUDIT_EVENTS
2015-03-10 15:34:27,871 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:34:27.871Z for table HIVE_AUDIT_EVENTS
2015-03-10 15:34:27,871 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:27,872 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:34:27.872Z for table IMPALA_AUDIT_EVENTS
2015-03-10 15:34:27,872 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Aging out long-lived data older than 2014-12-10T07:34:27.872Z for table SENTRY_AUDIT_EVENTS
2015-03-10 15:34:27,874 INFO org.mortbay.log: Started SocketConnector@0.0.0.0:8089
2015-03-10 15:34:27,878 INFO com.cloudera.enterprise.EnterpriseService: Starting NavigatorServer
2015-03-10 15:34:27,896 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: HIVE_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:27,900 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:27,902 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: IMPALA_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:27,912 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_10
2015-03-10 15:34:27,925 INFO com.cloudera.enterprise.dbpartition.PartitionEntityManager: Skipping addition of partition which already exists: SENTRY_AUDIT_EVENTS_2015_03_11
2015-03-10 15:34:28,000 INFO org.mortbay.log: jetty-6.1.26.cloudera.4
2015-03-10 15:34:28,007 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:7186
2015-03-10 15:34:28,009 INFO com.cloudera.navigator.NavigatorServer: Running NavigatorIPC on port 7186

2 REPLIES 2

Re: Navigator Audit Server keeps restarting (without error logs)

Community Manager

Just to clarify, you are seeing the Audit server restarting multiple times, but there are no errors reported in between restarts?

 

The restarts are most likely due to Auto-Restart being enabled by default.  So the question is why is it stopping?

 

In Cloudera Manager, go to Management Services -> Instances -> Navigator Audit Server -> Commands

 

On this page, if you do not see a start command, expand the time frame by clicking one of the time values near the upper right corner, like 6h.   Once you see the command " Start this Navigator Audit Server", look at the stderr.log.  You may be hitting an OOM condition.



David Wilder, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Highlighted

Re: Navigator Audit Server keeps restarting (without error logs)

Explorer

In my case, Navigator Metadata server is restarting. stdout.log has following info. How to stop this memory leak?Already allocated 30GB

 

# java.lang.OutOfMemoryError: PermGen space
# -XX:OnOutOfMemoryError="/usr/lib64/cmf/service/common/killparent.sh"
#   Executing /bin/sh -c "/usr/lib64/cmf/service/common/killparent.sh"...