@bgooley   Thanks for your response. What is an example of the payload to post to /services. I am using    { "name": streamsets, "type": STREAMSETS } and its failing ... View more

Suddenly my cloudera-scm-server has gone dead, when i restart it it again goes to being dead in 3 secs.   Getting the below  cloudera-scm-server.log output:   JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera_backup Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.cloudera.server.cmf.TrialState': Cannot resolve reference to bean 'entityManagerFactoryBean' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106) at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:616) at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:148) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1003) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:907) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:585) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425) at com.cloudera.server.cmf.Main.bootstrapSpringContext(Main.java:393) at com.cloudera.server.cmf.Main.<init>(Main.java:243) at com.cloudera.server.cmf.Main.main(Main.java:216) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:149) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:102) at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1440) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:247) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:192) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:322) ... 17 more Caused by: javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387) at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310) at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1397) at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:62) at com.cloudera.enterprise.AbstractWrappedEntityManager.beginForRollbackAndReadonly(AbstractWrappedEntityManager.java:89) at com.cloudera.enterprise.dbutil.DbUtil.isInnoDbEnabled(DbUtil.java:549) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.checkMysqlTableEngineType(EntityManagerFactoryBean.java:139) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:122) at com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean.getObject(EntityManagerFactoryBean.java:65) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:142) ... 22 more Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110) at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:221) at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:157) at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67) at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160) at org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1426) at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:59) ... 28 more Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database! at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529) at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128) at org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:84) at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292) at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:214) ... 33 more Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source. at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319) at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557) at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525) ... 37 more ... View more

Thank you.    Yes this worked after the certs were added to jsscacerts      ... View more

Thanks... Following the cloudera Doc's I was able to sucessfully setup Cross-realm trust. Issue is with DNS.  ... View more

Thank you for the information. This does not appear to be the same issue as initiated in this thread.   However, the error is clear: "ERROR Error, CM server guid updated, expected 93c963d9-3566-4f25-847a-faffb020866b, received ba4e4402-6ab3-4cdc-a403-39415df12824"   There is a 'cm_guid' file that is used by the Cloudera Manager (CM) server to know which host belong to it. It appears that this host may have been part of a different CM cluster or the database was recently changed. This keeps users from inadvertently pointing the agent ot another CM instance.   In any case, the solution is to: 1. Archive/remove the /var/lib/cloudera-scm-agent/cm_guid file. 2. Restart the agent: sudo service cloudera-scm-agent restart 3. Run the Host inspector, from the Hosts page in CM and verisfy that all hosts are functioning. If other hosts fail, they too may be in this state.   ... View more

Thanks for the reply!  To be honest, I'm not 100% sure what I'm doing haha.  HDFS is failing to kinit when the service tries to come back up after enabling Kerberos auth.  The above is my attempt to reproduce the error on the command line, since I am not in control when the hdfs service tries to come up.  I forget if I attached this above - this is the service error from the Cloudera Manager when the HDFS service tries to come up: stdout: Fri Feb 23 14:52:47 UTC 2018 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME using 5 as CDH_VERSION using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR using as SECURE_USER using as SECURE_GROUP unlimited /usr/bin/kinit using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache kinit was not successful. Fri Feb 23 14:52:49 UTC 2018 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME using 5 as CDH_VERSION using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR using as SECURE_USER using as SECURE_GROUP unlimited /usr/bin/kinit using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache kinit was not successful. Fri Feb 23 14:52:51 UTC 2018 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME using 5 as CDH_VERSION using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR using as SECURE_USER using as SECURE_GROUP unlimited /usr/bin/kinit using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache kinit was not successful. Fri Feb 23 14:52:54 UTC 2018 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME using 5 as CDH_VERSION using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE as CONF_DIR using as SECURE_USER using as SECURE_GROUP unlimited /usr/bin/kinit using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache kinit was not successful. stderr: + export HADOOP_CLASSPATH=/usr/share/cmf/lib/plugins/event-publish-5.7.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.7.0.jar:/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.6.0-shaded.jar + HADOOP_CLASSPATH=/usr/share/cmf/lib/plugins/event-publish-5.7.0-shaded.jar:/usr/share/cmf/lib/plugins/tt-instrumentation-5.7.0.jar:/usr/share/cmf/lib/plugins/navigator/cdh57/audit-plugin-cdh57-2.6.0-shaded.jar + set -x + replace_conf_dir + find /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE -type f '!' -path '/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/logs/' '!' -name '.log' '!' -name '*.keytab' '!' -name '*jceks' - exec perl -pi -e 's#CMF_CONF_DIR#/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE#g' '{}' ';' + make_scripts_executable + find /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';' + '[' DATANODE_MAX_LOCKED_MEMORY '!=' '' ']' + ulimit -l + export HADOOP_IDENT_STRING=hdfs + HADOOP_IDENT_STRING=hdfs + '[' -n '' ']' + acquire_kerberos_tgt hdfs.keytab + '[' -z hdfs.keytab ']' + '[' -n hdfs/quickstart.cloudera@REALM.COM ']' + '[' -d /usr/kerberos/bin ']' + which kinit + '[' 0 -ne 0 ']' ++ id -u + export KRB5CCNAME=/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 + KRB5CCNAME=/var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 + echo 'using hdfs/quickstart.cloudera@REALM.COM as Kerberos principal' + echo 'using /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 as Kerberos ticket cache' + kinit -c /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/krb5cc_497 -kt /var/run/cloudera-scm-agent/process/29-hdfs-NAMENODE/hdfs.keytab hdfs/quickstart.cloudera@REALM.COM kinit: Generic preauthentication failure while getting initial credentials + '[' 1 -ne 0 ']' + echo 'kinit was not successful.' + exit 1 For the record, I get the same error that I got above if I try to manually use the keytab in the `29-hdfs-NAMENODE` folder. Further, the reason that I added the `host` principal was because authenticating manually without it said it failed on needing the host principal.  So clearly my "reproduction" is not accurate, but I arrived at the same error (and an error that I do not encounter on other hosts), so I was hopeful to be onto something.  No luck so far though. The "Bad encryption type" has completely stymied me.  I have tried a bunch of the other available encryption types to no avail.  (Oh and myfile.keytab was created by me with the commands above since it was my attempt at a repro.  But as I mentioned, using the Cloudera-generated keytab also results in the same issue). ... View more

@GeKas,   You are correct.   Thank you for clarifying that EXTERNAL means NOT EMBEDDED. An external database server can be on the same host as Cloudera Manager.   -Ben ... View more

@nehanirmal,   If you have a new question regarding Spark, try here:   http://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/bd-p/Spark     ... View more

@scld,   Since you are using AES256, the most likely reason beeline cannot find any TGT is that the Unlimited JCE policy file is not installed in the JDK that beeline is using.   Also, why did you download the binaries?  beeline ships with CDH.   @srcnblgc,   Please show us the full command you are using and the output as it appears on the screen.  It is hard to tell what you are doing/seeing.   Check the HiveServer2 logs, as well, to see if there are errors or exceptions when you are having difficulty connecting. ... View more