Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Getting Kerberos to work after reinstalling FreeIPA server (CDH v5.3, Data Hub edition)

Highlighted

Getting Kerberos to work after reinstalling FreeIPA server (CDH v5.3, Data Hub edition)

Contributor

This is probably not a very common scenario but can certainly happen to many who use Kerberos along with some OpenLDAP solution. We are using FreeIPA for users management, and have recently successfully configured Kerberos following  Cloudera's documentation for configuring authentication using the Coudera Manager wizard (**because our cluster uses FreeIPA and FreeIPA does not allow direct access to Kerberos commands used to create principals, some deviations were necessary).

 

Because the server running FreeIPA server had to be rebuilt, we ended up with a fresh install and lost all of the FreeIPA config files stored there. We have made sure all Cloudera nodes have rejoined the cluster. We are, however, unable to start all services via Cloudera Manager, and the following warning shows up in the logs (from Namenode):

 

10:35:05.198 PMWARN

StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread

 

Failed to evict stale entities
java.lang.RuntimeException: javax.persistence.RollbackException: Error while committing the transaction
	at com.cloudera.cmf.persist.DatabaseExecutor.execReadWriteTaskNE(DatabaseExecutor.java:77)
	at com.cloudera.cmf.command.components.CommandManager.batchDeleteCommands(CommandManager.java:48)
	at com.cloudera.server.cmf.StaleEntityEvictionThread.reapDeletedCommands(StaleEntityEvictionThread.java:237)
	at com.cloudera.server.cmf.StaleEntityEvictionThread.innerLoop(StaleEntityEvictionThread.java:375)
	at com.cloudera.server.cmf.StaleEntityEvictionThread.run(StaleEntityEvictionThread.java:123)
Caused by: javax.persistence.RollbackException: Error while committing the transaction
	at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:92)
	at com.cloudera.enterprise.AbstractWrappedEntityManager.commit(AbstractWrappedEntityManager.java:110)
	at com.cloudera.cmf.persist.CmfEntityManager.commit(CmfEntityManager.java:366)
	at com.cloudera.cmf.persist.ReadWriteDatabaseTaskCallable.call(ReadWriteDatabaseTaskCallable.java:37)
	at com.cloudera.cmf.persist.DatabaseExecutor.execTask(DatabaseExecutor.java:92)
	at com.cloudera.cmf.persist.DatabaseExecutor.execReadWriteTask(DatabaseExecutor.java:66)
	at com.cloudera.cmf.persist.DatabaseExecutor.execReadWriteTaskNE(DatabaseExecutor.java:75)
	... 4 more
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement
	at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1387)
	at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1310)
	at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:80)
	... 10 more
Caused by: org.hibernate.exception.ConstraintViolationException: could not execute statement
	at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:129)
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:49)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110)
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:136)
	at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:3346)
	at org.hibernate.persister.entity.AbstractEntityPersister.delete(AbstractEntityPersister.java:3546)
	at org.hibernate.action.internal.EntityDeleteAction.execute(EntityDeleteAction.java:100)
	at org.hibernate.engine.spi.ActionQueue.execute(ActionQueue.java:377)
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:369)
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:293)
	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:339)
	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:52)
	at org.hibernate.internal.SessionImpl.flush(SessionImpl.java:1234)
	at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:404)
	at org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.beforeTransactionCommit(JdbcTransaction.java:101)
	at org.hibernate.engine.transaction.spi.AbstractTransactionImpl.commit(AbstractTransactionImpl.java:175)
	at org.hibernate.ejb.TransactionImpl.commit(TransactionImpl.java:75)
	... 10 more
Caused by: org.postgresql.util.PSQLException: ERROR: update or delete on table "commands" violates foreign key constraint "fk_command_parent" on table "commands"
  Detail: Key (command_id)=(2437) is still referenced from table "commands".
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:334)
	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105)
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:133)
	... 23 more

 

Also:

 

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

Re-generating Kerberos credentials in Cloudera Manager fails with (the script referenced in below error is custom script written by our team, it takes some configuration input from the Cloudera manager, creates principals for Hadoop, and saves their keytabs (passwords) in locations that are accessible to Hadoop services):

 

/usr/share/cmf/bin/gen_credentials.sh failed with exit code 1 and output of <<
kinit: Password incorrect while getting initial credentials
ipa: ERROR: Kerberos error: ('Unspecified GSS failure.  Minor code may provide more information', 851968)/('Decrypt integrity check failed', -1765328353)
SASL Bind failed Local error (-2) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information (Decrypt integrity check failed)!
chmod: cannot access `/var/run/cloudera-scm-server/cmf266477689285916699.keytab': No such file or directory

>>

 

1 REPLY 1

Re: Getting Kerberos to work after reinstalling FreeIPA server (CDH v5.3, Data Hub edition)

Contributor

Would it be possible to share you gen_credentials.sh script?  I am looking to utilize FreeIPA at serveral client sites and am interested in all the bits to make this work.

Don't have an account?
Coming from Hortonworks? Activate your account here