Reply
New Contributor
Posts: 5
Registered: ‎06-21-2016

[cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeployed

[ Edited ]

Hi,

 

We did some tests to see how usable a Kerberized cluster would be, and now we have decided to move on to a more production-like setup. We are starting fresh with a new realm, since the one we had initially picked will not work with our final naming convention.

 

I have been uninstalling Kerberos and deleting all Kerberos databases. The hostnames have also been changed (and changed within cloudera) to have fqdn which match the new realm. I have then been recreating everything related to Kerberos, and I can use kinit correctly, as long as I am on the machine running the KDC.

 

The configuration has also been updated in the Cloudera interface:

- KDC server host

Kerberos Security Realm

 

I have been trying to push manually the new /etc/krb5.conf, copying it on all hosts, in order to regenerate the credentials. If I deactivate "Manage krb5.conf through Cloudera Manager", I can then use kinit correctly on those hosts. But as soon as I enable it, the old (wrong) krb5.conf is redeployed. And anyway, when I try to regenerate the credentials and krb5.conf is not managed by Cloudera, it fails because some commands are still assuming the old realm.

 

I have been scanning the Cloudera Manager for this dirty config file, but I can't find where it is kept. Could you please help me and tell how to make Cloudera forget about this old Kerberos realm?

 

Thank you!

New Contributor
Posts: 5
Registered: ‎06-21-2016

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

I am a bit desperate here. I scanned quite exaustively the Cloudera manager host for occurences of the old realm, and besides log files, the only occurrences are found in '/var/lib/cloudera-scm-server-db/'. Does it mean I should edit the db manually with pgsql? How do I do that?

 

Thank you

New Contributor
Posts: 5
Registered: ‎06-21-2016

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

All right, so pursuing my investigation, here is what I did:

 

I found information about how to access the internal database here

https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_ig_embed_pstgrs.html

 

Basically, you'll find the info about the db in '/etc/cloudera-scm-server/db.properties', and '/var/lib/cloudera-scm-server-db/data/generated_password.txt'. From there, using a bit of postgresql foo, I managed to dump all tables of the database 'scm' as csv files.

 

I found out that 'kdc_admin_user' from the table 'configs' is still using the old realm. Not sure what to do yet with this information...

New Contributor
Posts: 5
Registered: ‎06-21-2016

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

So we went ahead and updated the value of 'kdc_admin_user' in the table 'configs'.

 

This yields better results, because now the commands I read in the log file of the server look like they could work. Before, we were getting:

 

KADMIN='kadmin -k -t /var/run/cloudera-scm-server/cmf9109851999959873240.keytab -p cloudera-scm/admin@OLDREALM -r OLDREALM

 

Now we get

 

KADMIN='kadmin -k -t /var/run/cloudera-scm-server/cmf9109851999959873240.keytab -p cloudera-scm/admin@NEWREALM -r NEWREALM'

 

The error is now:

 

kadmin: Key table entry not found while initializing kadmin interface

 

This makes sense, I think that cloudera is trying to use an outdated keytab to log as cloudera-scm/admin. I can only assume that this keytab is stored somewhere in the postgresql database. How could we find and update this entry?

 

New Contributor
Posts: 5
Registered: ‎06-21-2016

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

Sorry to bump the thread like this, but this is still an issue.

 

Does the lack of answer mean that no solution exists and we should re-install the whole cluster?

Posts: 869
Kudos: 100
Solutions: 53
Registered: ‎04-06-2015

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

I would say that it is most likely a lack of other users who have faced a similar issue. The community is mainly peer to peer but Clouderans do participate as time allows. Sometimes I can help by pointing out some documentation or a blog post but I cannot think of one that fits your situation. 



Cy Jervis, Community Manager


Was your question answered? Make sure to mark it as an accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

Highlighted
Explorer
Posts: 9
Registered: ‎07-18-2017

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

I have the same issue.

I have investigated the issue and it seems cloudera-manager-agent is generating krb5.conf file, I have tested with all hadoop services, cloudera manager, cloudera manager services and cloudera manager's database (in my case MySQL) in down state and if you restart cloudera manager agent then it will update the krb5.conf file. I have deleted the krb5.conf file but cloudera manager agent has re-created with old realm.

I don't know how its being re-created on every restart of agent.

Anyone here who provide input to resolve the issue.

Thanks
Posts: 948
Topics: 1
Kudos: 223
Solutions: 120
Registered: ‎04-22-2014

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

@bhaveshvv109,

 

I am a bit confused about what you are trying to do and what is happening.

I'll start off with how things should work:

 

If you are managing your own krb5.conf, then disable Manage krb5.conf through Cloudera Manager in your Cloudera Manager Kerberos configuration.

 

If that is disabled, you can then distribute your own /etc/krb5.conf files to all nodes and Cloudera Manager will not touch them.

 

Even if Manage krb5.conf through Cloudera Manager is enabled, Cloudera Manager does not push out krb5.conf files automatically.  It is a fully manual process.  The agent itself does not create /etc/krb5.conf.

 

The only way the agent could be doing what you say is if somehow there is a problem where the "DeployClientConfigsOfCluster" was running repeatedly.  We would see this in the agent log of the impacted host.

If this is happening, please share the agent log file with us or at least the parts that show evidence the agent is creating /etc/krb5.conf.

 

 

Explorer
Posts: 9
Registered: ‎07-18-2017

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

Hi @bgooley

 

Thanks for your quick reply.

 

I am playing with Cloudera Sandbox 5.13, (this issue present with 5.8 and 5.10 as well) we are plannig to implement kerberos on our PROD cluster so I wanted to try all config and wanted to check if we can update if in case we might need to in future.

 

I have checked the agent's log file but there is not such log entry for "DeployClientConfigsOfCluster".

 

Here are steps to replicate the issue:

 

1) Configure kerberos KDC with default realm as HADOOP.COM

2) Updated same in cloudera manager using "Enable Kerberos" wizard and select Manage krb5.conf through Cloudera Manager

3) Check all services are running without any issue

4) Now, destroy the kerberos KDC db using command "kdb5_util detroy" to update default realm to "QUICKSTART.CLOUDERA".

5)Update default realm in /etc/krb5.conf manually and re-create KDC db using "kdb5_util create -s"

6) Update default_realm in Cloudera Manager and Import Kerberos Account Manager Credentials

7) re-generate credentials (service principals) using "Generate Missing Credentials"

8) Re-start all services/roles

9) Now, restart Cloudera Manager Agent to check if it's taking updated default_realm or old. It will create krb5.conf with old default_realm

10) You can also test by removing /etc/krb5.conf file and restart Cloudera Manager Agent. It will re-create krb5.conf with old default_realm.

 

PFA Cloudera Manager Agent log file if it helps to identify the issue.

 

Thanks,

Bhavesh

 

 

Posts: 948
Topics: 1
Kudos: 223
Solutions: 120
Registered: ‎04-22-2014

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

@bhaveshvv109,

 

I would try restarting Cloudera Manager with the following:

 

# service cloudera-scm-server restart

 

It sounds to me as if the krb5.conf file elements from before the realm change may be retained in memory.

Restarting should help us define if it is a transient issue or if the old realm is being loaded from the CM database.

 

If restarting does not solve the issue for you, please provide access to the CM and the agent logs from the time you are seeing the issue.

 

The agent doest not have any code in it to update krb5.conf, so my point was that is would have to be taking its instructions from Cloudera Manager.

Announcements