Reply
Explorer
Posts: 9
Registered: ‎07-18-2017

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

Hi @bgooley

 

I don't think this is transient issue as restarting of the VM(Quickstart Sandbox VM) doesn't solved the issue.

 

As I stated in my previous post, I have shutdown all daemon processes related to Cloudera (Cloudera Manager Server, Management Services, All CDH Components and MySQL DB) and just restarted the Cloudera Manager Agent, it has re-deployed the old krb5.conf in /etc, So that means Agent is not taking any instruction from any of the Cloudera or Hadoop components.

 

I seems the Cloudera Manager Server or Agent keeps old copy of krb5.conf somewhere and on restart of agent is just copying config file from one place to another place.

 

You can check ccdeploy_etc_0's log, it has log entry to deploy krb5.conf file. Here is the log for your reference:

 

Sat Aug 25 19:17:57 IST 2018

+ locate_java_home_no_verify

+ JAVA6_HOME_CANDIDATES=('/usr/lib/j2sdk1.6-sun' '/usr/lib/jvm/java-6-sun' '/usr/lib/jvm/java-1.6.0-sun-1.6.0' '/usr/lib/jvm/j2sdk1.6-oracle' '/usr/lib/jvm/j2sdk1.6-oracle/jre' '/usr/java/jdk1.6' '/usr/java/jre1.6')

+ local JAVA6_HOME_CANDIDATES

+ OPENJAVA6_HOME_CANDIDATES=('/usr/lib/jvm/java-1.6.0-openjdk' '/usr/lib/jvm/jre-1.6.0-openjdk')

+ local OPENJAVA6_HOME_CANDIDATES

+ JAVA7_HOME_CANDIDATES=('/usr/java/jdk1.7' '/usr/java/jre1.7' '/usr/lib/jvm/j2sdk1.7-oracle' '/usr/lib/jvm/j2sdk1.7-oracle/jre' '/usr/lib/jvm/java-7-oracle')

+ local JAVA7_HOME_CANDIDATES

+ OPENJAVA7_HOME_CANDIDATES=('/usr/lib/jvm/java-1.7.0-openjdk' '/usr/lib/jvm/java-7-openjdk')

+ local OPENJAVA7_HOME_CANDIDATES

+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')

+ local JAVA8_HOME_CANDIDATES

+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk')

+ local OPENJAVA8_HOME_CANDIDATES

+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')

+ local MISCJAVA_HOME_CANDIDATES

+ case ${BIGTOP_JAVA_MAJOR} in

+ JAVA_HOME_CANDIDATES=(${JAVA7_HOME_CANDIDATES[@]} ${JAVA8_HOME_CANDIDATES[@]} ${JAVA6_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA7_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${OPENJAVA6_HOME_CANDIDATES[@]})

+ '[' -z '' ']'

+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'

++ ls -rvd /usr/java/jdk1.7.0_67-cloudera

+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'

+ '[' -e /usr/java/jdk1.7.0_67-cloudera/bin/java ']'

+ export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera

+ JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera

+ break 2

+ source_parcel_environment

+ '[' '!' -z '' ']'

+ echo 'using /usr/java/jdk1.7.0_67-cloudera as JAVA_HOME'

+ echo 'using -1 as CDH_VERSION'

+ echo 'using /var/run/cloudera-scm-agent/process/ccdeploy_etc_0 as CONF_DIR'

+ echo 'using krb-conf as DIRECTORY_NAME'

+ echo 'using /etc as DEST_PATH'

+ echo 'using  as ALT_NAME'

+ echo 'using  as ALT_LINK'

+ echo 'using -1 as PRIORITY'

+ echo 'using  as RUNNER_PROGRAM'

+ echo 'using  as RUNNER_ARGS'

+ echo 'using /usr/sbin/update-alternatives as UPDATE_ALTERNATIVES'

+ set -x

+ set -e

+ [[ -1 == -1 ]]

+ echo 'Deploying cluster client configs to /etc'

+ [[ -e /etc ]]

+ [[ ! -d /etc ]]

+ for TOCOPY in '"$CONF_DIR/$DIRECTORY_NAME/"*'

++ basename /var/run/cloudera-scm-agent/process/ccdeploy_etc_0/krb-conf/__cloudera_generation__

+ DEST_FILE=/etc/__cloudera_generation__

+ cp -a /var/run/cloudera-scm-agent/process/ccdeploy_etc_0/krb-conf/__cloudera_generation__ /etc/__cloudera_generation__.cm_tmp

+ chmod -R ugo+r /etc/__cloudera_generation__.cm_tmp

+ mv -Tf /etc/__cloudera_generation__.cm_tmp /etc/__cloudera_generation__

+ for TOCOPY in '"$CONF_DIR/$DIRECTORY_NAME/"*'

++ basename /var/run/cloudera-scm-agent/process/ccdeploy_etc_0/krb-conf/__cloudera_metadata__

+ DEST_FILE=/etc/__cloudera_metadata__

+ cp -a /var/run/cloudera-scm-agent/process/ccdeploy_etc_0/krb-conf/__cloudera_metadata__ /etc/__cloudera_metadata__.cm_tmp

+ chmod -R ugo+r /etc/__cloudera_metadata__.cm_tmp

+ mv -Tf /etc/__cloudera_metadata__.cm_tmp /etc/__cloudera_metadata__

+ for TOCOPY in '"$CONF_DIR/$DIRECTORY_NAME/"*'

++ basename /var/run/cloudera-scm-agent/process/ccdeploy_etc_0/krb-conf/krb5.conf

+ DEST_FILE=/etc/krb5.conf

+ cp -a /var/run/cloudera-scm-agent/process/ccdeploy_etc_0/krb-conf/krb5.conf /etc/krb5.conf.cm_tmp

+ chmod -R ugo+r /etc/krb5.conf.cm_tmp

+ mv -Tf /etc/krb5.conf.cm_tmp /etc/krb5.conf

+ exit 0

 

Thanks,

Bhavesh

Posts: 1,047
Topics: 1
Kudos: 263
Solutions: 131
Registered: ‎04-22-2014

Re: [cloudera 5.6] Reinitialize Kerberos with a new Realm: An old /etc/krb5.conf keeps being redeplo

@bhaveshvv109,

 

The log you provided is from the execution of "deploy-cc.sh" that only occurs when initiated from Cloudera Manager.

If the agent is executing that same process without instruction from CM, that indicates that there is some problem in the agent.

 

Please look at your supervisord.log in /var/log/cloudera-scm-agent. The cluster deploy is executed by the supervisor, so we should see when it runs.

The supervisor should only run it if that proces supervisor.conf still exists in /var/run/cloudera-scm-agent/supervisor/include.

My theory here is that for some reason the supervisor thinks it has a new process to execute so it is executing the "/var/run/cloudera-scm-agent/process/ccdeploy_etc_0" again and again.

 

When you restart your agent, do you use:

# service cloudera-scm-agent restart?

 

Perhaps grepping for "deploy-cc" in the agent log could help explain more too...

 

There could be something up here for real... I will test when I get a chance...

 

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.