Client configuration redeployment failing in most of the services

[#1]

Hi,

I have restarted the cluster and services like hive, yarn, Oozie etc. are not getting started because of following error, we are using CDP 7.1.7

Role failed to start due to error java.lang.RuntimeException: com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs.

Can anyone please support here as picture is not cleat what to check here.

2023-11-28 08:14:06,415 ERROR Staleness-Detector-1:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Failed to check staleness for service client configs: DbService{id=1546335860, name=hive}
com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs
at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFiles(AbstractClientConfigHandler.java:151)
at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfig(AbstractClientConfigHandler.java:163)
at com.cloudera.cmf.service.AbstractClientConfigHandler.createClientConfig(AbstractClientConfigHandler.java:64)
at com.cloudera.cmf.service.AbstractClientConfigHandler.createClientConfig(AbstractClientConfigHandler.java:57)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck.compareClientConfigs(ProcessStalenessDetector.java:665)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck.access$1100(ProcessStalenessDetector.java:344)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$3.apply(ProcessStalenessDetector.java:614)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$3.apply(ProcessStalenessDetector.java:605)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$1$1.run(ProcessStalenessDetector.java:557)
at com.cloudera.server.common.MovingStats$1.get(MovingStats.java:32)
at com.cloudera.server.common.MovingStats$1.get(MovingStats.java:29)
at com.cloudera.server.common.MovingStats.measure(MovingStats.java:41)
at com.cloudera.server.common.MovingStats.measure(MovingStats.java:29)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$1.measure(ProcessStalenessDetector.java:553)

at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$1.apply(ProcessStalenessDetector.java:546)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$1.apply(ProcessStalenessDetector.java:542)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$4.run(ProcessStalenessDetector.java:630)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs
at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:1182)
at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:1131)
at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFilesHelper(AbstractClientConfigHandler.java:158)
at com.cloudera.cmf.service.AbstractClientConfigHandler.access$000(AbstractClientConfigHandler.java:28)
at com.cloudera.cmf.service.AbstractClientConfigHandler$1.call(AbstractClientConfigHandler.java:147)
at com.cloudera.cmf.service.AbstractClientConfigHandler$1.call(AbstractClientConfigHandler.java:144)
at com.cloudera.server.web.cmf.AuthScopeContext.privilegedMode(AuthScopeContext.java:173)
at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFiles(AbstractClientConfigHandler.java:144)
... 23 more
Caused by: com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs
at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFiles(AbstractClientConfigHandler.java:151)
at com.cloudera.cmf.service.components.ConfigHelper.getClientConfigs(ConfigHelper.java:173)
at com.cloudera.cmf.service.AbstractRoleHandler.generateClientConfigsForDependencies(AbstractRoleHandler.java:1258)
at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:1160)
... 30 more

 

[#2]

The error is generic and doesnt provide much information, you may want to check the following:
* You are using kerberos ( was anything change on the KDC account ) ?
* If KDC account was changed, reimport the KDC account and generate missing credentials
CM UI -> Administration -> security ->Kerberos Credentials and try to "Import KDC Account Manager Credentials"

* Check on the "agent service logs" of the failed service ( in this case seems hive ).

[#3]

We are not using kerberos here, when i am downloading client configuration i am facing follwoing issue for hive, hive on tez, spark and oozie. and for other services its working fine 

[#4]

You may want to check the logs , this may be caused for example

• If you don't have enough space in a partition filesystem
• If the tmp folder is now allowed to write ( not able to create a file )

The specific error of the file and location may appear in your CM logs / agent logs /service logs of the failing service ( per last  screenshot / Hive )

Hope it helps.