Client configuration redeployment failing in most of the services

CK_c — Tue, 28 Nov 2023 05:26:30 GMT

Hi,

I have restarted the cluster and services like hive, yarn, Oozie etc. are not getting started because of following error, we are using CDP 7.1.7

Role failed to start due to error java.lang.RuntimeException: com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs.

Can anyone please support here as picture is not cleat what to check here.

2023-11-28 08:14:06,415 ERROR Staleness-Detector-1:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Failed to check staleness for service client configs: DbService{id=1546335860, name=hive}
com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs
at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFiles(AbstractClientConfigHandler.java:151)
at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfig(AbstractClientConfigHandler.java:163)
at com.cloudera.cmf.service.AbstractClientConfigHandler.createClientConfig(AbstractClientConfigHandler.java:64)
at com.cloudera.cmf.service.AbstractClientConfigHandler.createClientConfig(AbstractClientConfigHandler.java:57)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck.compareClientConfigs(ProcessStalenessDetector.java:665)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck.access$1100(ProcessStalenessDetector.java:344)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$3.apply(ProcessStalenessDetector.java:614)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$3.apply(ProcessStalenessDetector.java:605)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$1$1.run(ProcessStalenessDetector.java:557)
at com.cloudera.server.common.MovingStats$1.get(MovingStats.java:32)
at com.cloudera.server.common.MovingStats$1.get(MovingStats.java:29)
at com.cloudera.server.common.MovingStats.measure(MovingStats.java:41)
at com.cloudera.server.common.MovingStats.measure(MovingStats.java:29)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$1.measure(ProcessStalenessDetector.java:553)

at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$1.apply(ProcessStalenessDetector.java:546)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$1.apply(ProcessStalenessDetector.java:542)
at com.cloudera.cmf.service.config.components.ProcessStalenessDetector$ProcessStalenessCheck$4.run(ProcessStalenessDetector.java:630)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.RuntimeException: com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs
at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:1182)
at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:1131)
at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFilesHelper(AbstractClientConfigHandler.java:158)
at com.cloudera.cmf.service.AbstractClientConfigHandler.access$000(AbstractClientConfigHandler.java:28)
at com.cloudera.cmf.service.AbstractClientConfigHandler$1.call(AbstractClientConfigHandler.java:147)
at com.cloudera.cmf.service.AbstractClientConfigHandler$1.call(AbstractClientConfigHandler.java:144)
at com.cloudera.server.web.cmf.AuthScopeContext.privilegedMode(AuthScopeContext.java:173)
at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFiles(AbstractClientConfigHandler.java:144)
... 23 more
Caused by: com.cloudera.cmf.service.config.ConfigGenException: Failed to generate client configs
at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFiles(AbstractClientConfigHandler.java:151)
at com.cloudera.cmf.service.components.ConfigHelper.getClientConfigs(ConfigHelper.java:173)
at com.cloudera.cmf.service.AbstractRoleHandler.generateClientConfigsForDependencies(AbstractRoleHandler.java:1258)
at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:1160)
... 30 more

Re: Client configuration redeployment failing in most of the services

FelipeG — Tue, 28 Nov 2023 16:05:35 GMT

The error is generic and doesnt provide much information, you may want to check the following:
* You are using kerberos ( was anything change on the KDC account ) ?
* If KDC account was changed, reimport the KDC account and generate missing credentials
CM UI -> Administration -> security ->Kerberos Credentials and try to "Import KDC Account Manager Credentials"

* Check on the "agent service logs" of the failed service ( in this case seems hive ).

Re: Client configuration redeployment failing in most of the services

CK_c — Wed, 29 Nov 2023 05:59:51 GMT

We are not using kerberos here, when i am downloading client configuration i am facing follwoing issue for hive, hive on tez, spark and oozie. and for other services its working fine

Re: Client configuration redeployment failing in most of the services

FelipeG — Sat, 23 Dec 2023 19:55:51 GMT

You may want to check the logs , this may be caused for example

If you don't have enough space in a partition filesystem
If the tmp folder is now allowed to write ( not able to create a file )

The specific error of the file and location may appear in your CM logs / agent logs /service logs of the failing service ( per last screenshot / Hive )

Hope it helps.

question Re: Client configuration redeployment failing in most of the services in Support Questions

Client configuration redeployment failing in most of the services

Re: Client configuration redeployment failing in most of the services

Re: Client configuration redeployment failing in most of the services

Re: Client configuration redeployment failing in most of the services