Pods are Crashing in Namespace shared-services on Cloudera Data Services

[#1]

Hello all,

I am using Cloudera Datasservices on CDP Private Cloud. I am using using  the embedded container service (ECS).

If i check the kubernetes pods i see that the pods of the namespace shared-services are always crashing (Expect of openshift-idling-controller-manager)

kubectl get pods -n shared-services
NAME READY STATUS RESTARTS AGE
log-router-fznw2 0/2 Init:CrashLoopBackOff 17 46m
log-router-jnjrb 0/2 Init:CrashLoopBackOff 17 46m
log-router-tptbg 0/2 Init:CrashLoopBackOff 17 46m
openshift-idling-controller-manager-0 1/1 Running 0 45m

If i query the logs of the pod I get the following error:

kubectl logs -f -n shared-services log-router-jnjrb sdx-init-container

I have already tried to delete the pods but without success.

But the behaviour is unchanged:

Who can help?Thanks in advance.

Best regards

Uli

{"timestamp":"2022-10-07 07:35:52.120 +0000","level":"WARN","thread":"Thread-8","logger":"c.cloudera.cdp.config.ConfigTemplate","message":"[2022-10-07 07:35:52.120 +0000] WARN [Thread-8] - c.cloudera.cdp.config.ConfigTemplate: Failed to obtain initialization data due to {} com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND: NOT_FOUND 1c393538-72db-49e9-8ed0-2fd5df1927ed\n\tat com.cloudera.cdp.client.CdpParseResponseMiddleware.parseError(CdpParseResponseMiddleware.java:159)\n\tat com.cloudera.cdp.client.CdpParseResponseMiddleware.parse(CdpParseResponseMiddleware.java:127)\n\tat com.cloudera.cdp.client.CdpParseResponseMiddleware.invokeAPI(CdpParseResponseMiddleware.java:75)\n\tat com.cloudera.cdp.client.CdpRequestAuthMiddleware.invokeAPI(CdpRequestAuthMiddleware.java:50)\n\tat com.cloudera.cdp.client.CdpRequestHeadersMiddleware.invokeAPI(CdpRequestHeadersMiddleware.java:71)\n\tat com.cloudera.cdp.client.CdpClientRetryMiddleware.invokeAPI(CdpClientRetryMiddleware.java:55)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:224)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:200)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:148)\n\tat com.cloudera.cdp.environments.api.EnvironmentsClient.getConfigFiles(EnvironmentsClient.java:595)\n\tat com.cloudera.cdp.config.ConfigTemplate.getConfigFile(ConfigTemplate.java:1000)\n\tat com.cloudera.cdp.config.ConfigTemplate.getClientConfigsInfo(ConfigTemplate.java:952)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.lambda$2(ConfigTemplate.java:157)\n\tat com.cloudera.cdp.config.ConfigTemplate.init(ConfigTemplate.java:354)\n\tat com.cloudera.cdp.config.ConfigTemplate.exec(ConfigTemplate.java:223)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.run(ConfigTemplate.java:191)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n","stack_trace":"com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND: NOT_FOUND 1c393538-72db-49e9-8ed0-2fd5df1927ed\n\tat c.c.c.client.CdpParseResponseMiddleware.parseError(CdpParseResponseMiddleware.java:159)\n\tat c.c.c.client.CdpParseResponseMiddleware.parse(CdpParseResponseMiddleware.java:127)\n\tat c.c.c.client.CdpParseResponseMiddleware.invokeAPI(CdpParseResponseMiddleware.java:75)\n\tat c.c.cdp.client.CdpRequestAuthMiddleware.invokeAPI(CdpRequestAuthMiddleware.java:50)\n\tat c.c.c.client.CdpRequestHeadersMiddleware.invokeAPI(CdpRequestHeadersMiddleware.java:71)\n\tat c.c.cdp.client.CdpClientRetryMiddleware.invokeAPI(CdpClientRetryMiddleware.java:55)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:224)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:200)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:148)\n\tat c.c.c.e.api.EnvironmentsClient.getConfigFiles(EnvironmentsClient.java:595)\n\tat com.cloudera.cdp.config.ConfigTemplate.getConfigFile(ConfigTemplate.java:1000)\n\tat com.cloudera.cdp.config.ConfigTemplate.getClientConfigsInfo(ConfigTemplate.java:952)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.lambda$2(ConfigTemplate.java:157)\n\tat com.cloudera.cdp.config.ConfigTemplate.init(ConfigTemplate.java:354)\n\tat com.cloudera.cdp.config.ConfigTemplate.exec(ConfigTemplate.java:223)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.run(ConfigTemplate.java:191)\n\tat java.lang.Thread.run(Thread.java:829)\n"}

[#2 · Accepted solution]

Hello @utehrani 

Thanks for engaging Cloudera Community. Based on the Post, Pods for CML on ECS are failing, which isn't being resolved by Restarting the Pod as well. 

The Error [1] is generally observed for few Checks/Configs failing & frankly, the review of such an issue wouldn't be easy with Community.

Since your Team installed ECS & deploying CML, We believe it's quicker for your Team to engage Cloudera Support via a Support Case while sharing the Version (CDP Private Cloud Base Version, Private Cloud Data Service Version) along with the Pod Listing "kubectl get pods -A" & the Logs from the affected Pods.

Also, Whether your Team have deployed other Data Services like CDW (Cloudera Data Warehouse) & CDE (Cloudera Data Engineering) on the concerned ECS Setup yet. If No, We would like to confirm if their Deployment is working fine or facing similar concerns. 

Regards, Smarak

[1] Failed to obtain initialization data due to {} com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND

[#3]

@utehrani, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  

[#4]

Hello @utehrani 

We are marking the Post as Closed with the recommendation for your Team to engage Cloudera Support as the Troubleshooting of the issue would require sharing of Logs, which may be sensitive to be shared across Community along with Bundle/Config Files. 

Regards, Smarak