Support Questions

Find answers, ask questions, and share your expertise

Pods are Crashing in Namespace shared-services on Cloudera Data Services

New Contributor

Hello all,

I am using Cloudera Datasservices on CDP Private Cloud. I am using using  the embedded container service (ECS).

If i check the kubernetes pods i see that the pods of the namespace shared-services are always crashing (Expect of openshift-idling-controller-manager)

 

kubectl get pods -n shared-services
NAME READY STATUS RESTARTS AGE
log-router-fznw2 0/2 Init:CrashLoopBackOff 17 46m
log-router-jnjrb 0/2 Init:CrashLoopBackOff 17 46m
log-router-tptbg 0/2 Init:CrashLoopBackOff 17 46m
openshift-idling-controller-manager-0 1/1 Running 0 45m

 

If i query the logs of the pod I get the following error:

kubectl logs -f -n shared-services log-router-jnjrb sdx-init-container

 

I have already tried to delete the pods but without success.

But the behaviour is unchanged:

Who can help?Thanks in advance.

Best regards

Uli

 

 

{"timestamp":"2022-10-07 07:35:52.120 +0000","level":"WARN","thread":"Thread-8","logger":"c.cloudera.cdp.config.ConfigTemplate","message":"[2022-10-07 07:35:52.120 +0000] WARN [Thread-8] - c.cloudera.cdp.config.ConfigTemplate: Failed to obtain initialization data due to {} com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND: NOT_FOUND 1c393538-72db-49e9-8ed0-2fd5df1927ed\n\tat com.cloudera.cdp.client.CdpParseResponseMiddleware.parseError(CdpParseResponseMiddleware.java:159)\n\tat com.cloudera.cdp.client.CdpParseResponseMiddleware.parse(CdpParseResponseMiddleware.java:127)\n\tat com.cloudera.cdp.client.CdpParseResponseMiddleware.invokeAPI(CdpParseResponseMiddleware.java:75)\n\tat com.cloudera.cdp.client.CdpRequestAuthMiddleware.invokeAPI(CdpRequestAuthMiddleware.java:50)\n\tat com.cloudera.cdp.client.CdpRequestHeadersMiddleware.invokeAPI(CdpRequestHeadersMiddleware.java:71)\n\tat com.cloudera.cdp.client.CdpClientRetryMiddleware.invokeAPI(CdpClientRetryMiddleware.java:55)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:224)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:200)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:148)\n\tat com.cloudera.cdp.environments.api.EnvironmentsClient.getConfigFiles(EnvironmentsClient.java:595)\n\tat com.cloudera.cdp.config.ConfigTemplate.getConfigFile(ConfigTemplate.java:1000)\n\tat com.cloudera.cdp.config.ConfigTemplate.getClientConfigsInfo(ConfigTemplate.java:952)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.lambda$2(ConfigTemplate.java:157)\n\tat com.cloudera.cdp.config.ConfigTemplate.init(ConfigTemplate.java:354)\n\tat com.cloudera.cdp.config.ConfigTemplate.exec(ConfigTemplate.java:223)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.run(ConfigTemplate.java:191)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n","stack_trace":"com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND: NOT_FOUND 1c393538-72db-49e9-8ed0-2fd5df1927ed\n\tat c.c.c.client.CdpParseResponseMiddleware.parseError(CdpParseResponseMiddleware.java:159)\n\tat c.c.c.client.CdpParseResponseMiddleware.parse(CdpParseResponseMiddleware.java:127)\n\tat c.c.c.client.CdpParseResponseMiddleware.invokeAPI(CdpParseResponseMiddleware.java:75)\n\tat c.c.cdp.client.CdpRequestAuthMiddleware.invokeAPI(CdpRequestAuthMiddleware.java:50)\n\tat c.c.c.client.CdpRequestHeadersMiddleware.invokeAPI(CdpRequestHeadersMiddleware.java:71)\n\tat c.c.cdp.client.CdpClientRetryMiddleware.invokeAPI(CdpClientRetryMiddleware.java:55)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:224)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:200)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:148)\n\tat c.c.c.e.api.EnvironmentsClient.getConfigFiles(EnvironmentsClient.java:595)\n\tat com.cloudera.cdp.config.ConfigTemplate.getConfigFile(ConfigTemplate.java:1000)\n\tat com.cloudera.cdp.config.ConfigTemplate.getClientConfigsInfo(ConfigTemplate.java:952)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.lambda$2(ConfigTemplate.java:157)\n\tat com.cloudera.cdp.config.ConfigTemplate.init(ConfigTemplate.java:354)\n\tat com.cloudera.cdp.config.ConfigTemplate.exec(ConfigTemplate.java:223)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.run(ConfigTemplate.java:191)\n\tat java.lang.Thread.run(Thread.java:829)\n"}

1 ACCEPTED SOLUTION

Super Collaborator

Hello @utehrani 

 

Thanks for engaging Cloudera Community. Based on the Post, Pods for CML on ECS are failing, which isn't being resolved by Restarting the Pod as well. 

 

The Error [1] is generally observed for few Checks/Configs failing & frankly, the review of such an issue wouldn't be easy with Community.

 

Since your Team installed ECS & deploying CML, We believe it's quicker for your Team to engage Cloudera Support via a Support Case while sharing the Version (CDP Private Cloud Base Version, Private Cloud Data Service Version) along with the Pod Listing "kubectl get pods -A" & the Logs from the affected Pods.

 

Also, Whether your Team have deployed other Data Services like CDW (Cloudera Data Warehouse) & CDE (Cloudera Data Engineering) on the concerned ECS Setup yet. If No, We would like to confirm if their Deployment is working fine or facing similar concerns. 

 

Regards, Smarak

 

[1] Failed to obtain initialization data due to {} com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND

View solution in original post

3 REPLIES 3

Super Collaborator

Hello @utehrani 

 

Thanks for engaging Cloudera Community. Based on the Post, Pods for CML on ECS are failing, which isn't being resolved by Restarting the Pod as well. 

 

The Error [1] is generally observed for few Checks/Configs failing & frankly, the review of such an issue wouldn't be easy with Community.

 

Since your Team installed ECS & deploying CML, We believe it's quicker for your Team to engage Cloudera Support via a Support Case while sharing the Version (CDP Private Cloud Base Version, Private Cloud Data Service Version) along with the Pod Listing "kubectl get pods -A" & the Logs from the affected Pods.

 

Also, Whether your Team have deployed other Data Services like CDW (Cloudera Data Warehouse) & CDE (Cloudera Data Engineering) on the concerned ECS Setup yet. If No, We would like to confirm if their Deployment is working fine or facing similar concerns. 

 

Regards, Smarak

 

[1] Failed to obtain initialization data due to {} com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND

Community Manager

@utehrani, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

Super Collaborator

Hello @utehrani 

 

We are marking the Post as Closed with the recommendation for your Team to engage Cloudera Support as the Troubleshooting of the issue would require sharing of Logs, which may be sensitive to be shared across Community along with Bundle/Config Files. 

 

Regards, Smarak

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.