Skip to main content
utehrani
Bronze III
Oct 7, 2022 8:07 AM UTC
Solved

Pods are Crashing in Namespace shared-services on Cloudera Data Services

Hello all,

I am using Cloudera Datasservices on CDP Private Cloud. I am using using  the embedded container service (ECS).

If i check the kubernetes pods i see that the pods of the namespace shared-services are always crashing (Expect of openshift-idling-controller-manager)

 

kubectl get pods -n shared-services
NAME READY STATUS RESTARTS AGE
log-router-fznw2 0/2 Init:CrashLoopBackOff 17 46m
log-router-jnjrb 0/2 Init:CrashLoopBackOff 17 46m
log-router-tptbg 0/2 Init:CrashLoopBackOff 17 46m
openshift-idling-controller-manager-0 1/1 Running 0 45m

 

If i query the logs of the pod I get the following error:

kubectl logs -f -n shared-services log-router-jnjrb sdx-init-container

 

I have already tried to delete the pods but without success.

But the behaviour is unchanged:

Who can help?Thanks in advance.

Best regards

Uli

 

 

{"timestamp":"2022-10-07 07:35:52.120 +0000","level":"WARN","thread":"Thread-8","logger":"c.cloudera.cdp.config.ConfigTemplate","message":"[2022-10-07 07:35:52.120 +0000] WARN [Thread-8] - c.cloudera.cdp.config.ConfigTemplate: Failed to obtain initialization data due to {} com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND: NOT_FOUND 1c393538-72db-49e9-8ed0-2fd5df1927ed\n\tat com.cloudera.cdp.client.CdpParseResponseMiddleware.parseError(CdpParseResponseMiddleware.java:159)\n\tat com.cloudera.cdp.client.CdpParseResponseMiddleware.parse(CdpParseResponseMiddleware.java:127)\n\tat com.cloudera.cdp.client.CdpParseResponseMiddleware.invokeAPI(CdpParseResponseMiddleware.java:75)\n\tat com.cloudera.cdp.client.CdpRequestAuthMiddleware.invokeAPI(CdpRequestAuthMiddleware.java:50)\n\tat com.cloudera.cdp.client.CdpRequestHeadersMiddleware.invokeAPI(CdpRequestHeadersMiddleware.java:71)\n\tat com.cloudera.cdp.client.CdpClientRetryMiddleware.invokeAPI(CdpClientRetryMiddleware.java:55)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:224)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:200)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:148)\n\tat com.cloudera.cdp.environments.api.EnvironmentsClient.getConfigFiles(EnvironmentsClient.java:595)\n\tat com.cloudera.cdp.config.ConfigTemplate.getConfigFile(ConfigTemplate.java:1000)\n\tat com.cloudera.cdp.config.ConfigTemplate.getClientConfigsInfo(ConfigTemplate.java:952)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.lambda$2(ConfigTemplate.java:157)\n\tat com.cloudera.cdp.config.ConfigTemplate.init(ConfigTemplate.java:354)\n\tat com.cloudera.cdp.config.ConfigTemplate.exec(ConfigTemplate.java:223)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.run(ConfigTemplate.java:191)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n","stack_trace":"com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND: NOT_FOUND 1c393538-72db-49e9-8ed0-2fd5df1927ed\n\tat c.c.c.client.CdpParseResponseMiddleware.parseError(CdpParseResponseMiddleware.java:159)\n\tat c.c.c.client.CdpParseResponseMiddleware.parse(CdpParseResponseMiddleware.java:127)\n\tat c.c.c.client.CdpParseResponseMiddleware.invokeAPI(CdpParseResponseMiddleware.java:75)\n\tat c.c.cdp.client.CdpRequestAuthMiddleware.invokeAPI(CdpRequestAuthMiddleware.java:50)\n\tat c.c.c.client.CdpRequestHeadersMiddleware.invokeAPI(CdpRequestHeadersMiddleware.java:71)\n\tat c.c.cdp.client.CdpClientRetryMiddleware.invokeAPI(CdpClientRetryMiddleware.java:55)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:224)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:200)\n\tat com.cloudera.cdp.client.CdpClient.invokeAPI(CdpClient.java:148)\n\tat c.c.c.e.api.EnvironmentsClient.getConfigFiles(EnvironmentsClient.java:595)\n\tat com.cloudera.cdp.config.ConfigTemplate.getConfigFile(ConfigTemplate.java:1000)\n\tat com.cloudera.cdp.config.ConfigTemplate.getClientConfigsInfo(ConfigTemplate.java:952)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.lambda$2(ConfigTemplate.java:157)\n\tat com.cloudera.cdp.config.ConfigTemplate.init(ConfigTemplate.java:354)\n\tat com.cloudera.cdp.config.ConfigTemplate.exec(ConfigTemplate.java:223)\n\tat com.cloudera.cdp.config.ConfigTemplate$1.run(ConfigTemplate.java:191)\n\tat java.lang.Thread.run(Thread.java:829)\n"}

smdas
smdasAnswer
Gold ⭐️⭐️
Oct 12, 2022 10:16 AM UTC

Hello @utehrani 

 

Thanks for engaging Cloudera Community. Based on the Post, Pods for CML on ECS are failing, which isn't being resolved by Restarting the Pod as well. 

 

The Error [1] is generally observed for few Checks/Configs failing & frankly, the review of such an issue wouldn't be easy with Community.

 

Since your Team installed ECS & deploying CML, We believe it's quicker for your Team to engage Cloudera Support via a Support Case while sharing the Version (CDP Private Cloud Base Version, Private Cloud Data Service Version) along with the Pod Listing "kubectl get pods -A" & the Logs from the affected Pods.

 

Also, Whether your Team have deployed other Data Services like CDW (Cloudera Data Warehouse) & CDE (Cloudera Data Engineering) on the concerned ECS Setup yet. If No, We would like to confirm if their Deployment is working fine or facing similar concerns. 

 

Regards, Smarak

 

[1] Failed to obtain initialization data due to {} com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND

3 replies

smdas
Gold ⭐️⭐️
smdasAnswer
Gold ⭐️⭐️
Oct 12, 2022 10:16 AM UTC

Hello @utehrani 

 

Thanks for engaging Cloudera Community. Based on the Post, Pods for CML on ECS are failing, which isn't being resolved by Restarting the Pod as well. 

 

The Error [1] is generally observed for few Checks/Configs failing & frankly, the review of such an issue wouldn't be easy with Community.

 

Since your Team installed ECS & deploying CML, We believe it's quicker for your Team to engage Cloudera Support via a Support Case while sharing the Version (CDP Private Cloud Base Version, Private Cloud Data Service Version) along with the Pod Listing "kubectl get pods -A" & the Logs from the affected Pods.

 

Also, Whether your Team have deployed other Data Services like CDW (Cloudera Data Warehouse) & CDE (Cloudera Data Engineering) on the concerned ECS Setup yet. If No, We would like to confirm if their Deployment is working fine or facing similar concerns. 

 

Regards, Smarak

 

[1] Failed to obtain initialization data due to {} com.cloudera.cdp.CdpServiceException: com.cloudera.cdp.CdpServiceException: 404: NOT_FOUND

Translate
VidyaSargur
Community Manager
Community Manager
Oct 18, 2022 2:43 PM UTC

@utehrani, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.  

Translate
Regards, Vidya Sargur Madabhushi| Senior Community Manager | Was your question answered? Make sure to mark the answer as the Best Answer. If you find a reply useful, say thanks by clicking on the thumbs up button.
smdas
Gold ⭐️⭐️
Gold ⭐️⭐️
Oct 21, 2022 5:35 AM UTC

Hello @utehrani 

 

We are marking the Post as Closed with the recommendation for your Team to engage Cloudera Support as the Troubleshooting of the issue would require sharing of Logs, which may be sensitive to be shared across Community along with Bundle/Config Files. 

 

Regards, Smarak

Translate