Unable to create Data Engineering Cluster in CDP Private Cloud Data Services with ECS

[#1]

Hey Everyone,

We installed CDP Private Cloud Data Services i.e. ECS 1.4.1. After registering environment through management console, when we are trying to create CDE (Cloudera Data Engineering) cluster we are getting aws_key_id error although we have not deployed servers through AWS. Please find the logs below.

2023-10-18 08:06:48: Create: Cluster: Data-engineering-1, user:   <admin@cdp.example>, event: Create, properties: [internal-actor-crn=crn:altus:iam:us-west-1:altus:user:__internal__actor__]
2023-10-18 08:07:04: Log: Cluster provisioning initiated, cloud platform: STANDARD
2023-10-18 08:07:04: Log: Started provisioning a managed cluster, provisionerID: liftie-9bylvhxh
2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterProvisioningInitiated to ClusterProvisioningInProgress
2023-10-18 08:07:04: LogDebug: Checking underlying cluster status
2023-10-18 08:07:04: Log: Cluster provisioning completed
2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterProvisioningInProgress to ClusterProvisioningCompleted
2023-10-18 08:07:04: Log: Creating and syncing cluster access groups
2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterProvisioningCompleted to ClusterAccessGroupCreationCompleted
2023-10-18 08:07:04: Log: Creating Filesystem
2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterAccessGroupCreationCompleted to FSProvisioningInProgress
2023-10-18 08:07:04: Log: Filesystem creation completed successfully
2023-10-18 08:07:04: LogDebug: Updating cluster status from FSProvisioningInProgress to FSProvisioningCompleted
2023-10-18 08:07:04: LogDebug: Updating cluster status from FSProvisioningCompleted to FSMountTargetsCreated
2023-10-18 08:07:04: Log: Provisioning database
2023-10-18 08:07:04: LogDebug: Updating cluster status from FSMountTargetsCreated to DBProvisioningInProgress
2023-10-18 08:07:04: Log: Database provisioning completed
2023-10-18 08:07:04: LogDebug: Updating cluster status from DBProvisioningInProgress to DBProvisioningCompleted
2023-10-18 08:07:04: Log: TLS certificate not required by config, skipping
2023-10-18 08:07:04: LogDebug: Updating cluster status from DBProvisioningCompleted to ClusterTLSCertCreationCompleted
2023-10-18 08:07:04: Log: Skipping service mesh install and configuration
2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterTLSCertCreationCompleted to ClusterServiceMeshProvisioningCompleted
2023-10-18 08:07:04: Log: Installing dex-base charts for CDE 1.17.1
2023-10-18 08:07:25: LogDebug: Found JWT signing keys with issuer=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/e9df8270-bc8a-4f64-91fe-c1ab998d..., audience=https://de._NA_.apps.hostname-i0355.hostname.co.in, URL=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/oauth2/keys?accountId=e9df8270-b...
2023-10-18 08:07:25: Log: unable to find logging configurations from environment, error: unable to retrieve aws_key_id from the env service
2023-10-18 08:07:25: LogError: Could not add shared cluster overrides, error: unable to retrieve aws_key_id from the env service
2023-10-18 08:07:25: LogError: Chart installation failed, dex base overrides: unable to retrieve aws_key_id from the env service
2023-10-18 08:07:25: Failure: {"StartState":"ClusterServiceMeshProvisioningCompleted","EndState":"ClusterChartInstallationFailed","Attempt":1}
2023-10-18 08:08:04: Log: Installing dex-base charts for CDE 1.17.1
2023-10-18 08:08:19: LogDebug: Found JWT signing keys with issuer=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/e9df8270-bc8a-4f64-91fe-c1ab998d..., audience=https://de._NA_.apps.hostname-i0355.hostname.co.in, URL=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/oauth2/keys?accountId=e9df8270-b...
2023-10-18 08:08:19: Log: unable to find logging configurations from environment, error: unable to retrieve aws_key_id from the env service
2023-10-18 08:08:19: LogError: Could not add shared cluster overrides, error: unable to retrieve aws_key_id from the env service
2023-10-18 08:08:19: LogError: Chart installation failed, dex base overrides: unable to retrieve aws_key_id from the env service
2023-10-18 08:08:19: Failure: {"StartState":"ClusterServiceMeshProvisioningCompleted","EndState":"ClusterChartInstallationFailed","Attempt":2}
2023-10-18 08:09:04: Log: Installing dex-base charts for CDE 1.17.1
2023-10-18 08:09:25: LogDebug: Found JWT signing keys with issuer=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/e9df8270-bc8a-4f64-91fe-c1ab998d..., audience=https://de._NA_.apps.hostname-i0355.hostname.co.in, URL=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/oauth2/keys?accountId=e9df8270-b...
2023-10-18 08:09:25: Log: unable to find logging configurations from environment, error: unable to retrieve aws_key_id from the env service
2023-10-18 08:09:25: LogError: Could not add shared cluster overrides, error: unable to retrieve aws_key_id from the env service
2023-10-18 08:09:25: LogError: Chart installation failed, dex base overrides: unable to retrieve aws_key_id from the env service
2023-10-18 08:09:25: Failure: {"StartState":"ClusterServiceMeshProvisioningCompleted","EndState":"ClusterChartInstallationFailed","Attempt":3}
2023-10-18 08:09:25: LogDebug: Updating cluster status from ClusterServiceMeshProvisioningCompleted to ClusterChartInstallationFailed

-----------------------------------------------------------------------------------------

Cluster version details :-

CM version - 7.8.1

CDP Private Cloud  Base Version - 7.1.7

Data services (ECS) - 1.4.1-b741 

Any help or inputs would be appreciated.

[#2]

@Faisal_555 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our CDP experts @venkatsambath @aakulov  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3]

Hi @Faisal_555 , @DianaTorres  we've also encountered this error although the Ozone service in the Base cluster is healthy. We have seen all the containers' logs but haven't found any clear problems with Ozone service in the Base cluster. Please help us, we have stuck in this problem for one week by now. 

[#4 · Accepted solution]

Hello @Faisal_555 @Hai1 

The above Error happens for 2 Reasoning majorly:

(I) Each Environment creates an Ozone Bucket (When the Environment is Created first) & CDE uses the concerned Ozone Bucket upon being Enabled within the Environment. 1 Top Reason is the Environment was Created when Ozone wasn't Installed or Ozone wasn't Healthy, upon which Ozone Bucket won't be Created. Hence, the Error [1] shows CDE isn't able to fetch the Ozone Info from Env (Wherein "aws_key_id" is reference to Ozone). 

(II) To rule out the above, Ensure Ozone is Healthy & Create a New Environment, followed by Enabling the CDE Service. This would rule out any issues with Ozone & Environment Bucket.

(III) Next, Ensure the "hive" User has Ranger Permission to Create/List Ozone Bucket as documented in [2]. Even if Ozone is Healthy, the Bucket Creation is required & the same is facilitated via "hive" User & Ranger Permission may forbade the same.

If the above Action Plan doesn't resolve the Issue, Best to engage Cloudera Support for us to Collaborate further.

- Smarak

[1] 

Log: unable to find logging configurations from environment, error: unable to retrieve aws_key_id from the env service

 [2] https://docs.cloudera.com/cdp-private-cloud-data-services/1.5.3/installation/topics/cdppvc-installat...