Unable to create Data Engineering Cluster in CDP Private Cloud Data Services with ECS

Question

Hey Everyone,We installed CDP Private Cloud Data Services i.e. ECS 1.4.1. After registering environment through management console, when we are trying to create CDE (Cloudera Data Engineering) cluster we are getting aws_key_id error although we have not deployed servers through AWS. Please find the logs below.2023-10-18 08:06:48: Create: Cluster: Data-engineering-1, user:   <admin@cdp.example>, event: Create, properties: [internal-actor-crn=crn:altus:iam:us-west-1:altus:user:__internal__actor__]2023-10-18 08:07:04: Log: Cluster provisioning initiated, cloud platform: STANDARD2023-10-18 08:07:04: Log: Started provisioning a managed cluster, provisionerID: liftie-9bylvhxh2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterProvisioningInitiated to ClusterProvisioningInProgress2023-10-18 08:07:04: LogDebug: Checking underlying cluster status2023-10-18 08:07:04: Log: Cluster provisioning completed2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterProvisioningInProgress to ClusterProvisioningCompleted2023-10-18 08:07:04: Log: Creating and syncing cluster access groups2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterProvisioningCompleted to ClusterAccessGroupCreationCompleted2023-10-18 08:07:04: Log: Creating Filesystem2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterAccessGroupCreationCompleted to FSProvisioningInProgress2023-10-18 08:07:04: Log: Filesystem creation completed successfully2023-10-18 08:07:04: LogDebug: Updating cluster status from FSProvisioningInProgress to FSProvisioningCompleted2023-10-18 08:07:04: LogDebug: Updating cluster status from FSProvisioningCompleted to FSMountTargetsCreated2023-10-18 08:07:04: Log: Provisioning database2023-10-18 08:07:04: LogDebug: Updating cluster status from FSMountTargetsCreated to DBProvisioningInProgress2023-10-18 08:07:04: Log: Database provisioning completed2023-10-18 08:07:04: LogDebug: Updating cluster status from DBProvisioningInProgress to DBProvisioningCompleted2023-10-18 08:07:04: Log: TLS certificate not required by config, skipping2023-10-18 08:07:04: LogDebug: Updating cluster status from DBProvisioningCompleted to ClusterTLSCertCreationCompleted2023-10-18 08:07:04: Log: Skipping service mesh install and configuration2023-10-18 08:07:04: LogDebug: Updating cluster status from ClusterTLSCertCreationCompleted to ClusterServiceMeshProvisioningCompleted2023-10-18 08:07:04: Log: Installing dex-base charts for CDE 1.17.12023-10-18 08:07:25: LogDebug: Found JWT signing keys with issuer=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/e9df8270-bc8a-4f64-91fe-c1ab998dc182, audience=https://de._NA_.apps.hostname-i0355.hostname.co.in, URL=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/oauth2/keys?accountId=e9df8270-bc8a-4f64-91fe-c1ab998dc1822023-10-18 08:07:25: Log: unable to find logging configurations from environment, error: unable to retrieve aws_key_id from the env service2023-10-18 08:07:25: LogError: Could not add shared cluster overrides, error: unable to retrieve aws_key_id from the env service2023-10-18 08:07:25: LogError: Chart installation failed, dex base overrides: unable to retrieve aws_key_id from the env service2023-10-18 08:07:25: Failure: {"StartState":"ClusterServiceMeshProvisioningCompleted","EndState":"ClusterChartInstallationFailed","Attempt":1}2023-10-18 08:08:04: Log: Installing dex-base charts for CDE 1.17.12023-10-18 08:08:19: LogDebug: Found JWT signing keys with issuer=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/e9df8270-bc8a-4f64-91fe-c1ab998dc182, audience=https://de._NA_.apps.hostname-i0355.hostname.co.in, URL=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/oauth2/keys?accountId=e9df8270-bc8a-4f64-91fe-c1ab998dc1822023-10-18 08:08:19: Log: unable to find logging configurations from environment, error: unable to retrieve aws_key_id from the env service2023-10-18 08:08:19: LogError: Could not add shared cluster overrides, error: unable to retrieve aws_key_id from the env service2023-10-18 08:08:19: LogError: Chart installation failed, dex base overrides: unable to retrieve aws_key_id from the env service2023-10-18 08:08:19: Failure: {"StartState":"ClusterServiceMeshProvisioningCompleted","EndState":"ClusterChartInstallationFailed","Attempt":2}2023-10-18 08:09:04: Log: Installing dex-base charts for CDE 1.17.12023-10-18 08:09:25: LogDebug: Found JWT signing keys with issuer=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/e9df8270-bc8a-4f64-91fe-c1ab998dc182, audience=https://de._NA_.apps.hostname-i0355.hostname.co.in, URL=https://console-cdp.apps.hostname-i0355.hostname.co.in/authenticate/oauth2/keys?accountId=e9df8270-bc8a-4f64-91fe-c1ab998dc1822023-10-18 08:09:25: Log: unable to find logging configurations from environment, error: unable to retrieve aws_key_id from the env service2023-10-18 08:09:25: LogError: Could not add shared cluster overrides, error: unable to retrieve aws_key_id from the env service2023-10-18 08:09:25: LogError: Chart installation failed, dex base overrides: unable to retrieve aws_key_id from the env service2023-10-18 08:09:25: Failure: {"StartState":"ClusterServiceMeshProvisioningCompleted","EndState":"ClusterChartInstallationFailed","Attempt":3}2023-10-18 08:09:25: LogDebug: Updating cluster status from ClusterServiceMeshProvisioningCompleted to ClusterChartInstallationFailed-----------------------------------------------------------------------------------------Cluster version details :-CM version - 7.8.1CDP Private Cloud  Base Version - 7.1.7Data services (ECS) - 1.4.1-b741 Any help or inputs would be appreciated.

smdas · Accepted Answer

Hello @Faisal_555 @Hai1

The above Error happens for 2 Reasoning majorly:

(I) Each Environment creates an Ozone Bucket (When the Environment is Created first) & CDE uses the concerned Ozone Bucket upon being Enabled within the Environment. 1 Top Reason is the Environment was Created when Ozone wasn't Installed or Ozone wasn't Healthy, upon which Ozone Bucket won't be Created. Hence, the Error [1] shows CDE isn't able to fetch the Ozone Info from Env (Wherein "aws_key_id" is reference to Ozone).

(II) To rule out the above, Ensure Ozone is Healthy & Create a New Environment, followed by Enabling the CDE Service. This would rule out any issues with Ozone & Environment Bucket.

(III) Next, Ensure the "hive" User has Ranger Permission to Create/List Ozone Bucket as documented in [2]. Even if Ozone is Healthy, the Bucket Creation is required & the same is facilitated via "hive" User & Ranger Permission may forbade the same.

If the above Action Plan doesn't resolve the Issue, Best to engage Cloudera Support for us to Collaborate further.

- Smarak

[1]

Log: unable to find logging configurations from environment, error: unable to retrieve aws_key_id from the env service

[2] https://docs.cloudera.com/cdp-private-cloud-data-services/1.5.3/installation/topics/cdppvc-installation-cdp-data-center.html

DianaTorres · Answer

@Faisal_555 Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our CDP experts @venkatsambath @aakulov who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.