Hello @Faisal_555 @Hai1  The above Error happens for 2 Reasoning majorly: (I) Each Environment creates an Ozone Bucket (When the Environment is Created first) & CDE uses the concerned Ozone Bucket upon being Enabled within the Environment. 1 Top Reason is the Environment was Created when Ozone wasn't Installed or Ozone wasn't Healthy, upon which Ozone Bucket won't be Created. Hence, the Error [1] shows CDE isn't able to fetch the Ozone Info from Env (Wherein "aws_key_id" is reference to Ozone).  (II) To rule out the above, Ensure Ozone is Healthy & Create a New Environment, followed by Enabling the CDE Service. This would rule out any issues with Ozone & Environment Bucket. (III) Next, Ensure the "hive" User has Ranger Permission to Create/List Ozone Bucket as documented in [2]. Even if Ozone is Healthy, the Bucket Creation is required & the same is facilitated via "hive" User & Ranger Permission may forbade the same. If the above Action Plan doesn't resolve the Issue, Best to engage Cloudera Support for us to Collaborate further. - Smarak [1]  Log: unable to find logging configurations from environment, error: unable to retrieve aws_key_id from the env service  [2] https://docs.cloudera.com/cdp-private-cloud-data-services/1.5.3/installation/topics/cdppvc-installation-cdp-data-center.html  ... View more

Hello @wert_1311  This is an Old Post, yet I am answering to ensure the Post can be used for future reference.  Anytime your Team observe the above Issue, Capture the CDE Diagnostics Bundle covering the Timeframe of Issue (Example Is 2 days In The Screenshot). Next, Restart the Airflow Scheduler Pod by Deleting the Airflow Scheduler Pod, upon which the Airflow Scheduler Pod would be Recreated implicitly. Next, Engage Cloudera Support with the CDE Diagnostics Bundle captured above. Additionally, CDE Airflow has been significantly Scaled-Test in recent releases of CDE & your Team should consider Upgrading CDE to Latest Version as soon as possible. - Smarak [1] https://docs.cloudera.com/data-engineering/cloud/troubleshooting/topics/cde-download-diagnostic-bundle.html  ... View more

Hello @Hae  AFAIK, Data Connections is a Public Cloud Concept & isn't available in Private Cloud yet. In Public Cloud, [1] shows the Steps to configure Data Connections, which allows you to access the HMS of the DataLake (Unified HMS Source For The Environment). In Private Cloud, You may use the [2] to use Spark on CML. The same has Example on using Spark-On-Yarn on Base Cluster as well as Spark-On-Kubernetes on CML. - Smarak [1] https://docs.cloudera.com/machine-learning/cloud/mlde/topics/ml-mlde-spark-data-connection.html  [2] https://docs.cloudera.com/machine-learning/1.5.2/spark/topics/ml-apache-spark-overview.html  ... View more

Hello @RamaClouder  Hope you have Synced with Cloudera Sales Team [0] & have no further ask. To Sum up of the Post, Cloudera offers Unified Governance within the Platform on Public, Private & Hybrid Cloud Setup. The Link [1] has relevant Doc/Videos Link for further Understanding.  If you have any further ask, Feel free to post in our Community & We shall get back to you accordingly.  - Smarak [0] https://www.cloudera.com/contact-sales.html [1] https://www.cloudera.com/products/cloudera-data-platform/sdx.html  ... View more

Hello @ipson  Thanks for using Cloudera Community.  To the Q, Kindly use "dfworkload" Commands [1]. The concerned Command has the Sub-Command to address the requirement of fetching Definition & Configurations. If any of the requirement isn't met via [1], Kindly Submit a Support Case for us to get the same included in the CLI with the Collaboration of DFX Engineering. - Smarak [1] https://cloudera.github.io/cdp-dev-docs/cli-docs/dfworkload/index.html  ... View more

Hello @Ging  Thanks for engaging Cloudera Community. Based on the Post, Your Team is seeing Resource Allocation Issues with Jobs remaining in Waiting State even if available resources & believe the Issue is linked with YuniKorn as the Issue is being observed after Upgrade to CDE v1.20.3.  To your Q, Please find the requested details below: (I) YuniKorn UI: YuniKorn UI is available via CDE UI > Administration > CDE Service "Service Details" > Resource Scheduler (II) For any assumed Issue with YuniKorn, Always Capture the YuniKorn StateDump [1] while the Issue is being Observed. This is Extremely Important as YuniKorn StateDump is Realtime & doesn't help, if the StateDump is captured while the Issue isn't being observed. (III) Collect the YuniKorn Scheduler & YuniKorn Admission Controller Pod Logs in YuniKorn Namespace while the Issue is happening. (IV) Attempt a Restart of YuniKorn Scheduler to confirm if the Issue persists after Capturing all above Info. If Yes i.e. Issue persists, Engage Cloudera Support with the StateDump, Pod Logs & Job Event Log, showing the Job being Stuck in Waiting. Hope the above answers your ask.  - Smarak [1] https://yunikorn.apache.org/docs/1.3.0/user_guide/troubleshooting/#obtain-full-state-dump  ... View more

Hello @Rekha35    Thanks for using Cloudera Community. First, I shall answer the Q from Cloudera's perspective. Cloudera Data Engineering offering (CDE) offers Airflow built-in without any manual intervention. CDE is available on Cloudera Public Cloud & Private Cloud offering. The integration with other Platform Services is Seamless & offers an immersive experience to End-Users looking to explore Airflow. Note that your Team can integrate with an External Airflow [2] as well.    Considering your ask was with checking out Airflow, You may try out [3], wherein the "Astronomer Apache Airflow Fundamentals Certification" offers the best starting point (Including Installing Locally). The Course is led by the foremost Airflow Instructor Marc Lamberti & helps anyone delve into Airflow seamlessly. Additionally, You asked about checking Airflow on Cloud owing to having issues with deploying the same locally. Each Cloud Provider (AWS, Azure, GCP) offers a Managed Service for Airflow.    Hope your Q has been addressed by us. Feel free to share any further ask.    Regards, Smarak     [1] https://docs.cloudera.com/data-engineering/cloud/orchestrate-workflows/topics/cde-airflow-dag-pipeline.html  [2] https://docs.cloudera.com/data-engineering/cloud/orchestrate-workflows/topics/cde-airflow-provider.html?  [3] https://www.astronomer.io/certification/  ... View more