Hello @Knowledgeknow  Thanks for using Cloudera Community. This is an Old Post, yet Wish to convey that Cloudera shall assist with Airflow Issues (Install, Upgrade, Maintenance) dealing with Airflow shipped by Cloudera. Currently, CDE (Cloudera Data Engineering) allows Airflow to be deployed on Public Cloud & Private Cloud with End-To-End Support offered by Cloudera. You haven't shared whether your Q deals with CDE Airflow or Standalone Airflow. If your Post deals with reviewing an Issue with Installing Standalone Airflow which isn't Supported per-se. Henceforth, our engagement would be Limited. For CDE Airflow, the AuthN for Airflow UI is managed implicitly via Single-SignOn (Once your Team is Authenticated to Cloudera Management Console) & doesn't require any manual intervention. For CLI, CDE offers Token Based & Key Based AuthN.  If your Team is interested in CDE Airflow, Let us know & we can get in touch with your Team. - Smarak   ... View more

Hello @mandychen  Thanks for using Cloudera Community. This is an Old Post, yet Wish to convey that Cloudera shall assist with Airflow Issues (Install, Upgrade, Maintenance) dealing with Airflow shipped by Cloudera. Currently, CDE (Cloudera Data Engineering) allows Airflow to be deployed on Public Cloud & Private Cloud with End-To-End Support offered by Cloudera. Your Post deals with reviewing an Issue with Installing Standalone Airflow, which isn't Supported per-se. Henceforth, our engagement would be Limited. However, the Issue is likely linked with JDBC Parsing as discussed in [1]. If your Team is interested in CDE Airflow, Let us know & we can get in touch with your Team.  - Smarak [1] https://github.com/apache/airflow/issues/33442  ... View more

Hello @Mdismaik  Thanks for using Cloudera Community. This is an Old Post, yet Wish to convey that Cloudera shall assist with Airflow Issues (Install, Upgrade, Maintenance) dealing with Airflow shipped by Cloudera. Currently, CDE (Cloudera Data Engineering) allows Airflow to be deployed on Public Cloud & Private Cloud with End-To-End Support offered by Cloudera. Your Post deals with reviewing an Issue with Installing Standalone Airflow on Cloudera Cluster, which isn't Supported per-se. Henceforth, our engagement would be Limited. We did a Sanity Check on the Error internally, yet no Conclusion arrived with few references indicating Possible DB concerns.  If your Team is interested in CDE Airflow, Let us know & we can get in touch with your Team.  - Smarak ... View more

Hello @Rafe  Thanks for engaging Cloudera Community. To your Q, Cloudera recommends Non-Cloud-VMs to be used for CDP Private Cloud Data Services yet your Team can deploy ECS on any Hardware as long as the Hardware/Software Requirement are met as described in our Doc. - Smarak ... View more

Hello @wert_1311  Thanks for using Cloudera Community. May I know if your Team have used the "--arg" Flags of CDE Job Run Command. The same would meet the requirement. You may run "cde job run --help" to fetch all available Flags.  - Smarak ... View more

Hello @Faisal_555 @Hai1  The above Error happens for 2 Reasoning majorly: (I) Each Environment creates an Ozone Bucket (When the Environment is Created first) & CDE uses the concerned Ozone Bucket upon being Enabled within the Environment. 1 Top Reason is the Environment was Created when Ozone wasn't Installed or Ozone wasn't Healthy, upon which Ozone Bucket won't be Created. Hence, the Error [1] shows CDE isn't able to fetch the Ozone Info from Env (Wherein "aws_key_id" is reference to Ozone).  (II) To rule out the above, Ensure Ozone is Healthy & Create a New Environment, followed by Enabling the CDE Service. This would rule out any issues with Ozone & Environment Bucket. (III) Next, Ensure the "hive" User has Ranger Permission to Create/List Ozone Bucket as documented in [2]. Even if Ozone is Healthy, the Bucket Creation is required & the same is facilitated via "hive" User & Ranger Permission may forbade the same. If the above Action Plan doesn't resolve the Issue, Best to engage Cloudera Support for us to Collaborate further. - Smarak [1]  Log: unable to find logging configurations from environment, error: unable to retrieve aws_key_id from the env service  [2] https://docs.cloudera.com/cdp-private-cloud-data-services/1.5.3/installation/topics/cdppvc-installation-cdp-data-center.html  ... View more

Hello @wert_1311  This is an Old Post, yet I am answering to ensure the Post can be used for future reference.  Anytime your Team observe the above Issue, Capture the CDE Diagnostics Bundle covering the Timeframe of Issue (Example Is 2 days In The Screenshot). Next, Restart the Airflow Scheduler Pod by Deleting the Airflow Scheduler Pod, upon which the Airflow Scheduler Pod would be Recreated implicitly. Next, Engage Cloudera Support with the CDE Diagnostics Bundle captured above. Additionally, CDE Airflow has been significantly Scaled-Test in recent releases of CDE & your Team should consider Upgrading CDE to Latest Version as soon as possible. - Smarak [1] https://docs.cloudera.com/data-engineering/cloud/troubleshooting/topics/cde-download-diagnostic-bundle.html  ... View more

Hello @Hae  AFAIK, Data Connections is a Public Cloud Concept & isn't available in Private Cloud yet. In Public Cloud, [1] shows the Steps to configure Data Connections, which allows you to access the HMS of the DataLake (Unified HMS Source For The Environment). In Private Cloud, You may use the [2] to use Spark on CML. The same has Example on using Spark-On-Yarn on Base Cluster as well as Spark-On-Kubernetes on CML. - Smarak [1] https://docs.cloudera.com/machine-learning/cloud/mlde/topics/ml-mlde-spark-data-connection.html  [2] https://docs.cloudera.com/machine-learning/1.5.2/spark/topics/ml-apache-spark-overview.html  ... View more