Airflow is failing to start in cloudera with issue Execute command Upgrade Airflow DB on service Airflow Could not create process: null

[#1]

Hello all,

We are having an issue while adding airflow service to the CDH 5 cluster.

Problem: Executing the command "Upgrade Airflow DB" on the Airflow service in a Cloudera cluster on GCP (CentOS 7, CDH 5.16.2, Airflow 1.10.1) results in an error: "Could not create process: null". The attempt to upgrade the Airflow database fails while adding airflow service to the CDH cluster.

Context:

• The same infrastructure and parcel configurations have been successfully used on 6-7 on-premises clusters.
• Logs are empty, and the scm-server log provides the same error message, indicating a failure to create the process.

Environment:

• Operating System: CentOS 7
• CDH Version: 5.16.2
• Airflow Version: 1.10.1

Steps Taken:

• Added the Airflow parcels and csd in the proper location with cloudera-scm owner.
• Added the Airflow service.
• Created Mysql DB for airflow.
• Added Mysql lib/client on the all hosts.
• Added Redis server to manage the airflow in cloudera.
• Checked logs for detailed error information, but logs are empty.
• Verified the successful deployment of Airflow on multiple on-premises clusters with the same configuration.

Issue Impact:

• Unable to add the Airflow service on the Cloudera cluster in GCP, affecting the stability and functionality of Airflow.

Request for Assistance:

• Seeking community insights and assistance in identifying the root cause of the issue.
• Open to recommendations, troubleshooting steps, or insights from the community to resolve the "Could not create process: null" error during the Airflow DB upgrade.

Additional Information:

• No recent changes in the configuration that could explain the failure.
• SCM-server logs are not providing additional information.

Environment Details:

• Cloudera Manager: Version 5.16.2
• CDH: Version 5.16.2
• Airflow: Version 1.10.1
• Operating System: CentOS 7

Your collaboration and expertise are highly appreciated in resolving this issue.

[#2]

@Mdismaik Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our Airflow experts @SanthoshGowda @smdas  who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

[#3 · Accepted solution]

Hello @Mdismaik 

Thanks for using Cloudera Community. This is an Old Post, yet Wish to convey that Cloudera shall assist with Airflow Issues (Install, Upgrade, Maintenance) dealing with Airflow shipped by Cloudera. Currently, CDE (Cloudera Data Engineering) allows Airflow to be deployed on Public Cloud & Private Cloud with End-To-End Support offered by Cloudera.

Your Post deals with reviewing an Issue with Installing Standalone Airflow on Cloudera Cluster, which isn't Supported per-se. Henceforth, our engagement would be Limited. We did a Sanity Check on the Error internally, yet no Conclusion arrived with few references indicating Possible DB concerns. 

If your Team is interested in CDE Airflow, Let us know & we can get in touch with your Team. 

- Smarak