Solved
Airflow is failing to start in cloudera with issue Execute command Upgrade Airflow DB on service Airflow Could not create process: null
Hello all,
We are having an issue while adding airflow service to the CDH 5 cluster.
Problem: Executing the command "Upgrade Airflow DB" on the Airflow service in a Cloudera cluster on GCP (CentOS 7, CDH 5.16.2, Airflow 1.10.1) results in an error: "Could not create process: null". The attempt to upgrade the Airflow database fails while adding airflow service to the CDH cluster.
Context:
- The same infrastructure and parcel configurations have been successfully used on 6-7 on-premises clusters.
- Logs are empty, and the scm-server log provides the same error message, indicating a failure to create the process.
Environment:
- Operating System: CentOS 7
- CDH Version: 5.16.2
- Airflow Version: 1.10.1
Steps Taken:
- Added the Airflow parcels and csd in the proper location with cloudera-scm owner.
- Added the Airflow service.
- Created Mysql DB for airflow.
- Added Mysql lib/client on the all hosts.
- Added Redis server to manage the airflow in cloudera.
- Checked logs for detailed error information, but logs are empty.
- Verified the successful deployment of Airflow on multiple on-premises clusters with the same configuration.
Issue Impact:
- Unable to add the Airflow service on the Cloudera cluster in GCP, affecting the stability and functionality of Airflow.
Request for Assistance:
- Seeking community insights and assistance in identifying the root cause of the issue.
- Open to recommendations, troubleshooting steps, or insights from the community to resolve the "Could not create process: null" error during the Airflow DB upgrade.
Additional Information:
- No recent changes in the configuration that could explain the failure.
- SCM-server logs are not providing additional information.
Environment Details:
- Cloudera Manager: Version 5.16.2
- CDH: Version 5.16.2
- Airflow: Version 1.10.1
- Operating System: CentOS 7
Your collaboration and expertise are highly appreciated in resolving this issue.
