- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Airflow is failing to start in cloudera with issue Execute command Upgrade Airflow DB on service Airflow Could not create process: null
- Labels:
-
Apache Airflow
Created on 01-18-2024 08:41 AM - edited 01-18-2024 08:51 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello all,
We are having an issue while adding airflow service to the CDH 5 cluster.
Problem: Executing the command "Upgrade Airflow DB" on the Airflow service in a Cloudera cluster on GCP (CentOS 7, CDH 5.16.2, Airflow 1.10.1) results in an error: "Could not create process: null". The attempt to upgrade the Airflow database fails while adding airflow service to the CDH cluster.
Context:
- The same infrastructure and parcel configurations have been successfully used on 6-7 on-premises clusters.
- Logs are empty, and the scm-server log provides the same error message, indicating a failure to create the process.
Environment:
- Operating System: CentOS 7
- CDH Version: 5.16.2
- Airflow Version: 1.10.1
Steps Taken:
- Added the Airflow parcels and csd in the proper location with cloudera-scm owner.
- Added the Airflow service.
- Created Mysql DB for airflow.
- Added Mysql lib/client on the all hosts.
- Added Redis server to manage the airflow in cloudera.
- Checked logs for detailed error information, but logs are empty.
- Verified the successful deployment of Airflow on multiple on-premises clusters with the same configuration.
Issue Impact:
- Unable to add the Airflow service on the Cloudera cluster in GCP, affecting the stability and functionality of Airflow.
Request for Assistance:
- Seeking community insights and assistance in identifying the root cause of the issue.
- Open to recommendations, troubleshooting steps, or insights from the community to resolve the "Could not create process: null" error during the Airflow DB upgrade.
Additional Information:
- No recent changes in the configuration that could explain the failure.
- SCM-server logs are not providing additional information.
Environment Details:
- Cloudera Manager: Version 5.16.2
- CDH: Version 5.16.2
- Airflow: Version 1.10.1
- Operating System: CentOS 7
Your collaboration and expertise are highly appreciated in resolving this issue.
Created 05-06-2024 05:51 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @Mdismaik
Thanks for using Cloudera Community. This is an Old Post, yet Wish to convey that Cloudera shall assist with Airflow Issues (Install, Upgrade, Maintenance) dealing with Airflow shipped by Cloudera. Currently, CDE (Cloudera Data Engineering) allows Airflow to be deployed on Public Cloud & Private Cloud with End-To-End Support offered by Cloudera.
Your Post deals with reviewing an Issue with Installing Standalone Airflow on Cloudera Cluster, which isn't Supported per-se. Henceforth, our engagement would be Limited. We did a Sanity Check on the Error internally, yet no Conclusion arrived with few references indicating Possible DB concerns.
If your Team is interested in CDE Airflow, Let us know & we can get in touch with your Team.
- Smarak
Created 01-18-2024 02:00 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Mdismaik Welcome to the Cloudera Community!
To help you get the best possible solution, I have tagged our Airflow experts @SanthoshGowda @smdas who may be able to assist you further.
Please keep us updated on your post, and we hope you find a satisfactory solution to your query.
Regards,
Diana Torres,Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
Created 05-06-2024 05:51 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello @Mdismaik
Thanks for using Cloudera Community. This is an Old Post, yet Wish to convey that Cloudera shall assist with Airflow Issues (Install, Upgrade, Maintenance) dealing with Airflow shipped by Cloudera. Currently, CDE (Cloudera Data Engineering) allows Airflow to be deployed on Public Cloud & Private Cloud with End-To-End Support offered by Cloudera.
Your Post deals with reviewing an Issue with Installing Standalone Airflow on Cloudera Cluster, which isn't Supported per-se. Henceforth, our engagement would be Limited. We did a Sanity Check on the Error internally, yet no Conclusion arrived with few references indicating Possible DB concerns.
If your Team is interested in CDE Airflow, Let us know & we can get in touch with your Team.
- Smarak
