Created 10-24-2017 10:23 AM
Our team, as well as many known companies use Apache Airflow as Orchestrating system for ML tasks over Hadoop ecosystem. And it would be great to have it installed and managed via Ambari. Are there plans to release an Ambari-service-Airflow for such needs?
Thanks in advance!
Andrey
Created 10-24-2017 02:21 PM
Nothing that I've heard of internally just yet. Ambari, however, is extensible. You can always write an extension that allows you to add/manage/delete Airflow as a service if you wish. There are many examples online including one in the HCC Repo: https://community.hortonworks.com/repos/3912/r-service-ambari-service.html
Created 01-22-2018 09:08 AM
I have made Mpack for Ambari with airflow service for purposes of my company, but also posted it to GitHub. With this Mpack you're able to install and manage airflow in Ambari. Enjoy 🙂
Created 05-21-2018 06:06 AM
I have tried with mpack but services are getting start and going down in a second. After debugging the issue found there is not /usr/local/airflow/airflow_control.sh script. Could you please guide me from where I can get this.
Created 05-21-2018 11:27 AM
See link
https://github.com/miho120/ambari-airflow-mpack
In bottom page
#!/bin/bash
export AIRFLOW_HOME=/usr/local/airflow/airflow/ && source /usr/local/airflow/airflow_venv/airflow/bin/activate && /usr/local/airflow/airflow_venv/airflow/bin/airflow $1 --pid /usr/local/airflow/airflow/airflow-sys-$1.pid
Created 05-29-2018 12:56 PM
Hi @Ashnee Sharma
Issue has been fixed in commit 92dd09b. Now airflow_control.sh is created by Ambari during installation process.
Created 12-06-2018 09:48 AM
Hi @Mykola Mykhalov,
I'm getting error when I install Apache Airflow with Ambari.
ambari-server install-mpack --mpack=airflow-service-mpack.tar.gz
Using python /usr/bin/python Installing management pack ERROR: Download airflow-service-mpack.tar.gz with python lib [urllib2] failed with error: (<type 'exceptions.ValueError'>, ValueError('unknown url type: airflow-service-mpack.tar.gz',), <traceback object at 0x7f270d9c03b0>) Trying to download airflow-service-mpack.tar.gz to /var/lib/ambari-server/data/tmp/airflow-service-mpack.tar.gz with [curl] command. ERROR: Download file airflow-service-mpack.tar.gz with [curl] command failed with error: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: airflow-service-mpack.tar.gz ERROR: Unable to download file airflow-service-mpack.tar.gz! ERROR: unable to donwload file airflow-service-mpack.tar.gz! ERROR: Management pack could not be downloaded! ERROR: Exiting with exit code -1. REASON: Management pack could not be downloaded!
Please help here.
Thanks!
Created 12-06-2018 03:00 PM
Hi @Shesh Kumar,
That's strange that ambari-server tries to download it with curl, because it should take it from local system.
Try to:
wget https://github.com/miho120/ambari-airflow-mpack/raw/master/airflow-service-mpack.tar.gz -O /tmp/airflow-service-mpack.tar.gz
ambari-server install-mpack --mpack=/tmp/airflow-service-mpack.tar.gz
Let me know if it helps you.
Created 12-07-2018 10:00 AM
Hi @Mykola Mykhalov,
Thanks for the workaround. It worked for me. Was able to proceed.
Do we have an option to change the DB to MySQL ?
Is it compatible with Kerberos ?
Created 12-07-2018 10:20 AM
Hi @Shesh Kumar,
Good to hear that.
Sure you have, you just need to change sqlalchemy connection string in configuration (core section sql_alchemy_conn). E.g.:
sql_alchemy_conn = mysql://{USERNAME}:{PASSWORD}@{MYSQL_HOST}:3306/airflow
And celery_result_backend (if you use celery executor). E.g.:
celery_result_backend = db+mysql://{USERNAME}:{PASSWORD}@{MYSQL_HOST}:3306/airflow
But after you need to run Initdb in ambari "Service actions" - "initdb" (if you new db is empty).
As compatible as airflow is.
Created 12-13-2019 02:57 AM
Hi @mykola_mykhalov my airflow install errors out due to pymssql being discontinued? is there any work around?
DeprecationWarning: The pymssql project has been discontinued. To install the last working released version, use a version specifier like "pymssql<3.0". For details and alternatives see: https://github.com/pymssql/pymssql/issues/668 ---------------------------------------- ERROR: Failed building wheel for pymssql Running setup.py clean for pymssql Successfully built pendulum Failed to build pymssql ERROR: Could not build wheels for pymssql which use PEP 517 and cannot be installed directly
Created 12-24-2019 03:44 PM
I am also getting same error:
DeprecationWarning: The pymssql project has been discontinued. To install the last working released version, use a version specifier like "pymssql<3.0". For details and alternatives see: https://github.com/pymssql/pymssql/issues/668 ---------------------------------------- ERROR: Failed building wheel for pymssql Running setup.py clean for pymssql Successfully built pendulum Failed to build pymssql ERROR: Could not build wheels for pymssql which use PEP 517 and cannot be installed directly