Ambari service Airflow

[#1]

Our team, as well as many known companies use Apache Airflow as Orchestrating system for ML tasks over Hadoop ecosystem. And it would be great to have it installed and managed via Ambari. Are there plans to release an Ambari-service-Airflow for such needs?

Thanks in advance!

Andrey

[#2]

Hi @Andrey Emelyanenko

Nothing that I've heard of internally just yet. Ambari, however, is extensible. You can always write an extension that allows you to add/manage/delete Airflow as a service if you wish. There are many examples online including one in the HCC Repo: https://community.hortonworks.com/repos/3912/r-service-ambari-service.html

[#3]

Hi @Andrey Emelyanenko

I have made Mpack for Ambari with airflow service for purposes of my company, but also posted it to GitHub. With this Mpack you're able to install and manage airflow in Ambari. Enjoy 🙂

https://github.com/miho120/ambari-airflow-mpack

[#4]

@Mykola Mykhalov

I have tried with mpack but services are getting start and going down in a second. After debugging the issue found there is not /usr/local/airflow/airflow_control.sh script. Could you please guide me from where I can get this.

[#5]

See link

https://github.com/miho120/ambari-airflow-mpack

In bottom page

#!/bin/bash

export AIRFLOW_HOME=/usr/local/airflow/airflow/ && source /usr/local/airflow/airflow_venv/airflow/bin/activate && /usr/local/airflow/airflow_venv/airflow/bin/airflow $1 --pid /usr/local/airflow/airflow/airflow-sys-$1.pid

[#6]

Hi @Ashnee Sharma
Issue has been fixed in commit 92dd09b. Now airflow_control.sh is created by Ambari during installation process.

[#7]

Hi @Mykola Mykhalov,

I'm getting error when I install Apache Airflow with Ambari.

ambari-server install-mpack --mpack=airflow-service-mpack.tar.gz
Using python  /usr/bin/python
Installing management pack
ERROR: Download airflow-service-mpack.tar.gz with python lib [urllib2] failed with error: (<type 'exceptions.ValueError'>, ValueError('unknown url type: airflow-service-mpack.tar.gz',), <traceback object at 0x7f270d9c03b0>)
Trying to download airflow-service-mpack.tar.gz to /var/lib/ambari-server/data/tmp/airflow-service-mpack.tar.gz with [curl] command.
ERROR: Download file airflow-service-mpack.tar.gz with [curl] command failed with error:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: airflow-service-mpack.tar.gz
ERROR: Unable to download file airflow-service-mpack.tar.gz!
ERROR: unable to donwload file airflow-service-mpack.tar.gz!
ERROR: Management pack could not be downloaded!
ERROR: Exiting with exit code -1.
REASON: Management pack could not be downloaded!

Please help here.

Thanks!

[#8]

Hi @Shesh Kumar,

That's strange that ambari-server tries to download it with curl, because it should take it from local system.
Try to:

wget https://github.com/miho120/ambari-airflow-mpack/raw/master/airflow-service-mpack.tar.gz -O /tmp/airflow-service-mpack.tar.gz

ambari-server install-mpack --mpack=/tmp/airflow-service-mpack.tar.gz

Let me know if it helps you.

[#9]

Hi @Mykola Mykhalov,

Thanks for the workaround. It worked for me. Was able to proceed.
Do we have an option to change the DB to MySQL ?
Is it compatible with Kerberos ?

[#10]

Hi @Shesh Kumar,

Good to hear that.

Sure you have, you just need to change sqlalchemy connection string in configuration (core section sql_alchemy_conn). E.g.:

sql_alchemy_conn = mysql://{USERNAME}:{PASSWORD}@{MYSQL_HOST}:3306/airflow

And celery_result_backend (if you use celery executor). E.g.:

celery_result_backend = db+mysql://{USERNAME}:{PASSWORD}@{MYSQL_HOST}:3306/airflow

But after you need to run Initdb in ambari "Service actions" - "initdb" (if you new db is empty).

As compatible as airflow is.

[#11]

Hi @mykola_mykhalov  my airflow install errors out due to pymssql being discontinued? is there any work around?

  DeprecationWarning: The pymssql project has been discontinued.  To install the last working released version, use a
  version specifier like "pymssql<3.0".  For details and alternatives see:
  https://github.com/pymssql/pymssql/issues/668
  ----------------------------------------
  ERROR: Failed building wheel for pymssql
  Running setup.py clean for pymssql
Successfully built pendulum
Failed to build pymssql
ERROR: Could not build wheels for pymssql which use PEP 517 and cannot be installed directly

[#12]

@mykola_mykhalov 

I am also getting same error:

DeprecationWarning: The pymssql project has been discontinued.  To install the last working released version, use a
  version specifier like "pymssql<3.0".  For details and alternatives see:
  https://github.com/pymssql/pymssql/issues/668
  ----------------------------------------
  ERROR: Failed building wheel for pymssql
  Running setup.py clean for pymssql
Successfully built pendulum
Failed to build pymssql
ERROR: Could not build wheels for pymssql which use PEP 517 and cannot be installed directly