Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari service Airflow

Ambari service Airflow

New Contributor

Our team, as well as many known companies use Apache Airflow as Orchestrating system for ML tasks over Hadoop ecosystem. And it would be great to have it installed and managed via Ambari. Are there plans to release an Ambari-service-Airflow for such needs?

Thanks in advance!

Andrey

10 REPLIES 10
Highlighted

Re: Ambari service Airflow

Guru

Hi @Andrey Emelyanenko

Nothing that I've heard of internally just yet. Ambari, however, is extensible. You can always write an extension that allows you to add/manage/delete Airflow as a service if you wish. There are many examples online including one in the HCC Repo: https://community.hortonworks.com/repos/3912/r-service-ambari-service.html

Re: Ambari service Airflow

New Contributor

Hi @Andrey Emelyanenko

I have made Mpack for Ambari with airflow service for purposes of my company, but also posted it to GitHub. With this Mpack you're able to install and manage airflow in Ambari. Enjoy :)

https://github.com/miho120/ambari-airflow-mpack

Re: Ambari service Airflow

@Mykola Mykhalov

I have tried with mpack but services are getting start and going down in a second. After debugging the issue found there is not /usr/local/airflow/airflow_control.sh script. Could you please guide me from where I can get this.

Re: Ambari service Airflow

Expert Contributor

See link

https://github.com/miho120/ambari-airflow-mpack

In bottom page

#!/bin/bash

export AIRFLOW_HOME=/usr/local/airflow/airflow/ && source /usr/local/airflow/airflow_venv/airflow/bin/activate && /usr/local/airflow/airflow_venv/airflow/bin/airflow $1 --pid /usr/local/airflow/airflow/airflow-sys-$1.pid

Re: Ambari service Airflow

New Contributor

Hi @Ashnee Sharma
Issue has been fixed in commit 92dd09b. Now airflow_control.sh is created by Ambari during installation process.

Re: Ambari service Airflow

New Contributor

Hi @Mykola Mykhalov,

I'm getting error when I install Apache Airflow with Ambari.

ambari-server install-mpack --mpack=airflow-service-mpack.tar.gz
Using python /usr/bin/python Installing management pack ERROR: Download airflow-service-mpack.tar.gz with python lib [urllib2] failed with error: (<type 'exceptions.ValueError'>, ValueError('unknown url type: airflow-service-mpack.tar.gz',), <traceback object at 0x7f270d9c03b0>) Trying to download airflow-service-mpack.tar.gz to /var/lib/ambari-server/data/tmp/airflow-service-mpack.tar.gz with [curl] command. ERROR: Download file airflow-service-mpack.tar.gz with [curl] command failed with error: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: airflow-service-mpack.tar.gz ERROR: Unable to download file airflow-service-mpack.tar.gz! ERROR: unable to donwload file airflow-service-mpack.tar.gz! ERROR: Management pack could not be downloaded! ERROR: Exiting with exit code -1. REASON: Management pack could not be downloaded!

Please help here.

Thanks!

Re: Ambari service Airflow

New Contributor

Hi @Shesh Kumar,

That's strange that ambari-server tries to download it with curl, because it should take it from local system.
Try to:

wget https://github.com/miho120/ambari-airflow-mpack/raw/master/airflow-service-mpack.tar.gz -O /tmp/airflow-service-mpack.tar.gz
ambari-server install-mpack --mpack=/tmp/airflow-service-mpack.tar.gz

Let me know if it helps you.

Re: Ambari service Airflow

New Contributor

Hi @Mykola Mykhalov,

Thanks for the workaround. It worked for me. Was able to proceed.
Do we have an option to change the DB to MySQL ?
Is it compatible with Kerberos ?

Re: Ambari service Airflow

New Contributor

Hi @Shesh Kumar,

Good to hear that.

Sure you have, you just need to change sqlalchemy connection string in configuration (core section sql_alchemy_conn). E.g.:

sql_alchemy_conn = mysql://{USERNAME}:{PASSWORD}@{MYSQL_HOST}:3306/airflow

And celery_result_backend (if you use celery executor). E.g.:

celery_result_backend = db+mysql://{USERNAME}:{PASSWORD}@{MYSQL_HOST}:3306/airflow

But after you need to run Initdb in ambari "Service actions" - "initdb" (if you new db is empty).

As compatible as airflow is.

Don't have an account?
Coming from Hortonworks? Activate your account here