question Re: Issue running spark jobs with Airflow in Support Questions

Issue running spark jobs with Airflow

imule — Tue, 21 Apr 2026 06:49:13 GMT

Hi everyone,

So I've inherited a kerberized Cloudera cluster and I'm learning as I go. Right now I'm trying to get Airflow to work with our Spark jobs but without success. As I understand Airflow was installed by our OS team only after the cluster was configured by Cloudera. It runs on our edge node from where we run our jobs.

Basically I'm using bash operators for my test DAG with the following tasks:

Task 1:

Kinit the user that is running the script:
"echo 'password' | kinit user@domain"

Task 2:

Download some files from some location.

Task 3:
spark-submit /path/to/script.py

Task 1 and 2 work fine, but task 3 fails with the following:

py4j.protocol.Py4JJavaError: An error occurred while calling o32.csv.

: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

I am a bit confused by this as I am am authenticating the user as a first step. This exact workflow executes just fine when I run it manually in the CL.

Has anyone dealt with a similar issue? Any input would be appreciated as we need to transition to using Airflow.

Many thanks,

Mario

Re: Issue running spark jobs with Airflow

RangaReddy — Thu, 31 Aug 2023 08:20:43 GMT

Hi @imule

In step3, could you please pass --keytab <key_tab_path> --principal <principal_name> to the spark-submit command.

Note: In CDP, Airflow integration is not yet we are supported.

Re: Issue running spark jobs with Airflow

imule — Thu, 31 Aug 2023 14:16:55 GMT

Hi @RangaReddy ,

Is there a way to generate the file myself or do I need to contact our Active Directory administrators for that?

Thank you

Re: Issue running spark jobs with Airflow

RangaReddy — Mon, 04 Sep 2023 04:53:49 GMT

Hi @imule

You can follow the following steps to generate the keytab and if you don't have permission, please check with your admin team.

https://docs.cloudera.com/data-hub/cloud/access-clusters/topics/dh-retrieving-keytabs.html

Re: Issue running spark jobs with Airflow

VidyaSargur — Wed, 20 Sep 2023 14:02:05 GMT

@imule, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.

Re: Issue running spark jobs with Airflow

imule — Mon, 02 Oct 2023 13:29:01 GMT

Hi again @RangaReddy ,

I'm sorry for the huge delay in reply, unfortunately this triggered a lengthy discussion between us and the AD team.

In the end we managed to get our hands on a keytab file, and we confirmed it works fine by manually submitting the below command:

kinit -k -t /path/to/keytab/file.keytab username

Unfortunately when we attempt to pass this with a bash operator from an Airflow DAG we get the same error:

py4j.protocol.Py4JJavaError: An error occurred while calling o32.csv.

: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

Thank you,

Mario

Re: Issue running spark jobs with Airflow

RangaReddy — Tue, 03 Oct 2023 05:58:09 GMT

There are two solutions you can try.

1. Create one more shell operator and perform kinit and after that submit your spark

2. Pass the keytab and principal to the spark-submit