question Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati in Support Questions

SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

mike_bronson7 — Mon, 02 Dec 2019 09:30:27 GMT

we are runs spark application on hadoop cluster ( HDP version - 2.6.5 from hortonworks )

from the logs we can see the following Diagnostics

User: airflow
Application Type: SPARK
User class threw exception: org.apache.hadoop.security.AccessControlException: Permission denied. user=airflow is not the owner of inode=alapati

not clearly what we need to search in `HDFS` in order to find why we get Permission denied

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

Shelton — Mon, 02 Dec 2019 07:56:29 GMT

@mike_bronson7

That's a classic permissions issue "airflow" is trying to write to that directory but has no permissions as it's owned by alapati user the inode=alapati.

The easiest solution is to grant the permissions as the hdfs user

$ hdfs dfs -chown airflow:{$airflow_group}

Most components like Spark, Hive, sqoop need to access HDFS

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

mike_bronson7 — Mon, 02 Dec 2019 08:10:15 GMT

how to found the airflow_group ?

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

mike_bronson7 — Mon, 02 Dec 2019 08:17:25 GMT

Dear Shelton

do you mean :

grep airflow /etc/passwd
airflow:x:1016:1016::/home/airflow:/sbin/nologin
# id 1016
uid=1016(airflow) gid=1016(airflow) groups=1016(airflow),1005(hdfs)

so we need to perform :

$ hdfs dfs -chown airflow:1016

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

Shelton — Mon, 02 Dec 2019 08:45:34 GMT

@mike_bronson7

The gid is numeric value just to indicate the id but the valid groups for airflow are [airflow and hdfs]

$ hdfs dfs -chown airflow:hdfs

Should do the magic, please revert

Cheers

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

mike_bronson7 — Mon, 02 Dec 2019 09:01:17 GMT

we got the following , because missing PATH

$ hdfs dfs -chown airflow:hdfs

-chown: Not enough arguments: expected 2 but got 1
Usage: hadoop fs [generic options]

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

mike_bronson7 — Mon, 02 Dec 2019 11:43:50 GMT

ok , we meed to find the path's for the changing

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

Shelton — Mon, 02 Dec 2019 09:42:31 GMT

@mike_bronson7
That's correct you need to give the path of the directory 🙂 ie usually in hdfs

$ hdfs dfs -chown airflow:hdfs /path/in/hdfs/where/you/failed/to/write

As you didn't include the path I assumed you'd do that with the -chown command

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

mike_bronson7 — Mon, 02 Dec 2019 10:12:22 GMT

when we do

hdfs dfs -ls -R / | grep "airflow " | awk '{print $1" "$2" "$3" "$4" "}'

we get:

drwxrwx--- - airflow hadoop
drwxrwx--- - airflow hadoop
drwxrwx--- - airflow hadoop
-rw-r----- 3 airflow hadoop
-rw-r----- 3 airflow hadoop
-rw-r----- 3 airflow hadoop
drwxrwx--- - airflow hadoop
-rw-r----- 3 airflow hadoop
-rw-r----- 3 airflow hadoop
-rw-r----- 3 airflow hadoop

do you means to change every hadoop group to hdfs ?

or in simple words - how to know the HDFS path , that we need to change?

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

Shelton — Mon, 02 Dec 2019 10:25:14 GMT

@mike_bronson7

The hadoop group encapsulates all the users including hdfs

You do run a

# cat /etc/group

You should see someing like like

hadoop:x:1007:yarn-ats,hive,storm,infra-solr,zookeeper,oozie,atlas,ams,ranger,tez,zeppelin,kms,accumulo,livy,druid,spark,ambari-qa,kafka,hdfs,s qoop,yarn,mapred,hbase,knox

So running the -chown should only target the directory in the Diagnostics logs NEVER run the -chown command on / which is the root directory !!

Can you share your log please

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

mike_bronson7 — Mon, 02 Dec 2019 10:49:51 GMT

yes we get the following:

cat /etc/group | grep -i hadoop
hadoop:x:1006:hive,livy,zookeeper,spark,ams,kafka,yarn,hcat,mapred

cat /etc/group | grep -i airflow
hdfs:x:1005:hdfs,hive,airflow
airflow:x:1016:

cat /etc/group | grep -i hdfs
hdfs:x:1005:hdfs,hive,airflow

let me know if you need additional info?

Re: SPARK Application + HDFS + User Airflow is not the owner of inode=alapati

Shelton — Mon, 02 Dec 2019 22:06:09 GMT

@mike_bronson7

You can change the ownership of the HDFS directory to airflow:hadoop please do run the -chown command on / ??? It should something like /users/airflow/xxx

Please let me know