NiFi user getting Tez exception

[#1]

Hi,

We are experiencing a strange situation , my flow has 1 PutHDFS,5 PutHiveQL processes. When i use my user keytab it all works as expected but it fails when i use NiFi user keytab. That too on one particular PutHiveQL processor where it uses the following syntax.

insert overwrite table db_test.tbl_test PARTITION(date)

SELECT col1,col2,col3.....,col100 FROM temp_table where date= 20170717 DISTRIBUTE BY col100

getting this error

PutHiveQL[id=b1b830f3-d843-13d4-af26-f3e1aa74942d] Failed to update Hive for StandardFlowFileRecord[uuid=0992aeb1-4a00-4b40-b012-bdb055917ed2,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1500476164378-589, container=default, section=589], offset=138, length=1463],offset=0,name=20170717.csv,size=1463] due to java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask; it is possible that retrying the operation will succeed, so routing to retry: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask

Other 4 PutHiveQLs use ALTER TABLE Commands which works fine with NiFi user , only the above SQL is failing. But it works with my user id. All users are setup same in Ranger. only issue may be my user is created in all nodes where nifi user may not be..will that be causing this issue or anything else.??

Regards,

Sai

[#2 · Accepted solution]

We fixed it , NiFi is missing permissions on the temp folders\files that it creates during INSERT..SELECT command.

[#3]

@Saikrishna Tarapareddy, can you please specify what folders and files needed the permissions? Are they on the HiveServer2 server or on HDFS?

[#4]

I found it. In my case it was /user/nifi on HDFS, which is the user that the flow connects with.

It moves a .jar file under the .hiveJars subdirectory on that directory in order to run commands.