Skip to main content
ashsskum
ashsskum
Gold ⭐️
December 17, 2016
Solved

Fixing HiveAccessControlException: Why INSERT OVERWRITE LOCAL DIRECTORY Fails in Apache NiFi PutHiveQL

I cannot get the following query run using PutHiveQL processor, it fails with the permission denied exception. I see that the processor emulates the same behavior as in Beeline.

But, I have been able to run the query from Hive CLI and writes to a file as expected. So, we know that Hive shell is an option, but can you let me know if there is any specific settings that causes this behavior in Beeline (Hive2) preventing to write to local filesystem?

insert overwrite local directory '/tmp' select current_date from dual

Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xxxxx] does not have [WRITE] privilege on [/tmp] (state=42000,code=40000)

eberezitsky
eberezitskyAnswer
Gold ⭐️
December 20, 2016

Regardless the reason you've decided to use PutHiveQL for select statement (I would use SelectHiveQL and then PutFile processor to store a result of select)...

But... back to the use case. Both PutHiveQL and Beeline are JDBC based clients. Using "insert overwrite LOCAL directory" doesn't seem to have a sense, as you never know what LOCAL directory (which node, etc) will be in the context. If still temp files are required to be created, you can go with:

1. PutHiveQL (insert overwrite directory, not local).

2. GetHDFS

... continue your processes/transformations with FlowFiles as usual.

Please note, HDFS files will be deleted after GetHDFS will finish.

View in thread

10 replies

mpandit
Gold ⭐️
mpandit
Gold ⭐️
December 17, 2016

It seems nifi user dont have permissions to write into /tmp dir. You have two options. 1. Change the permission on /tmp folder to allow everyone to write into it. 2. If you have configured Ranger, make sure in the resource based policy for HDFS, nifi user is allowed access to all paths or specific paths you want write to.

10406-screen-shot-2016-12-16-at-52349-pm.gif

Translate
ashsskum
ashsskumAuthor
Gold ⭐️
December 18, 2016

Ranger is specific to HDFS. I am referring to the issue with Hive writing to the local file system, see Hive statement in my summary above. And, /tmp directory is accessible by any user

Translate
sunile_manjee
Gold ⭐️
sunile_manjee
Gold ⭐️
December 17, 2016
@milind pandit

is right. if you do not have ranger enabled then add nifi user to the linux group which owns /tmp. This is ACL security. if you have ranger enabled then don't do this.

Translate
    ashsskum
    ashsskumAuthor
    Gold ⭐️
    December 18, 2016

    Please see my response above

    Translate
    eberezitsky
    eberezitskyAnswer
    Gold ⭐️
    December 20, 2016

    Regardless the reason you've decided to use PutHiveQL for select statement (I would use SelectHiveQL and then PutFile processor to store a result of select)...

    But... back to the use case. Both PutHiveQL and Beeline are JDBC based clients. Using "insert overwrite LOCAL directory" doesn't seem to have a sense, as you never know what LOCAL directory (which node, etc) will be in the context. If still temp files are required to be created, you can go with:

    1. PutHiveQL (insert overwrite directory, not local).

    2. GetHDFS

    ... continue your processes/transformations with FlowFiles as usual.

    Please note, HDFS files will be deleted after GetHDFS will finish.

    Translate
      ashsskum
      ashsskumAuthor
      Gold ⭐️
      December 20, 2016

      My use case requires to write in a delimited format. INSERT OVERWRITE LOCAL fits perfectly for this. I wish we had a way to custom delimit the content retrieved through SelectHiveQL processor, so I couldn't opt it.

      I agree it's a good option of writing to HDFS instead. I will attempt to modify the process, but I still wonder why it wouldn't work with write to Local filesystem, through NiFi.

      Translate
      eberezitsky
      eberezitsky
      Gold ⭐️
      December 20, 2016

      When using JDBC, local files will be created in local directories of a host running HiveServer2 you are connecting to (if you have more than one and connecting thru zookeeper - check all server hosts). So make sure you have writing permissions on those hosts, not on Nifi's one.

      Note, the story is totally different when you are using Hive CLI of Beeline embedded mode.

      Translate
      Terms & ConditionsAccessibility statement