Create table causes HiveServer2 to crash

[#1]

SYMPTOM: A create table in a cluster enforcing authorization using Ranger, fails to create the table and post that HiveServer2 process crashes. Create table fails as follows

0: jdbc:hive2://xxxx.hk.example.com> CREATE EXTERNAL TABLE TMP_HIVE2PHOENIX_E32E8 (CUSTOMER_ID STRING, ACCOUNT_ID STRING, ROLE_ID STRING, ROLE_NAME STRING, START_DATE STRING, END_DATE STRING, PRIORITY STRING, ACTIVE_ACCOUNT_ROLE STRING) 
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' 
LINES TERMINATED BY '\n' 
STORED AS TEXTFILE LOCATION '/tmp/example' 
TBLPROPERTIES ('serialization.null.format'=''); 
Error: org.apache.thrift.transport.TTransportException (state=08S01,code=0)

Following errors are observed in hiveserver2.log:

2016-11-15 11:42:06,721 WARN  [HiveServer2-Handler-Pool: Thread-32350]: thrift.ThriftCLIService (ThriftCLIService.java:ExecuteStatement(492)) - Error executing statement:
org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [1503524] does not have [READ] privilege on [hdfs://hadooprad/tmp/hive2phoenix_e32e8]
...
Caused by: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException: Permission denied: user [xxxx] does not have [READ] privilege on [hdfs://hadooprad/tmp/example]
        at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.checkPrivileges(RangerHiveAuthorizer.java:253)

Along with the above errors, hiveserver2.log also shows repetitive GC pauses and subsequently HiveServer2 service crashes:

2016-11-15 12:39:54,428 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@24197b13]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(192)) - Detected pause in JVM or host machine (eg GC): pause of approximately 24000ms 
GC pool 'PS MarkSweep' had collection(s): count=6 time=26445ms

ROOT CAUSE: HIVE-10022 / Hortonworks Internal BUG-42569/BUG-67204 To check for a permission (read or write) on a given path of query, Ranger checks for permissions on a given directory and all its children. However, if the directory does not exist, it will try to check the parent directory, or its parent directory, and so on. Eventually the table creation fails and at the same time as this operation uses toom uch memory and causes GC pauses.

In this case, Ranger checks for permission on /tmp/<databasename>, and since it does not exist it starts checking /tmp/ and its child directories, causing the GC Pauses and HiveServer2 service crash.

RESOLUTION: The fix is not part of the HDP releases currently. Talk to Hortonworks Technical Support and check if a hotfix is possible for the given version.

WORKAROUND: Ensure that the Storage Location specified in the create table statement does exist in the system.