Hive NullPointerException from Atlas with Basic Query

[#1]

We have a third party product that tries to execute a simple bit of SQL to verify that the user can execute MR/Tez jobs against a source. It attempts to run:

SELECT COUNT(1)

Which works under a normal Hive installation. However, in the HDP 2.4 sandbox it results in an error. Logs indicate:

Query ID = hive_20160607173731_61e36107-5139-477b-a52a-fe20b0ee04b6Total jobs = 1Launching Job 1 out of 1Status: Running (Executing on YARN cluster with App id application_1465317865651_0003)Map 1: -/-Reducer 2: 0/1Map 1: 0/1Reducer 2: 0/1Map 1: 0(+1)/1Reducer 2: 0/1Map 1: 1/1Reducer 2: 0/1Map 1: 1/1Reducer 2: 0(+1)/1Map 1: 1/1Reducer 2: 1/1OKFAILED: Hive Internal Error: java.lang.NullPointerException(null)java.lang.NullPointerExceptionat org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerDatabase(HiveMetaStoreBridge.java:109)at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerTable(HiveMetaStoreBridge.java:270)at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:321)at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:214)at org.apache.atlas.hive.hook.HiveHook.run(HiveHook.java:172)at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1585)at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1254)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1113)at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:154)at org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)at org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:415)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)at java.util.concurrent.FutureTask.run(FutureTask.java:262)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:745)

Assuming we can't change the source query, can we get a fix?

[#2]

Can you disable Atlas and see if the query works?

[#3]

Thanks for the response. I tried disabling Atlas (and only Atlas) in Ambari as admin. Afterward, I tried executing any Hive query through both Ambari or Beeline. In both cases I get an error:

Error: Error while processing statement: FAILED: Hive Internal Error: com.sun.jersey.api.client.ClientHandlerException(java.io.IOException: java.net.ConnectException: Connection refused) (state=08S01,code=12)

It seems Hive and Atlas are tightly coupled. Working against any sort of "dummy database" causes an error in the Atlas bridge which propagates up to Hive. Example:

8: jdbc:hive2://localhost:10000> select count(1);
INFO  : Tez session hasn't been created yet. Opening session
INFO  : Dag name: select count(1)(Stage-1)
INFO  : 
INFO  : Status: Running (Executing on YARN cluster with App id application_1465411333149_0001)
INFO  : Map 1: -/-Reducer 2: 0/1
INFO  : Map 1: 0/1Reducer 2: 0/1
INFO  : Map 1: 0(+1)/1Reducer 2: 0/1
INFO  : Map 1: 1/1Reducer 2: 0(+1)/1
INFO  : Map 1: 1/1Reducer 2: 1/1
Error: Error while processing statement: FAILED: Hive Internal Error: java.lang.NullPointerException(null) (state=08S01,code=12)
8: jdbc:hive2://localhost:10000> select 1;
Error: Error while processing statement: FAILED: Hive Internal Error: java.lang.NullPointerException(null) (state=08S01,code=12)

[#4]

Can you post what is "hive.exec.post.hooks" in Hive configs set to? If it has the Atlas plugin in that list then remove that from the list and restart the HiveServer2.

[#5]

Thanks for the tip. It was set to this:

<property>
      <name>hive.exec.post.hooks</name>
      <value>org.apache.hadoop.hive.ql.hooks.ATSHook, org.apache.atlas.hive.hook.HiveHook</value>
    </property>

Removing the Atlas hook (org.apache.atlas.hive.hook.HiveHook) allowed the query to run unimpeded.

[#6 · Accepted solution]

@William Bowen is disabling Atlas an acceptable workaround?

To summarize the workaround, you disabled Atlas, modified hive config property "hive.exec.post.hooks" to get rid of "org.apache.atlas.hive.hook.HiveHook" from its value and then bounced the HiveServer2.