NiFi 1.10 Hive processors and backticks

[#1]

When running Hive queries from NiFi 1.10 that contain backticks, I get the following error:

2019-12-23 15:17:00,191 WARN [Timer-Driven Process Thread-2] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=075b3a1e-7632-1647-68d9-338231b5921b] Failed to parse query: select 1 as `asdf` due to java.lang.NullPointerException:

I thought Hive queries allowed backticks to escape column names, so I'm not sure why NiFi can't parse this.  The actual query runs fine on the Hive server, and I get a valid flow file with the query results, but it still raises a red NiFi bulletin (which we would prefer not to have if there is not a real problem).

The Hive server running the query is the one that comes with HDP 2.5.

[#2 · Accepted solution]

Based on your input and other cases I found I logged this jira: https://issues.apache.org/jira/browse/NIFI-6969#

In short: It appears to be a problem, but not in the execution of the query but in the evaluation of it for lineage purposes.

If you have any additional information, consider adding it to the jira.

[#3]

I found that it's possible to fix this problem (as well as a different problem we were having with accessing Hive via a zookeeper connection string) by doing the following:

Use a custom NiFi Hive NAR file that has the Hortonworks versions of the hive, hadoop and zookeeper jars. This will get rid of the problem with backticks and the problem with the ZooKeeper connection string. To create the NAR file I just unzipped nifi-hive-nar-1.10.0.nar that comes with the Apache NiFi distro, then replaced all the the hive-*, hadoop-*, and zookeeper-* jars with the ones in http://repo.spring.io/hortonworks/org/apache/nifi/nifi-hive-nar/1.9.0.3.4.1.9-2/

You can just treat the NAR files as regular ZIP files.  There is no need to compile anything or use Maven.

We have been using this custom NAR for a few weeks and the NiFi Hive processors seem to be working without any problems.

[#4]

Hi,
Could you provide a link to the Hortonworks hive nar file. The one provided in your answer seems to be broken. I am having a similar issue with NIFI and zookeeper and would like to test your solution out. Thank you in advance.

[#5]

Looks like the files are available here: https://repo.hortonworks.com/content/repositories/releases/org/apache/nifi/nifi-hive-nar/