Support Questions
Find answers, ask questions, and share your expertise

NiFi 1.10 Hive processors and backticks

Solved Go to solution

NiFi 1.10 Hive processors and backticks

When running Hive queries from NiFi 1.10 that contain backticks, I get the following error:

 

2019-12-23 15:17:00,191 WARN [Timer-Driven Process Thread-2] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=075b3a1e-7632-1647-68d9-338231b5921b] Failed to parse query: select 1 as `asdf` due to java.lang.NullPointerException:

 

I thought Hive queries allowed backticks to escape column names, so I'm not sure why NiFi can't parse this.  The actual query runs fine on the Hive server, and I get a valid flow file with the query results, but it still raises a red NiFi bulletin (which we would prefer not to have if there is not a real problem).

 

The Hive server running the query is the one that comes with HDP 2.5.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: NiFi 1.10 Hive processors and backticks

Expert Contributor

Based on your input and other cases I found I logged this jira: https://issues.apache.org/jira/browse/NIFI-6969#

 

In short: It appears to be a problem, but not in the execution of the query but in the evaluation of it for lineage purposes.

 

If you have any additional information, consider adding it to the jira.


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'. Also check out my techincal portfolio at https://portfolio.jaheruddin.nl

View solution in original post

4 REPLIES 4

Re: NiFi 1.10 Hive processors and backticks

Expert Contributor

Based on your input and other cases I found I logged this jira: https://issues.apache.org/jira/browse/NIFI-6969#

 

In short: It appears to be a problem, but not in the execution of the query but in the evaluation of it for lineage purposes.

 

If you have any additional information, consider adding it to the jira.


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'. Also check out my techincal portfolio at https://portfolio.jaheruddin.nl

View solution in original post

Re: NiFi 1.10 Hive processors and backticks

I found that it's possible to fix this problem (as well as a different problem we were having with accessing Hive via a zookeeper connection string) by doing the following:

 

Use a custom NiFi Hive NAR file that has the Hortonworks versions of the hive, hadoop and zookeeper jars. This will get rid of the problem with backticks and the problem with the ZooKeeper connection string. To create the NAR file I just unzipped nifi-hive-nar-1.10.0.nar that comes with the Apache NiFi distro, then replaced all the the hive-*, hadoop-*, and zookeeper-* jars with the ones in http://repo.spring.io/hortonworks/org/apache/nifi/nifi-hive-nar/1.9.0.3.4.1.9-2/

 

You can just treat the NAR files as regular ZIP files.  There is no need to compile anything or use Maven.

 

We have been using this custom NAR for a few weeks and the NiFi Hive processors seem to be working without any problems.

Re: NiFi 1.10 Hive processors and backticks

New Contributor

Hi,
Could you provide a link to the Hortonworks hive nar file. The one provided in your answer seems to be broken. I am having a similar issue with NIFI and zookeeper and would like to test your solution out. Thank you in advance.

Re: NiFi 1.10 Hive processors and backticks