Skip to main content
KFredrickson
KFredrickson
Gold ⭐️
December 23, 2019
Solved

NiFi 1.10 Hive processors and backticks

When running Hive queries from NiFi 1.10 that contain backticks, I get the following error:

 

2019-12-23 15:17:00,191 WARN [Timer-Driven Process Thread-2] o.a.nifi.processors.hive.SelectHiveQL SelectHiveQL[id=075b3a1e-7632-1647-68d9-338231b5921b] Failed to parse query: select 1 as `asdf` due to java.lang.NullPointerException:

 

I thought Hive queries allowed backticks to escape column names, so I'm not sure why NiFi can't parse this.  The actual query runs fine on the Hive server, and I get a valid flow file with the query results, but it still raises a red NiFi bulletin (which we would prefer not to have if there is not a real problem).

 

The Hive server running the query is the one that comes with HDP 2.5.

DennisJaheruddi
DennisJaheruddiAnswer
Platinum ⭐️
December 24, 2019

Based on your input and other cases I found I logged this jira: https://issues.apache.org/jira/browse/NIFI-6969#

 

In short: It appears to be a problem, but not in the execution of the query but in the evaluation of it for lineage purposes.

 

If you have any additional information, consider adding it to the jira.

- Dennis Jaheruddin If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.
View in thread

4 replies

DennisJaheruddi
Platinum ⭐️
DennisJaheruddiAnswer
Platinum ⭐️
December 24, 2019

Based on your input and other cases I found I logged this jira: https://issues.apache.org/jira/browse/NIFI-6969#

 

In short: It appears to be a problem, but not in the execution of the query but in the evaluation of it for lineage purposes.

 

If you have any additional information, consider adding it to the jira.

Translate
- Dennis Jaheruddin If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.
    KFredrickson
    KFredricksonAuthor
    Gold ⭐️
    January 2, 2020

    I found that it's possible to fix this problem (as well as a different problem we were having with accessing Hive via a zookeeper connection string) by doing the following:

     

    Use a custom NiFi Hive NAR file that has the Hortonworks versions of the hive, hadoop and zookeeper jars. This will get rid of the problem with backticks and the problem with the ZooKeeper connection string. To create the NAR file I just unzipped nifi-hive-nar-1.10.0.nar that comes with the Apache NiFi distro, then replaced all the the hive-*, hadoop-*, and zookeeper-* jars with the ones in http://repo.spring.io/hortonworks/org/apache/nifi/nifi-hive-nar/1.9.0.3.4.1.9-2/

     

    You can just treat the NAR files as regular ZIP files.  There is no need to compile anything or use Maven.

     

    We have been using this custom NAR for a few weeks and the NiFi Hive processors seem to be working without any problems.

    Translate
    mickt
    mickt
    Silver ⭐️⭐️
    February 12, 2021

    Hi,
    Could you provide a link to the Hortonworks hive nar file. The one provided in your answer seems to be broken. I am having a similar issue with NIFI and zookeeper and would like to test your solution out. Thank you in advance.

    Translate
    KFredrickson
    KFredricksonAuthor
    Gold ⭐️
    February 12, 2021
    Looks like the files are available here: https://repo.hortonworks.com/content/repositories/releases/org/apache/nifi/nifi-hive-nar/




    Translate
      Terms & ConditionsAccessibility statement