Support Questions

Find answers, ask questions, and share your expertise

NiFi Hive Connection Pool Error.

Super Collaborator

Hi,

I am trying to connect to create a connection Hive using different formats DB Connection URLs. All of them are giving different errors.

8633-hivedb.png

first DB Connection URL and Error:

jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain;

error:

21:37:29 UTCERRORd3b62ee6-0157-1000-b66f-364970fcfa98 SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] Unable to execute HiveQL select query show tables due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain: GSS initiate failed). No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain: GSS initiate failed)

Second DB Connection URL and Error:

jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain;transportMode=http;httpPath=cliservice

Error

21:47:09 UTCERRORd3b62ee6-0157-1000-b66f-364970fcfa98 SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] Unable to execute HiveQL select query show tables due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL ' jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain;transportMode=http;httpPath=cliservice'. No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.hive.jdbc.HiveDriver' for connect URL ' jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain;transportMode=http;httpPath=cliservice'

Third DB Connection and Error

jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain;transportMode=http;httpPath=cliservice;auth=KERBEROS

Error 21:49:56 UTCERRORd3b62ee6-0157-1000-b66f-364970fcfa98 SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] failed to process session due to java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory: java.lang.NoClassDefFoundError: Could not initialize class org.apache.http.conn.ssl.SSLConnectionSocketFactory 21:49:56 UTCWARNINGd3b62ee6-0157-1000-b66f-364970fcfa98 SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98]

Processor Administratively Yielded for 1 sec due to processing failure

for all of these above i am commonly getting a warning

HiveConnectionPool[id=d3b68593-0157-1000-2894-db6013bf04f3] Configuration does not have security enabled, Keytab and Principal will be ignored

Thanks in advance,

Sai

15 REPLIES 15

Hi, looks like you are accessing Hive over HTTP. There was an issue fixed here: https://issues.apache.org/jira/browse/NIFI-2828 (will be in the NiFi 1.1.0 and HDF 2.0.1).

You can also try switching to a binary Hive endpoint for now.

Super Collaborator

What should I do to get the fix.??

Also how do I switch to binary end point.??

Super Collaborator

Can someone help me or point me to an example on how i can get connected to kerberos enabled Hive from NiFi.?

Super Collaborator

I am trying to connect to Hive from NiFi using Kerberos. Looks like there is no way I can do that until the newer version

NiFi 1.1.0? I cannot use any other modes as our Hadoop cluster is Kerberised.

Can I go back to previous version and use dbcp connection pool.? Will it work with Kerberos.? any other options I have (from NiFi).??

Explorer

@Saikrishna Tarapareddy

Just as @Andrew Grande mentioned, there is an issue with hiveQL processors + http.

Please connect to hive using the "binary" transport mode instead of the "http" transport mode.

To change this, In hive-site.xml or in Ambari, please update the property hive.server2.transport.mode to "binary".

Then all you need to change is the JDBC URL to "jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain;"

For the common error : "Configuration does not have security enabled, Keytab and Principal will be ignored" , you have not provided the path to the KDC server in the nifi.properties.

Please set the value of nifi.kerberos.krb5.file to the path of the krb5.conf which will be having the information of the KDC server. Then restart nifi. you should be able to connect to the said hive server.

Super Collaborator

Hi @mkalyanpur

I have verified that transport mode is set to "binary"

8696-hive2.png

my DB Connection URL is

jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain

8698-hive3.png

and these are all set in NiFi.config file..

nifi.kerberos.krb5.file=/etc/krb5.conf

nifi.kerberos.service.principal=nifi/Myserver@MyDomain.COM nifi.kerberos.service.keytab.location=/etc/security/keytabs/nifi.keytab

nifi.kerberos.spnego.principal=HTTP/nifi/Myserver@MyDomain.COM

nifi.kerberos.spnego.keytab.location=/etc/security/keytabs/spnego.service.keytab nifi.kerberos.spnego.authentication.expiration=12 hours

getting the error below..

19:47:43 UTC ERROR d3b68593-0157-1000-2894-db6013bf04f3
HiveConnectionPool[id=d3b68593-0157-1000-2894-db6013bf04f3] Error getting Hive connection
19:47:43 UTC ERROR d3b62ee6-0157-1000-b66f-364970fcfa98
SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] Unable to execute HiveQL select query show tables due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain.COM: GSS initiate failed). No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/hdf_moat;principal=hive/MyServer@MyDomain: GSS initiate failed)

Explorer

Hey

You are getting a gss initiate fail error. Which means nifi is unable to connect to hive using kerberos .

Please note that you have not provided the kerberos credentials. Please try after providing that

Super Collaborator

@mkalyanpur

i tried that too , but getting the same error. i am still getting the warning ,even though my nifi properties file has the settings about kerberos.

I am using HIVE user for kerberos credentials , i also tried with mine. both are resulting same error.

8716-hive3.png

14:26:12 UTC ERROR d3b62ee6-0157-1000-b66f-364970fcfa98

SelectHiveQL[id=d3b62ee6-0157-1000-b66f-364970fcfa98] Unable to execute HiveQL select query show tables due to org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/default;principal=hive/MyDomain.COM;: GSS initiate failed). No FlowFile to route to failure: org.apache.nifi.processor.exception.ProcessException: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Could not open client transport with JDBC Uri: jdbc:hive2://MyServer:10000/default;principal=hive/MyDomain.COM;: GSS initiate failed)
14:26:12 UTC WARNING d3b68593-0157-1000-2894-db6013bf04f3
HiveConnectionPool[id=d3b68593-0157-1000-2894-db6013bf04f3] Configuration does not have security enabled, Keytab and Principal will be ignored 

Explorer

@Saikrishna Tarapareddy

GSS initiate failed exception simply means that the provided credentials are incorrect.

You must have kerberised the cluster with a principal/keytab combination. Please provide that as the kerberos principa;/keytab. The credentials that you have provided are that of the hive service.

Super Collaborator

@mkalyanpur

our NiFi is running as NiFi service account with NiFi principal user , Hive is running as Hive user. I tried with both and getting the same error.

is that what you are asking.? sorry for the confusion.

New Contributor

@Saikrishna Tarapareddy Were you able to resolve this issue. I am having similar issue. Please let me know.

Could you also add core-site.xml in Hive Configuration Properties from your kerberosed cluster please?

New Contributor

I was having the same issue but after I added core-site.xml alongside my hive-site.xml file I was able to connect. Thanks!

/usr/hdp/current/hive/conf/hive-site.xml,/etc/hadoop/current/0/core-site.xml

New Contributor

@mkalyanpur do I have to copy the krb5.conf file from my Hive server to the NiFi server?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.