NiFi Error: failed to process session due to java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient

[#1]

Hi,

I am trying to connect Hive from Apache NiFi 1.3.0

but its giving me error

SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] failed to process session due to java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient: java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient 

SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] org.apache.nifi.processors.hive.SelectHiveQL$Lambda$182/1006763062@50dd1ed1 failed to process due to java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient; rolling back session: java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient 

Its clearly indicating that its not getting HttpClient.jar. Now i have added httpclient-4.5.2.jar in NiFi lib folder and tried to run NiFi again but NiFi failed to start. If i add any httpclient.jar in Nifi lib folder, nifi is failing to run.

Its giving me this error this time in NiFi-app.log

2017-10-06 05:00:00,835 ERROR [Timer-Driven Process Thread-8] org.apache.hive.jdbc.HiveConnection Error 
opening session org.apache.thrift.TApplicationException: Required field 'client_protocol' is unset! 

I think its a error related to version mismatch. Now i tried with httpclient-4.3.3.jar and httpclient-4.2.3.jar, Now for httpclient-4.3.3.jar its giving the same client protocol is unset error. And For httpclient-4.2.3.jar its giving me this error

2017-10-06 06:00:19,820 ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.standard.GetHTTP could not be instantiated
java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor: Provider org.apache.nifi.processors.standard.GetHTTP could not be instantiated 

Now My NiFi Settings:

In nifi.properties:

nifi.kerberos.krb5.file=/path/krb5.conf

In NiFi Lib folder:

I have added couple of libs

hive-jdbc-1.1.0-cdh5.7.0.jar

hive-service-1.1.0-cdh5.7.0.jar

libthrift-0.9.2.jar

Without this 3 libs nifi was giving other types of errors.

In HiveConnectionPool Setting

Database Connection Url: jdbc:hive2://MyServer:10000/;principal=hive/MyServer@MyDomain

Hive Conf Resources: /path/core-site.xml,/path/hive-site.xml

DB User:

Password:

Max Wait Time: 500 millis

Max Total Connections: 8

Validation query:

Kerberos principle: hive/MyServer@MyDomain

Kerberos keytab: /path/keytabs/hive.service.keytab

for all of these above i am commonly getting a warning

SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] Processor Administratively Yielded for 1 sec due to processing failure 

and an error

HiveConnectionPool[id=f0a678e0-015e-1000-1235-94651d8210e9] Kerberos Authentication for Hive failed: org.apache.nifi.util.hive.AuthenticationFailedException: Kerberos Authentication for Hive failed 

After reading a similar thread https://community.hortonworks.com/questions/62014/nifi-hive-connection-pool-error.html

I have added this property to Hive-site.xml . And tried to connect to hive using the "binary" transport mode.

<property>
<name>hive.server2.transport.mode</name>
<value>binary</value>
</property> 

But It still giving me same error

SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] SelectHiveQL[id=f0a5404d-015e-1000-9cda-00f1242476aa] failed to process session due to java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient: java.lang.NoClassDefFoundError: org/apache/http/client/HttpClient 

What should i do now? Any suggestions or Solutions

Thanks in advance,

Pratik

[#2 · Accepted solution]

Hi @Pratik Ghatak
This looks like what happens when you try to connect NiFi to an old fork of Hive like the one in CDH, I have an article about it here: https://community.hortonworks.com/articles/93771/connecting-nifi-to-cdh-hive.html

There's an excellent comment from @Rudolf Schimmel on a workaround at the bottom of that post if you are in the same situation, but give it a read and let me know if you need help with an alternative solution.

[#3]

Thanks for the Help. Its working.

[#4]

Check out: https://community.hortonworks.com/articles/93771/connecting-nifi-to-cdh-hive.html

This is due to Apache Hive 1.1 and Apache NiFi using Apache Hive 1.2.

[#5]

https://community.hortonworks.com/questions/64463/selecthiveql-processor-hive-connection-pool-config...

[#6]

Its working using DBCPConnectionPool.

[#7]

Hi @Pratik Ghatak @Rudolf Schimmel @Timothy Spann

I am getting the similar error while connecting to Hive using DBCPConnectionPool in ExecuteSQL processor in NiFi 1.7.1
Hive version : Hive 1.1.0-cdh5.12.0

My DBCP Connection Pool:
DB Connection URL: jdbc:hive2://MyServer:10000/default;AuthMech=GSSAPI;KrbRealm=HADOOP.TEST.COMPANY.COM;KrbHostFQDN=_HOST;KrbServiceName=hive

DB Driver Location: /opt/flow/nifi/lib/hive-jdbc-1.1.0-cdh5.12.0.jar, /opt/flow/nifi/lib/hive-service-1.1.0-cdh5.12.0.jar, /opt/flow/nifi/lib/libthrift-0.9.3.jar

DB Driver Class: org.apache.hive.jdbc.Hive

Driver Database User : No value set

Password : No value set

Further,

I have added below property in hive-site.xml

<property>

<name>hive.server2.transport.mode</name>

<value>binary</value>

<property>

If possible, could you please share your working DBCPConnection Pool settings.

I am able to connect to Hive in Python using pyhs2 and impala.dbapi modules providing

pyhs2.connect(host='myserver',port=10000,authMechanism='KERBEROS')
conn=connect(host='myserver', port=10000, kerberos_service_name='hive', auth_mechanism='GSSAPI')

Kindly advise. Thanks so much.