Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

[NIFI] : Tuto Stream data into HIVE like a Boss using NiFi HiveStreaming

Highlighted

[NIFI] : Tuto Stream data into HIVE like a Boss using NiFi HiveStreaming

Explorer

Hi all,

I'm trying this tuto

https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html

But my mapreduce is kerberized so i'm using in my puthiveStreaming

principal : hive/hosts@DOMAIN

keytab : /home/user/hive.service.keytab

i'm running with this errors:

2017-06-26 10:45:48,246 INFO [Timer-Driven Process Thread-6] hive.metastore Trying to connect to metastore with URI thrift://master003:9083
2017-06-26 10:45:48,253 INFO [Timer-Driven Process Thread-6] hive.metastore Connected to metastore.
2017-06-26 10:45:48,280 WARN [Timer-Driven Process Thread-6] o.a.h.h.m.RetryingMetaStoreClient MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.transport.TTransportException: null
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
        at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
        at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
        at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863)
        at sun.reflect.GeneratedMethodAccessor1374.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
        at com.sun.proxy.$Proxy370.lock(Unknown Source)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.access$700(HiveEndPoint.java:461)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl$2.run(HiveEndPoint.java:555)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl$2.run(HiveEndPoint.java:552)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:551)
        at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:261)
        at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:73)
        at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:46)
        at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:964)
        at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:875)
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$null$40(PutHiveStreaming.java:676)
        at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$44(PutHiveStreaming.java:673)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2136)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2106)
        at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:627)
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$36(PutHiveStreaming.java:551)
        at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
        at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
        at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:551)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

        2017-06-26 10:45:53,280 INFO [Timer-Driven Process Thread-6] hive.metastore Trying to connect to metastore with URI thrift://master003:9083
2017-06-26 10:45:53,287 INFO [Timer-Driven Process Thread-6] hive.metastore Connected to metastore.
2017-06-26 10:45:53,312 WARN [Timer-Driven Process Thread-6] o.a.h.h.m.RetryingMetaStoreClient MetaStoreClient lost connection. Attempting to reconnect.
org.apache.thrift.transport.TTransportException: null
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
        at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
        at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
        at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863)
        at sun.reflect.GeneratedMethodAccessor1374.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
        at com.sun.proxy.$Proxy370.lock(Unknown Source)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.access$700(HiveEndPoint.java:461)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl$2.run(HiveEndPoint.java:555)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl$2.run(HiveEndPoint.java:552)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:551)
        at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:261)
        at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:73)
        at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:46)
        at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:964)
        at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:875)
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$null$40(PutHiveStreaming.java:676)
        at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$44(PutHiveStreaming.java:673)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2136)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2106)
        at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:627)
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$36(PutHiveStreaming.java:551)
        at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
        at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
        at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:551)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

2017-06-26 10:44:52,557 INFO [Timer-Driven Process Thread-6] hive.metastore Trying to connect to metastore with URI thrift://master003:9083
2017-06-26 10:44:52,563 INFO [Timer-Driven Process Thread-6] hive.metastore Connected to metastore.
2017-06-26 10:44:52,591 ERROR [Timer-Driven Process Thread-6] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=b738fdbd-0156-1000-899f-bf90221e8cf4] org.apache.nifi.processors.hive.PutHiveStreaming$$Lambda$668/1892130175@66b5fb91 failed to process due to org.apache.nifi.processor.exception.ProcessException: Error writing [org.apache.nifi.processors.hive.PutHiveStreaming$HiveStreamingRecord@4a69d63b] to Hive Streaming transaction due to java.lang.reflect.UndeclaredThrowableException; rolling back session: {}
org.apache.nifi.processor.exception.ProcessException: Error writing [org.apache.nifi.processors.hive.PutHiveStreaming$HiveStreamingRecord@4a69d63b] to Hive Streaming transaction due to java.lang.reflect.UndeclaredThrowableException
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$33(PutHiveStreaming.java:534)
        at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$11(ExceptionHandler.java:54)
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$34(PutHiveStreaming.java:541)
        at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148)
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$44(PutHiveStreaming.java:673)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2136)
        at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2106)
        at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:627)
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$36(PutHiveStreaming.java:551)
        at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
        at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
        at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:551)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.UndeclaredThrowableException: null
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:551)
        at org.apache.nifi.util.hive.HiveWriter.nextTxnBatch(HiveWriter.java:261)
        at org.apache.nifi.util.hive.HiveWriter.<init>(HiveWriter.java:73)
        at org.apache.nifi.util.hive.HiveUtils.makeHiveWriter(HiveUtils.java:46)
        at org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter(PutHiveStreaming.java:964)
        at org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter(PutHiveStreaming.java:875)
        at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$null$40(PutHiveStreaming.java:676)
        at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:127)
        ... 19 common frames omitted
Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://master003:9083', database='', table='olympics', partitionVals=[] }
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:578)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.access$700(HiveEndPoint.java:461)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl$2.run(HiveEndPoint.java:555)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl$2.run(HiveEndPoint.java:552)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        ... 27 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: null
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
        at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
        at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
        at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3906)
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3893)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1863)
        at sun.reflect.GeneratedMethodAccessor1374.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152)
        at com.sun.proxy.$Proxy370.lock(Unknown Source)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:573)
        ... 33 common frames omitted

thanks

6 REPLIES 6
Highlighted

Re: [NIFI] : Tuto Stream data into HIVE like a Boss using NiFi HiveStreaming

Contributor

@mayki wogno This looks like a known issue I have seen earlier. Could you please try the following workaround.

  1. Copy hive-site.xml, core-site.xml, hdfs-site.xml to the conf directory of NiFi
  2. Clear the Hive Configuration Resources property in PutHiveStreaming processor.
  3. Create an ExecuteScript processor on the canvas scheduled to run as often as the ticket needs to be refreshed (every hour should do for most setups) with the following groovy script, replacing nifi@HDF.COM with your principal and /etc/nifi.headless.keytab with your keytab
import org.apache.nifi.nar.NarClassLoader
import org.apache.nifi.nar.NarClassLoaders

NarClassLoaders.instance.extensionClassLoaders.each { c ->
  if (c instanceof NarClassLoader && c.workingDirectory.absolutePath.contains('nifi-hive')) {
    def originalClassloader = Thread.currentThread().getContextClassLoader();
    Thread.currentThread().setContextClassLoader(c);
    try {
      def configClass = c.loadClass('org.apache.hadoop.conf.Configuration', true)
      def hiveConfigurator = c.loadClass('org.apache.nifi.util.hive.HiveConfigurator', true).newInstance();
      def config = hiveConfigurator.getConfigurationFromFiles('')
      hiveConfigurator.preload(config)
      c.loadClass('org.apache.hadoop.security.UserGroupInformation', true).getMethod('setConfiguration', configClass).invoke(null, config)
      c.loadClass('org.apache.hadoop.security.UserGroupInformation', true).getMethod('loginUserFromKeytab', String.class, String.class).invoke(null, 'nifi@HDF.COM', '/etc/nifi.headless.keytab')
      log.info('Successfully logged in')
      session.transfer(session.create(), REL_SUCCESS)
    } catch (Exception e) {
      log.error('Unable to login with keytab', e)
      session.transfer(session.create(), REL_FAILURE)
    } finally {
      Thread.currentThread().setContextClassLoader(originalClassloader);
    }
  }
}
Highlighted

Re: [NIFI] : Tuto Stream data into HIVE like a Boss using NiFi HiveStreaming

Explorer

@Shashank Chandhok , the workaround is for making a kinit ?

i have this error :

2017-06-26 13:53:53,134 ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=79fafc92-3bb2-1978-0000-000077c9b131] ExecuteScript[id=79fafc92-3bb2-1978-0000-000077c9b131] failed to process due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: extensionClassLoaders for class: org.apache.nifi.nar.NarClassLoaders; rolling back session: {}
org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: extensionClassLoaders for class: org.apache.nifi.nar.NarClassLoaders
        at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:230)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: extensionClassLoaders for class: org.apache.nifi.nar.NarClassLoaders
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:159)
        at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
        at org.apache.nifi.script.impl.GroovyScriptEngineConfigurator.eval(GroovyScriptEngineConfigurator.java:54)
        at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:220)
        ... 11 common frames omitted
Caused by: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: extensionClassLoaders for class: org.apache.nifi.nar.NarClassLoaders
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:355)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:153)
        ... 14 common frames omitted
Caused by: groovy.lang.MissingPropertyException: No such property: extensionClassLoaders for class: org.apache.nifi.nar.NarClassLoaders
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:53)
        at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:66)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
        at Script2.run(Script2.groovy:14)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:352)
        ... 15 common frames omitted
2017-06-26 13:53:53,135 ERROR [Timer-Driven Process Thread-10] o.a.nifi.processors.script.ExecuteScript ExecuteScript[id=79fafc92-3bb2-1978-0000-000077c9b131] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: extensionClassLoaders for class: org.apache.nifi.nar.NarClassLoaders: {}
org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: extensionClassLoaders for class: org.apache.nifi.nar.NarClassLoaders
        at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:230)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: extensionClassLoaders for class: org.apache.nifi.nar.NarClassLoaders
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:159)
        at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264)
        at org.apache.nifi.script.impl.GroovyScriptEngineConfigurator.eval(GroovyScriptEngineConfigurator.java:54)
        at org.apache.nifi.processors.script.ExecuteScript.onTrigger(ExecuteScript.java:220)
        ... 11 common frames omitted
Caused by: javax.script.ScriptException: groovy.lang.MissingPropertyException: No such property: extensionClassLoaders for class: org.apache.nifi.nar.NarClassLoaders
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:355)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:153)
        ... 14 common frames omitted
Caused by: groovy.lang.MissingPropertyException: No such property: extensionClassLoaders for class: org.apache.nifi.nar.NarClassLoaders
        at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:53)
        at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:66)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296)
        at Script2.run(Script2.groovy:14)
        at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:352)
        ... 15 common frames omitted

Highlighted

Re: [NIFI] : Tuto Stream data into HIVE like a Boss using NiFi HiveStreaming

Contributor

@mayki wogno What is the nifi version and can you please upload a screenshot of the ExecuteScripct processor config

Highlighted

Re: [NIFI] : Tuto Stream data into HIVE like a Boss using NiFi HiveStreaming

Explorer

17528-puthivestreaming.jpg

@Shashank : use the last version of nifi 1.3

17527-hive-execut.jpg

          c.loadClass('org.apache.hadoop.security.UserGroupInformation', true).getMethod('loginUserFromKeytab', String.class, String.class).invoke(null, 'hive/master003@DOMAIN', '/etc/security/keytabs/hive.service.keytab')

and for the putHiveStreaming :

Highlighted

Re: [NIFI] : Tuto Stream data into HIVE like a Boss using NiFi HiveStreaming

Super Guru

NarClassLoaders has changed since that script was written, try this updated one:

import org.apache.nifi.nar.NarClassLoader
import org.apache.nifi.nar.NarClassLoaders
NarClassLoaders.instance.bundles.each { b ->
  def c = b.classLoader
  if (c instanceof NarClassLoader && b.bundleDetails.workingDirectory.absolutePath.contains('nifi-hadoop-nar')) {
    def originalClassloader = Thread.currentThread().getContextClassLoader();
    Thread.currentThread().setContextClassLoader(c);
    try {
      def configClass = c.loadClass('org.apache.hadoop.conf.Configuration', true)
      def config = configClass.newInstance();
      config.addResource('/path/to/core-site.xml');
      config.addResource('/path/to/hdfs-site.xml');
      c.loadClass('org.apache.hadoop.security.UserGroupInformation', true).getMethod('setConfiguration', configClass).invoke(null, config)
      c.loadClass('org.apache.hadoop.security.UserGroupInformation', true).getMethod('loginUserFromKeytab', String.class, String.class).invoke(null, 'nifi@HDF.COM', '/path/to/nifi.headless.keytab')
      log.info('Successfully logged in')
      session.transfer(session.create(), REL_SUCCESS)
    } catch (Exception e) {
      log.error('Unable to login with keytab', e)
      session.transfer(session.create(), REL_FAILURE)
    } finally {
      Thread.currentThread().setContextClassLoader(originalClassloader);
    }
  }
}
Highlighted

Re: [NIFI] : Tuto Stream data into HIVE like a Boss using NiFi HiveStreaming

Explorer

It is possible that it was an issue this PutHiveStreaming 1.3.0 must be replace with PutHiveStreaming 1.1.0 like in this post :

https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.htm...

Don't have an account?
Coming from Hortonworks? Activate your account here