Member since
11-22-2016
16
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1532 | 07-24-2018 03:37 PM |
06-18-2019
12:24 PM
Hi @Pierre Villard , The GET request for /nifi-api/flow/process-groups/{id}/controller-services is returning all Controller Services, not only the ones for the Process Group {id}. Do you mean I need to parse the returned JSON and look only for the ones with parentGroupId = {id} ? Thanks in advance.
... View more
04-25-2019
12:50 PM
Hello everyone, Does anybody know where I can configure a new flow persistence provider for Nifi Registry using Cloudera Manager? I tried the escape valve for staging/providers.xml, but I get a validation error if I enter the required value to integrate NiFi Registry with Git: <flowPersistenceProvider>
<class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class>
<property name="Flow Storage Directory">{{nifi_registry_internal_dir}}/git/perso-recognition-nifi-flows</property>
<property name="Remote To Push">origin</property>
<property name="Remote Access User">paulvid</property>
<property name="Remote Access Password">[your_key]</property>
</flowPersistenceProvider> I'm following this guide to configure Nifi Registry with Git: https://community.hortonworks.com/articles/222365/how-to-configure-git-for-nifi-registry-in-hdf-32.h... , but the xml configuration differs from Ambari to Cloudera Manager. I was able to make CM generate the correct providers.xml by using the properties below: Which results on the following providers.xml: [root@nifi1 399-nifiregistry-NIFI_REGISTRY_SERVER]# cat providers.xml
<providers>
<flowPersistenceProvider>
<class>org.apache.nifi.registry.provider.flow.FileSystemFlowPersistenceProvider</class>
<property name="Flow Storage Directory">/var/lib/nifiregistry/flow_storage</property>
</flowPersistenceProvider>
<flowPersistenceProvider>
<class>org.apache.nifi.registry.provider.flow.git.GitFlowPersistenceProvider</class>
<property name="Remote Access User">yyy</property>
<property name="Flow Storage Directory">/var/lib/nifi-registry/git/nifi_registry_demo</property>
<property name="Remote Access Password">xxx</property>
<property name="Remote to Push">origin</property>
</flowPersistenceProvider> But now I'm getting this error starting the Nifi service: ... Caused by: org.apache.nifi.registry.provider.ProviderFactoryException: Unable to load the providers configuration file at: /run/cloudera-scm-agent/process/398-nifiregistry-NIFI_REGISTRY_SERVER/providers.xml at org.apache.nifi.registry.provider.StandardProviderFactory.initialize(StandardProviderFactory.java:112) ~[na:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_192] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_192] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_192] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_192] at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366) ~[na:na] at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:309) ~[na:na] at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136) ~[na:na] ... 78 common frames omitted Caused by: javax.xml.bind.UnmarshalException: null at javax.xml.bind.helpers.AbstractUnmarshallerImpl.createUnmarshalException(AbstractUnmarshallerImpl.java:335) ~[na:1.8.0_192] at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.createUnmarshalException(UnmarshallerImpl.java:563) ~[na:1.8.0_192] at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:249) ~[na:1.8.0_192] at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:221) ~[na:1.8.0_192] at com.sun.xml.internal.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:276) ~[na:1.8.0_192] at org.apache.nifi.registry.provider.StandardProviderFactory.initialize(StandardProviderFactory.java:109) ~[na:na] ... 85 common frames omitted Caused by: org.xml.sax.SAXParseException: cvc-complex-type.2.4.a: Invalid content was found starting with element 'flowPersistenceProvider'. One of '{eventHookProvider}' is expected.
...
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera Manager
08-23-2018
12:19 AM
@Sandeep Nemuri, the name resolution works, but using the broker's FQDN didn't fix the problem or change the error message. I (and the other cluster services) can authenticate fine with Kerberos on all machines, only Kafka is not working on Nifi.
... View more
08-22-2018
06:37 PM
Hi @Sandeep Nemuri. That didn't help. This is a small proof of concept environment which doesn't use DNS. Was that why you recommended the FQDN? So it has a relationship with the Kerberos realm?
... View more
08-21-2018
08:43 PM
We’ve 2 clusters that were built for a proof of concept, one
with HDP 2.6.2.0, the other with HDF 3.1.2.0, Both were kerberized using a MIT KDC on the
same realm. The kerberization was done entirely through the Ambari wizard
without issues.
After this, the PublishKafka_1_0 and Consume_Kafka_1_0
processors started giving out errors. The Publish_Kafka processor is configured
as shown below.
The HDFS and Hive processors worked fine after similar setups,
only Kafka is not working.
Here’s the nifi-app.log snippet with the error:
2018-08-21 17:13:05,009 WARN [kafka-producer-network-thread
| producer-3] org.apache.kafka.clients.NetworkClient [Producer
clientId=producer-3] Connection to node -1 could not be established. Broker may
not be available.<br>2018-08-21 17:13:05,862 WARN [kafka-producer-network-thread
| producer-3] org.apache.kafka.clients.NetworkClient [Producer
clientId=producer-3] Connection to node -1 could not be established. Broker may
not be available.<br>2018-08-21 17:13:06,933 WARN [kafka-producer-network-thread
| producer-3] org.apache.kafka.clients.NetworkClient [Producer
clientId=producer-3] Connection to node -1 could not be established. Broker may
not be available.<br>2018-08-21 17:13:07,760 WARN [kafka-producer-network-thread
| producer-3] org.apache.kafka.clients.NetworkClient [Producer
clientId=producer-3] Connection to node -1 could not be established. Broker may
not be available.<br>2018-08-21 17:13:08,713 WARN [kafka-producer-network-thread
| producer-3] org.apache.kafka.clients.NetworkClient [Producer
clientId=producer-3] Connection to node -1 could not be established. Broker may
not be available.<br>2018-08-21 17:13:09,406 ERROR [Timer-Driven Process
Thread-2] o.a.n.p.kafka.pubsub.PublishKafka_1_0
PublishKafka_1_0[id=58dabab0-0165-1000-0000-00005b66debd] Failed to send
StandardFlowFileRecord[uuid=b8d0d94b-e241-448d-9c00-5b7aaca393b6,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1534875917068-1, container=default,
section=1], offset=46931, length=334],offset=0,name=448674237495637,size=332]
to Kafka: org.apache.kafka.common.errors.TimeoutException: Failed to update
metadata after 5000 ms.<br>org.apache.kafka.common.errors.TimeoutException: Failed to
update metadata after 5000 ms.<br>2018-08-21 17:13:09,410 WARN
[kafka-kerberos-refresh-thread-hdf_user1@---]
o.a.k.c.security.kerberos.KerberosLogin [Principal=hdf_user1@---]: TGT
renewal thread has been interrupted and will exit.<br>2018-08-21 17:13:09,411 ERROR [Timer-Driven Process
Thread-2] o.a.n.p.kafka.pubsub.PublishKafka_1_0
PublishKafka_1_0[id=58dabab0-0165-1000-0000-00005b66debd]
PublishKafka_1_0[id=58dabab0-0165-1000-0000-00005b66debd] failed to process due
to org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=46169fd4-1fd9-47af-969b-9fc4301a6f1d,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1534875917068-1, container=default,
section=1], offset=47265, length=334],offset=0,name=448673432494738,size=332]
is already marked for transfer; rolling back session: {}<br>org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=46169fd4-1fd9-47af-969b-9fc4301a6f1d,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1534875917068-1, container=default,
section=1], offset=47265, length=334],offset=0,name=448673432494738,size=332]
is already marked for transfer<br> at org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3073)<br> at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3055)<br> at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3086)<br> at
org.apache.nifi.controller.repository.StandardProcessSession.transfer(StandardProcessSession.java:1902)<br> at
org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0.onTrigger(PublishKafka_1_0.java:416)<br> at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)<br> at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1124)<br> at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)<br> at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)<br> at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)<br> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br> at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)<br> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)<br> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)<br> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br> at
java.lang.Thread.run(Thread.java:745)<br>2018-08-21 17:13:09,412 ERROR [Timer-Driven Process
Thread-2] o.a.n.p.kafka.pubsub.PublishKafka_1_0
PublishKafka_1_0[id=58dabab0-0165-1000-0000-00005b66debd]
PublishKafka_1_0[id=58dabab0-0165-1000-0000-00005b66debd] failed to process
session due to org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=46169fd4-1fd9-47af-969b-9fc4301a6f1d,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1534875917068-1, container=default,
section=1], offset=47265, length=334],offset=0,name=448673432494738,size=332]
is already marked for transfer: {}<br>org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=46169fd4-1fd9-47af-969b-9fc4301a6f1d,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1534875917068-1, container=default,
section=1], offset=47265, length=334],offset=0,name=448673432494738,size=332]
is already marked for transfer<br> at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3073)<br> at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3055)<br> at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3086)<br> at
org.apache.nifi.controller.repository.StandardProcessSession.transfer(StandardProcessSession.java:1902)<br> at
org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0.onTrigger(PublishKafka_1_0.java:416)<br> at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)<br> at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1124)<br> at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)<br> at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)<br> at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)<br> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br> at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)<br> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)<br> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)<br> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br> at
java.lang.Thread.run(Thread.java:745)<br>2018-08-21 17:13:09,412 WARN [Timer-Driven Process Thread-2]
o.a.n.p.kafka.pubsub.PublishKafka_1_0
PublishKafka_1_0[id=58dabab0-0165-1000-0000-00005b66debd] Processor
Administratively Yielded for 1 sec due to processing failure<br>2018-08-21 17:13:09,412 WARN [Timer-Driven Process Thread-2]
o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
PublishKafka_1_0[id=58dabab0-0165-1000-0000-00005b66debd] due to uncaught
Exception: org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=46169fd4-1fd9-47af-969b-9fc4301a6f1d,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1534875917068-1, container=default,
section=1], offset=47265, length=334],offset=0,name=448673432494738,size=332]
is already marked for transfer<br>2018-08-21 17:13:09,412 WARN [Timer-Driven Process Thread-2]
o.a.n.c.t.ContinuallyRunProcessorTask<br>org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=46169fd4-1fd9-47af-969b-9fc4301a6f1d,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1534875917068-1, container=default,
section=1], offset=47265, length=334],offset=0,name=448673432494738,size=332]
is already marked for transfer<br> at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3073)<br> at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3055)<br> at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3086)<br> at
org.apache.nifi.controller.repository.StandardProcessSession.transfer(StandardProcessSession.java:1902)<br> at
org.apache.nifi.processors.kafka.pubsub.PublishKafka_1_0.onTrigger(PublishKafka_1_0.java:416)<br> at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)<br> at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1124)<br> at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)<br> at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)<br> at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)<br> at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br> at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)<br> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)<br> at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)<br> at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br> at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br> at
java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
07-24-2018
05:28 PM
I found it. In my case it was /user/nifi on HDFS, which is the user that the flow connects with. It moves a .jar file under the .hiveJars subdirectory on that directory in order to run commands.
... View more
07-24-2018
05:15 PM
@Saikrishna Tarapareddy, can you please specify what folders and files needed the permissions? Are they on the HiveServer2 server or on HDFS?
... View more
07-24-2018
03:37 PM
Update: this was due to special characters on the <file>, which amqsput can't handle correctly. I replaced them to regular chars and it went well. For example, the string "São" was corrupted. "Sao" went fine.
... View more
07-24-2018
12:11 PM
The ConsumeJMS processor which is the origin of data on a processor group is failing whenever I cat a file and send it over to amqsput client application.
When I run:
cat <file> | /opt/mqm/samp/bin/amqsput <queue> <queue manager>
It fails. I then tried a script which echoes the file line by line but it also returns an error. I also made sure the file I cat is encoded as UTF-8 (ran iconv on it). I found a JIRA treating a similar error, NIFI-4976, from March 2018, but I'm not sure if it made it into HDF 3.1.0.0, which is the version I'm using.
When I echo a single message, it works fine.
Here's what I see on nifi-app.log:
2018-07-24 09:04:41,144 WARN [Timer-Driven Process Thread-4] o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] Processor Administratively Yielded for 1 sec due to processing failure
2018-07-24 09:04:41,144 WARN [Timer-Driven Process Thread-4] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] due to uncaught Exception: org.apache.nifi.jms.processors.MessageBodyToBytesConverter$MessageConversionException: Failed to convert BytesMessage to byte[]
2018-07-24 09:04:41,144 WARN [Timer-Driven Process Thread-4] o.a.n.c.t.ContinuallyRunProcessorTask
org.apache.nifi.jms.processors.MessageBodyToBytesConverter$MessageConversionException: Failed to convert BytesMessage to byte[]
at org.apache.nifi.jms.processors.MessageBodyToBytesConverter.toBytes(MessageBodyToBytesConverter.java:42)
at org.apache.nifi.jms.processors.JMSConsumer$1.doInJms(JMSConsumer.java:106)
at org.apache.nifi.jms.processors.JMSConsumer$1.doInJms(JMSConsumer.java:66)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:494)
at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
at org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
at org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSCMQ1049: The character set '1208(UTF-8) Unmappable Action: REPORT Unmappable Replacement: 63' cannot convert some or all of the string '[B@653dd9ad'
at sun.reflect.GeneratedConstructorAccessor608.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
at com.ibm.msg.client.commonservices.nls.NLSServices.createException(NLSServices.java:226)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromBytes(WMQUtils.java:382)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromByteBuffer(WMQUtils.java:421)
at com.ibm.msg.client.wmq.common.internal.messages.WMQTextMessage.getText(WMQTextMessage.java:240)
at com.ibm.msg.client.jms.internal.JmsTextMessageImpl.getText(JmsTextMessageImpl.java:205)
at com.ibm.jms.JMSTextMessage.getText(JMSTextMessage.java:124)
at org.apache.nifi.jms.processors.MessageBodyToBytesConverter.toBytes(MessageBodyToBytesConverter.java:40)
... 19 common frames omitted
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:816)
at com.ibm.mq.jmqi.system.JmqiCodepage.bytesToString(JmqiCodepage.java:690)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromBytes(WMQUtils.java:375)
... 24 common frames omitted
2018-07-24 09:04:42,148 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] failed to process due to org.apache.nifi.jms.processors.MessageBodyToBytesConverter$MessageConversionException: Failed to convert BytesMessage to byte[]; rolling back session: {}
org.apache.nifi.jms.processors.MessageBodyToBytesConverter$MessageConversionException: Failed to convert BytesMessage to byte[]
at org.apache.nifi.jms.processors.MessageBodyToBytesConverter.toBytes(MessageBodyToBytesConverter.java:42)
at org.apache.nifi.jms.processors.JMSConsumer$1.doInJms(JMSConsumer.java:106)
at org.apache.nifi.jms.processors.JMSConsumer$1.doInJms(JMSConsumer.java:66)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:494)
at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
at org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
at org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSCMQ1049: The character set '1208(UTF-8) Unmappable Action: REPORT Unmappable Replacement: 63' cannot convert some or all of the string '[B@65f756f8'
at sun.reflect.GeneratedConstructorAccessor608.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
at com.ibm.msg.client.commonservices.nls.NLSServices.createException(NLSServices.java:226)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromBytes(WMQUtils.java:382)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromByteBuffer(WMQUtils.java:421)
at com.ibm.msg.client.wmq.common.internal.messages.WMQTextMessage.getText(WMQTextMessage.java:240)
at com.ibm.msg.client.jms.internal.JmsTextMessageImpl.getText(JmsTextMessageImpl.java:205)
at com.ibm.jms.JMSTextMessage.getText(JMSTextMessage.java:124)
at org.apache.nifi.jms.processors.MessageBodyToBytesConverter.toBytes(MessageBodyToBytesConverter.java:40)
... 19 common frames omitted
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:816)
at com.ibm.mq.jmqi.system.JmqiCodepage.bytesToString(JmqiCodepage.java:690)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromBytes(WMQUtils.java:375)
... 24 common frames omitted
2018-07-24 09:04:42,148 ERROR [Timer-Driven Process Thread-8] o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] failed to process session due to org.apache.nifi.jms.processors.MessageBodyToBytesConverter$MessageConversionException: Failed to convert BytesMessage to byte[]: {}
org.apache.nifi.jms.processors.MessageBodyToBytesConverter$MessageConversionException: Failed to convert BytesMessage to byte[]
at org.apache.nifi.jms.processors.MessageBodyToBytesConverter.toBytes(MessageBodyToBytesConverter.java:42)
at org.apache.nifi.jms.processors.JMSConsumer$1.doInJms(JMSConsumer.java:106)
at org.apache.nifi.jms.processors.JMSConsumer$1.doInJms(JMSConsumer.java:66)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:494)
at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
at org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
at org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSCMQ1049: The character set '1208(UTF-8) Unmappable Action: REPORT Unmappable Replacement: 63' cannot convert some or all of the string '[B@65f756f8'
at sun.reflect.GeneratedConstructorAccessor608.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
at com.ibm.msg.client.commonservices.nls.NLSServices.createException(NLSServices.java:226)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromBytes(WMQUtils.java:382)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromByteBuffer(WMQUtils.java:421)
at com.ibm.msg.client.wmq.common.internal.messages.WMQTextMessage.getText(WMQTextMessage.java:240)
at com.ibm.msg.client.jms.internal.JmsTextMessageImpl.getText(JmsTextMessageImpl.java:205)
at com.ibm.jms.JMSTextMessage.getText(JMSTextMessage.java:124)
at org.apache.nifi.jms.processors.MessageBodyToBytesConverter.toBytes(MessageBodyToBytesConverter.java:40)
... 19 common frames omitted
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:816)
at com.ibm.mq.jmqi.system.JmqiCodepage.bytesToString(JmqiCodepage.java:690)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromBytes(WMQUtils.java:375)
... 24 common frames omitted
2018-07-24 09:04:42,149 WARN [Timer-Driven Process Thread-8] o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] Processor Administratively Yielded for 1 sec due to processing failure
2018-07-24 09:04:42,149 WARN [Timer-Driven Process Thread-8] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] due to uncaught Exception: org.apache.nifi.jms.processors.MessageBodyToBytesConverter$MessageConversionException: Failed to convert BytesMessage to byte[]
2018-07-24 09:04:42,149 WARN [Timer-Driven Process Thread-8] o.a.n.c.t.ContinuallyRunProcessorTask
org.apache.nifi.jms.processors.MessageBodyToBytesConverter$MessageConversionException: Failed to convert BytesMessage to byte[]
at org.apache.nifi.jms.processors.MessageBodyToBytesConverter.toBytes(MessageBodyToBytesConverter.java:42)
at org.apache.nifi.jms.processors.JMSConsumer$1.doInJms(JMSConsumer.java:106)
at org.apache.nifi.jms.processors.JMSConsumer$1.doInJms(JMSConsumer.java:66)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:494)
at org.apache.nifi.jms.processors.JMSConsumer.consume(JMSConsumer.java:66)
at org.apache.nifi.jms.processors.ConsumeJMS.rendezvousWithJms(ConsumeJMS.java:156)
at org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:147)
at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:58)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.ibm.msg.client.jms.DetailedJMSException: JMSCMQ1049: The character set '1208(UTF-8) Unmappable Action: REPORT Unmappable Replacement: 63' cannot convert some or all of the string '[B@65f756f8'
at sun.reflect.GeneratedConstructorAccessor608.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.ibm.msg.client.commonservices.j2se.NLSServices.createException(NLSServices.java:319)
at com.ibm.msg.client.commonservices.nls.NLSServices.createException(NLSServices.java:226)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromBytes(WMQUtils.java:382)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromByteBuffer(WMQUtils.java:421)
at com.ibm.msg.client.wmq.common.internal.messages.WMQTextMessage.getText(WMQTextMessage.java:240)
at com.ibm.msg.client.jms.internal.JmsTextMessageImpl.getText(JmsTextMessageImpl.java:205)
at com.ibm.jms.JMSTextMessage.getText(JMSTextMessage.java:124)
at org.apache.nifi.jms.processors.MessageBodyToBytesConverter.toBytes(MessageBodyToBytesConverter.java:40)
... 19 common frames omitted
Caused by: java.nio.charset.MalformedInputException: Input length = 1
at java.nio.charset.CoderResult.throwException(CoderResult.java:281)
at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:816)
at com.ibm.mq.jmqi.system.JmqiCodepage.bytesToString(JmqiCodepage.java:690)
at com.ibm.msg.client.wmq.common.internal.WMQUtils.computeTextFromBytes(WMQUtils.java:375)
... 24 common frames omitted
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
07-24-2018
11:59 AM
Thanks Matt. I installed the "aligned" HDP and HDF versions and it worked.
... View more
06-29-2018
06:54 PM
It is the same when you setup Postgres for the Streaming Analytics Manager. The path is /var/lib/ambari-agent/cache/common-services/STREAMLINE/0.5.0/package/scripts/params.py
... View more
06-28-2018
03:49 PM
In my experience, the connection error goes away if you remove "thrift://" from the URI.
... View more
06-28-2018
03:45 PM
I've a Nifi datalow where I get data from a source, build an Avro file and am ready to stream to Hive. I setup Hive as requested on https://community.hortonworks.com/questions/59411/how-to-use-puthivestreaming.html. I also moved the entire hive-client and hadoop-client directories to the Nifi node, and reference this hive-site.xml on the PutHiveStreaming processor. But I'm getting the following error on this processor: 2018-06-28 11:29:21,053 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=46762ec7-0164-1000-0000-00001008870a] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: Error writing [org.apache.nifi.processors.hive.PutHiveStreaming$HiveStreamingRecord@733df43a] to Hive Streaming transaction due to com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient: {}
org.apache.nifi.processor.exception.ProcessException: Error writing [org.apache.nifi.processors.hive.PutHiveStreaming$HiveStreamingRecord@733df43a] to Hive Streaming transaction due to com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordsError$1(PutHiveStreaming.java:555)
at org.apache.nifi.processor.util.pattern.ExceptionHandler$OnError.lambda$andThen$0(ExceptionHandler.java:54)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onHiveRecordError$2(PutHiveStreaming.java:562)
at org.apache.nifi.processor.util.pattern.ExceptionHandler.execute(ExceptionHandler.java:148)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$12(PutHiveStreaming.java:740)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2175)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2145)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:694)
at org.apache.nifi.processors.hive.PutHiveStreaming.lambda$onTrigger$4(PutHiveStreaming.java:572)
at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184)
at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger(PutHiveStreaming.java:572)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1122)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.apache.hive.hcatalog.common.HiveClientCache.getOrCreate(HiveClientCache.java:291)
at org.apache.hive.hcatalog.common.HiveClientCache.get(HiveClientCache.java:266)
at org.apache.hive.hcatalog.common.HCatUtil.getHiveMetastoreClient(HCatUtil.java:558)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.getMetaStoreClient(HiveEndPoint.java:544)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:312)
at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:278)
at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:215)
at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:192)
at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:122)
at org.apache.nifi.util.hive.HiveWriter.lambda$newConnection$1(HiveWriter.java:246)
at org.apache.nifi.util.hive.HiveWriter.lambda$callWithTimeout$4(HiveWriter.java:365)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1566)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:92)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:138)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:124)
at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:295)
at org.apache.hive.hcatalog.common.HiveClientCache$5.call(HiveClientCache.java:291)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 17 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1564)
... 27 common frames omitted
Caused by: java.lang.NullPointerException: null
at org.apache.thrift.transport.TSocket.open(TSocket.java:209)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:487)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:282)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:188)
at org.apache.hive.hcatalog.common.HiveClientCache$CacheableHiveMetaStoreClient.<init>(HiveClientCache.java:406)
... 32 common frames omitted I thought that by copying and referencing the hive-client dir I'd have all the jars needed. Is this a problem with the CLASSPATH, with the processor configuration or something else?
... View more
Labels:
- Labels:
-
Apache Hive
-
Cloudera DataFlow (CDF)
01-18-2018
03:44 PM
Hi Dinesh. These are default values for recent versions of Hive (0.13.0 and later). Sources: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.server2.async.exec.threads and http://atlas.apache.org/Bridge-Hive.html
... View more
05-09-2017
06:12 PM
I could at least start Zookeeper by doing this (change for your environment's exact paths): cd /usr/hdp/2.5.3.0-37/zookeeper; mkdir bin; cd bin; cp /var/lib/ambari-agent/cache/common-services/ZOOKEEPER/3.4.5/package/files/* .; ln -s /usr/hdp/2.3.0-37/zookeeper /usr/hdp/current/zookeeper-server
Source for the last command in the string of commands was: https://github.com/KaveIO/AmbariKave/wiki/Detailed-Guides-Troubleshooting
... View more
11-22-2016
06:45 PM
@Gilsomar N Resende The original HortonWorks blog post that announces the NFS gateway feature is at http://hortonworks.com/blog/simplifying-data-management-nfs-access-to-hdfs/.
The text itself is a bit dated and lists information right out of the documentation, but the discussion after it is very informative. If you didn't read the discussion, I suggest you do. The NFS Gateway is just a skin over a DFS client. But this skin can grow thick with overhead when doing too many writes. It also is not HA, and has poor support for a kerberized cluster. The comments on that post list a way to scale the gateway to achieve throughput, which is to have several mount points using several instances of the NFS gateway and write to them in a round-robin fashion, but this just adds complexity to something that would be adopted only to make ingestion simpler. There is no free lunch, and the added skin/complexity of the NFS gateway is not worth the trouble in my experience. Better stick with the actual DFS client and work on optimizing the file write and processing on the HDP cluster. Maybe aggregate the files through a Flume pipeline, or a custom script that does that and then copies to HDFS using the HDFS client.
... View more
