1973
Posts
1225
Kudos Received
124
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1915 | 04-03-2024 06:39 AM | |
| 3012 | 01-12-2024 08:19 AM | |
| 1644 | 12-07-2023 01:49 PM | |
| 2421 | 08-02-2023 07:30 AM | |
| 3366 | 03-29-2023 01:22 PM |
10-08-2017
10:33 PM
Call Spark: https://community.hortonworks.com/articles/73828/submitting-spark-jobs-from-apache-nifi-using-livy.html Or call Machine learning from NiFi or write Machine Learning or Deep Learning components in nifi https://community.hortonworks.com/articles/80418/open-nlp-example-apache-nifi-processor.html https://community.hortonworks.com/articles/81270/adding-stanford-corenlp-to-big-data-pipelines-apac-1.html https://community.hortonworks.com/articles/116803/building-a-custom-processor-in-apache-nifi-12-for.html
... View more
10-06-2017
06:54 PM
There has been a major upgrade of the processor thanks to @Simon Elliston Ball Check out the latest version 2.1. I am also prototyping a DL4J processor based on this and some code from the SkyMind guys.
... View more
10-05-2017
02:58 PM
Removed SSL context and it worked
... View more
10-05-2017
01:38 AM
Caused by: javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292)
at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1045)
at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:741)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:224)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performTasks(SSLSocketChannel.java:273)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:259)
... 3 common frames omitted on same machine client and server
... View more
10-05-2017
01:35 AM
2017-10-04 21:34:10,539 ERROR [Timer-Driven Process Thread-8] o.a.n.p.standard.PutDistributedMapCache PutDistributedMapCache[id=c36247a2-015e-1000-57eb-1d718333b305] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=7891619c-81ce-414c-86ad-173b61210d59,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1507166772244-3, container=default, section=3], offset=34844, length=86],offset=0,name=285687216644919.orc,size=86] due to java.nio.channels.ClosedChannelException: {}
java.nio.channels.ClosedChannelException: null
at java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:289)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:141)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.write(SSLSocketChannel.java:603)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelOutputStream.write(SSLSocketChannelOutputStream.java:42)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.nifi.distributed.cache.protocol.ProtocolHandshake.initiateHandshake(ProtocolHandshake.java:58)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:324)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:376)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.put(DistributedMapCacheClientService.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy94.put(Unknown Source)
at org.apache.nifi.processors.standard.PutDistributedMapCache.onTrigger(PutDistributedMapCache.java:201)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-10-04 21:34:10,544 ERROR [Distributed Cache Server Communications Thread: c36fdffc-015e-1000-8e90-eaafc1aded82] o.a.n.r.io.socket.ssl.SSLSocketChannel org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel@7a44ad3a Failed to connect due to {}
javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1478)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:210)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:163)
at org.apache.nifi.distributed.cache.server.AbstractCacheServer$1$1.run(AbstractCacheServer.java:107)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292)
at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:1045)
at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:741)
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:224)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:966)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:963)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1416)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performTasks(SSLSocketChannel.java:273)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:259)
... 3 common frames omitted
2017-10-04 21:34:10,544 ERROR [Distributed Cache Server Communications Thread: c36fdffc-015e-1000-8e90-eaafc1aded82] o.a.n.d.cache.server.AbstractCacheServer Cannot create input and/or output streams for [c36fdffc-015e-1000-8e90-eaafc1aded82]
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1561)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:177)
at org.apache.nifi.distributed.cache.server.AbstractCacheServer$1$1.run(AbstractCacheServer.java:107)
at java.lang.Thread.run(Thread.java:745)
2017-10-04 21:34:10,556 ERROR [Timer-Driven Process Thread-8] o.a.n.r.io.socket.ssl.SSLSocketChannel org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel@4c763d85 Failed to connect due to {}
javax.net.ssl.SSLHandshakeException: Reached End-of-File marker while performing handshake
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:248)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:163)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.write(SSLSocketChannel.java:603)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelOutputStream.write(SSLSocketChannelOutputStream.java:42)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.nifi.distributed.cache.protocol.ProtocolHandshake.initiateVersionNegotiation(ProtocolHandshake.java:79)
at org.apache.nifi.distributed.cache.protocol.ProtocolHandshake.initiateHandshake(ProtocolHandshake.java:56)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:324)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:376)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.put(DistributedMapCacheClientService.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy94.put(Unknown Source)
at org.apache.nifi.processors.standard.PutDistributedMapCache.onTrigger(PutDistributedMapCache.java:201)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-10-04 21:34:10,556 ERROR [Timer-Driven Process Thread-8] o.a.n.r.io.socket.ssl.SSLSocketChannel org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel@4c763d85 Failed to connect due to {}
java.nio.channels.ClosedChannelException: null
at java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:289)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:141)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.write(SSLSocketChannel.java:603)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelOutputStream.write(SSLSocketChannelOutputStream.java:42)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.nifi.distributed.cache.protocol.ProtocolHandshake.initiateHandshake(ProtocolHandshake.java:58)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:324)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:376)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.put(DistributedMapCacheClientService.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy94.put(Unknown Source)
at org.apache.nifi.processors.standard.PutDistributedMapCache.onTrigger(PutDistributedMapCache.java:201)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-10-04 21:34:10,557 ERROR [Timer-Driven Process Thread-8] o.a.n.p.standard.PutDistributedMapCache PutDistributedMapCache[id=c36247a2-015e-1000-57eb-1d718333b305] Unable to communicate with cache when processing StandardFlowFileRecord[uuid=e88c8e2a-17dc-4bfd-9979-8d099a7e75ac,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1507166772245-4, container=default, section=4], offset=35462, length=76],offset=0,name=285687216644919.orc,size=76] due to java.nio.channels.ClosedChannelException: {}
java.nio.channels.ClosedChannelException: null
at java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:289)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.connect(SSLSocketChannel.java:141)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.write(SSLSocketChannel.java:603)
at org.apache.nifi.remote.io.socket.ssl.SSLSocketChannelOutputStream.write(SSLSocketChannelOutputStream.java:42)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.nifi.distributed.cache.protocol.ProtocolHandshake.initiateHandshake(ProtocolHandshake.java:58)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.leaseCommsSession(DistributedMapCacheClientService.java:324)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.withCommsSession(DistributedMapCacheClientService.java:376)
at org.apache.nifi.distributed.cache.client.DistributedMapCacheClientService.put(DistributedMapCacheClientService.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy94.put(Unknown Source)
at org.apache.nifi.processors.standard.PutDistributedMapCache.onTrigger(PutDistributedMapCache.java:201)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache NiFi
10-03-2017
01:55 PM
just a schema is need to do those very easy see https://community.hortonworks.com/articles/138632/data-flow-enrichment-with-nifi-lookuprecord-proces.html https://community.hortonworks.com/articles/106450/record-oriented-data-with-nifi.html https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi I used QueryRecord to convert so I could limit what I wanted to see: https://community.hortonworks.com/articles/118132/minifi-capturing-converting-tensorflow-inception-t.html https://community.hortonworks.com/articles/130814/sensors-and-image-capture-and-deep-learning-analys.html
... View more
09-26-2017
05:49 PM
Can I have one schema for read and one for write on an UpdateRecord processor?
... View more
Labels:
- Labels:
-
Apache NiFi
09-26-2017
03:54 PM
sounds like a good custom processor
... View more
09-26-2017
03:46 PM
1 Kudo
https://github.com/apache/nifi/pull/1920/files Didn't see docs for it but concat(/last_name,', ',/first_name) works
... View more
09-26-2017
03:23 PM
I want to create a new field in the output that is the concatenation of two input fields, example ouput schema name input schema firstname lastname name = firstname (space) lastname How could I do that with just the record processors?
... View more
Labels:
- Labels:
-
Apache NiFi