Member since
10-08-2016
59
Posts
16
Kudos Received
0
Solutions
02-01-2019
04:59 AM
1 Kudo
So I added a "class" section to the yml (See below) and then I get this error at the bottom instead: Controller Services:
- id: 9e991b8d-e2e5-3955-0000-000000000000
name: JMSConnectionFactoryProvider
type: org.apache.nifi.jms.cf.JMSConnectionFactoryProvider
class: JMSConnectionFactoryProvider
Properties:
SSL Context Service:
broker: tcp://10.0.0.60:61616
cf: org.apache.activemq.ActiveMQConnectionFactory
cflib: /home/ast/apache-activemq-5.15.8/lib
JMSConnectionFactoryProvider: org.apache.activemq.ActiveMQConnectionFactory
Does this mean My cpp minifi does not have the publishJMS processor? [2019-01-31 23:55:14.052] [org::apache::nifi::minifi::core::FlowConfiguration] [error] No Processor defined for PublishJMS
[2019-01-31 23:55:14.052] [org::apache::nifi::minifi::core::YamlConfiguration] [error] Could not create a processor PublishJMS with id 0bb3be56-0758-39a3-0000-000000000000
[2019-01-31 23:55:14.053] [main] [error] Failed to load configuration due to exception: Could not create processor PublishJMS
... View more
02-01-2019
04:22 AM
2 Kudos
Hi All! I took an Apache NiFi 1.8.0 NiFi Template publishjms.xml that was fully working publishing to an activeMQ queue (attached) Ran it through the minifi 0.5.0 cpp minifi converter and got the publishjmsyml.txt (attached) When I boot up minifi it gives the following error: [2019-01-31 23:08:31.014] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /home/ast/nifi-minifi-cpp-0.5.0/conf/minifi-uid.properties
[2019-01-31 23:08:31.014] [main] [info] MINIFI_HOME=/home/ast/nifi-minifi-cpp-0.5.0
[2019-01-31 23:08:31.014] [org::apache::nifi::minifi::Properties] [info] Using configuration file located at /home/ast/nifi-minifi-cpp-0.5.0/conf/minifi.properties
[2019-01-31 23:08:31.032] [org::apache::nifi::minifi::FlowController] [info] FlowController NiFi Configuration file /home/ast/nifi-minifi-cpp-0.5.0/conf/config.yml
[2019-01-31 23:08:31.032] [main] [info] Loading FlowController
[2019-01-31 23:08:31.032] [org::apache::nifi::minifi::FlowController] [info] Load Flow Controller from file /home/ast/nifi-minifi-cpp-0.5.0/conf/config.yml
[2019-01-31 23:08:31.034] [org::apache::nifi::minifi::core::YamlConfiguration] [error] Unable to parse configuration file for component named 'JMSConnectionFactoryProvider' as required field 'class' is missing [in 'Controller Services' section of configuration file]
[2019-01-31 23:08:31.034] [main] [error] Failed to load configuration due to exception: Unable to parse configuration file for component named 'JMSConnectionFactoryProvider' as required field 'class' is missing [in 'Controller Services' section of configuration file]
Any ideas? The MQ client library directory is specified and the files are all there. And ideas what the 'class' field that is missing should be called? or how it should be put in the config.yml? Thank you all!
... View more
Labels:
- Labels:
-
Apache MiNiFi
09-19-2018
02:47 PM
I have 2 ListenHttp processor listening on the same port, with two different paths and I get the following error: Any ideas? why? Im using nifi 0.7.4 @Matt Clarke @Matt Burgess ListenHTTP[id=ed7a45f5-4c4e-485a-a53d-8178649ec1e0] ListenHTTP[id=ed7a45f5-4c4e-485a-a53d-8178649ec1e0] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30000 milliseconds: java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
... View more
Labels:
- Labels:
-
Apache NiFi
08-14-2018
04:18 PM
Here's a simple xslt created by @Jayson Ramey Convert <?xml version="1.0" encoding="UTF-8"?>
<property>
<path>./</path>
<filename>1447519809631049</filename>
<uuid>ae5f9a71-15fb-4206-92d6-6b7efaacf4b3</uuid>
<fragment.identifier></fragment.identifier>
</property> To <?xml version="1.0" encoding="UTF-8"?>
<property>
<entry key="path">./</entry>
<entry key="filename">1453076046536069</entry>
<entry key="uuid">24957de2-7db4-4d86-b305-746eccf34222</entry>
<entry key="fragment.identifier"/>
</property>
By using this: <?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:xs="http://www.w3.org/2001/XMLSchema" exclude-result-prefixes="xs" version="2.0">
<xsl:template match="node()|@*" name="identity">
<xsl:copy>
<xsl:apply-templates select="node()|@*"/>
</xsl:copy>
</xsl:template>
<xsl:template match="property/*">
<entry>
<xsl:attribute name="key" select="name()"/>
<xsl:apply-templates select="@* | node()"/>
</entry>
</xsl:template>
</xsl:stylesheet>
... View more
06-13-2018
12:31 AM
1 Kudo
I'm using NiFi 1.6.0, in a 3 node cluster. When I use GETSFTP (Set to ALL Nodes) in a clustered nifi the cluster seems to distribute the data acquired evenly among nodes. Does this mean that all 3 servers GetSFTP the data evenly? I also tried using FETCH SFTP to get the listings and then did a site to site, back to my own cluster and It did NOT distribute the Fetch 0 byte files evenly among the nodes for the fetch SFTP load to be evenly distributed. What would be the best practice to Load Balance SFTPGET in a nifi cluster? John
... View more
Labels:
- Labels:
-
Apache NiFi
06-12-2018
02:00 PM
@Matt Clarke Any ideas? This thing is killing us!
... View more
06-11-2018
01:22 AM
@Matt Burgess any ideas as well?
... View more
06-11-2018
01:21 AM
@Matt Clarke Any ideas?
... View more
06-10-2018
11:35 PM
1 Kudo
So I have 4 740xd Servers with 80 Threads on each and 384GB of ram each. How should I best handle heap for each nifi VM on the server? Option 1: 1, VM 80 cpus, NiFi set to 300GB of Ram for heap? And have the 4 nifis clustered. Option 2: 10, VMs 8 cpus each, NiFi set to 32GB of Ram for heap for each VM, These NiFis would also be clustered, across the Virtualized 740 and across the 4 hardware 740s Note: all these files going through these nifis are less than 4GB, and 90% of them are under 10MB ALSO what generation should I use for Java Garbage Collection? Attached are two pictures explaining my options. John
... View more
06-07-2018
01:43 PM
Turns out I had: nifi.state.management.embedded.zookeeper.start=false Once I changed that to true it worked
... View more
06-05-2018
05:53 PM
@Matt Burgess or @Matt Clarke Any ideas?
... View more
06-05-2018
04:33 AM
Hi Everyone! I keep getting a: ERROR [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl Background retry gave up org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss My EDITABLE config files are here in the associated directories in my google drive below and attached, this is a 3 node cluster: https://drive.google.com/drive/folders/11xM-sz8mUvpaiOOS4aiZ94TQGHHzConF?usp=sharing I made sure FirewallD was off, all ports used are free, and I followed these guides: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.1/bk_administration/content/clustering.html AND https://community.hortonworks.com/articles/135820/configuring-an-external-zookeeper-to-work-with-apa.html Any help would be greatly appreciated!! Here's my configs in plain text for reference: ###################################################10.0.0.89 Server######################################################
####ZooKeeper.properties File#####
clientPort=24489
initLimit=10
autopurge.purgeInterval=24
syncLimit=5
tickTime=2000
dataDir=./state/zookeeper
autopurge.snapRetainCount=30
server.1=10.0.0.89:2888:3888
server.2=10.0.0.227:2888:3888
server.3=10.0.0.228:2888:3888
####nifi.properties cluster section####
# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=false
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=10.0.0.89
nifi.cluster.node.protocol.port=24489
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=3
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=10.0.0.89:24489,10.0.0.227:24427,10.0.0.28:24428
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi
###################################################################10.0.0.227 Server################################################################################################
####ZooKeeper.properties File#####
clientPort=24427
initLimit=10
autopurge.purgeInterval=24
syncLimit=5
tickTime=2000
dataDir=./state/zookeeper
autopurge.snapRetainCount=30
server.1=10.0.0.89:2888:3888
server.2=10.0.0.227:2888:3888
server.3=10.0.0.228:2888:3888
#####nifi.properties cluster section ###################
# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=false
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=10.0.0.227
nifi.cluster.node.protocol.port=24427
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=3
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=10.0.0.89:24489,10.0.0.227:24427,10.0.0.28:24428
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi
##################################################################################10.0.0.228 Server####### ##############################################################################
####ZooKeeper.properties File#####
clientPort=24428
initLimit=10
autopurge.purgeInterval=24
syncLimit=5
tickTime=2000
dataDir=./state/zookeeper
autopurge.snapRetainCount=30
server.1=10.0.0.89:2888:3888
server.2=10.0.0.227:2888:3888
server.3=10.0.0.228:2888:3888
#####nifi.properties cluster section ##########
# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=false
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=10.0.0.228
nifi.cluster.node.protocol.port=24428
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=3
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=10.0.0.89:24489,10.0.0.227:24427,10.0.0.28:24428
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi
... View more
Labels:
- Labels:
-
Apache NiFi
05-07-2018
04:53 PM
Matt, Thank you for your thorough feedback! How does nifi handle load balancing on its own if there was 1 listenHTTP processor on a cluster? Would we give the endpoint communicating customer 1 IP to reach us? And then nifi would broker that connection to one of the nodes in the cluster? Or would the data come through 1 node and then distribute to the other nodes? Basically I'm trying to use nifi clustering as the load balancer 🙂 Any suggestions? If that doesen't work I was going to try out HAproxy in front of the cluster IP. John
... View more
05-07-2018
03:41 PM
What would be the recommendation for a 40gbps NiFi cluster? And How would load balancing be handled with clustering? Would 1 IP be given for the cluster, lets say 1 HandleHTTP processor. and then the Network load will be distributed to the other nodes? Thank you!
... View more
Labels:
- Labels:
-
Apache NiFi
04-22-2018
03:39 AM
I keep getting the follwoing error from the ValidateXML nifi processor: Any ideas? 22:56:10 EDTERROR9c604c7b-e42a-4841-89f2-d0c176b63beb
ValidateXml[id=9c604c7b-e42a-4841-89f2-d0c176b63beb] ValidateXml[id=9c604c7b-e42a-4841-89f2-d0c176b63beb] failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.; processor will not be scheduled to run for 30000 milliseconds: java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task. I get no errors saying the xsd does not exist. and the permissions on the xsd are valid. John
... View more
Labels:
- Labels:
-
Apache NiFi
03-05-2018
08:35 PM
Hey @Matt Burgess just wondering if you had a chance to point me in the right direction. 🙂 I appreciate the help!
... View more
02-27-2018
08:33 PM
Yeah! the path is ./docProps/app.xml Its taking apart a docx/xlsx/pptx document and looking at the tag that tells the version of office it was created with. Any office document is really just a zip file. Hence what I'm trying to do 🙂 Thanks for your help!
... View more
02-26-2018
10:46 PM
@Matt Burgess I want to look at an xml doc inside the zip within a directory in the zip. And grab a value of a tag. The thing is I want the original flowfile to pass along with the tag value. Thanks Matt!
... View more
02-26-2018
10:19 PM
1 Kudo
I am creating some code to inspect a zip file with multiple sub folders and I want to look through the files in the zip and then pass the original content. I am trying to do this without writing the file to disk. Any ideas for how I could do this all in ram? or the nifi content repositories? I also used a guide to write the code in groovy and I was wondering how I could pass original content, in groovy? John
... View more
Labels:
- Labels:
-
Apache NiFi
02-19-2018
09:15 PM
Also, the reporting logs no longer appear in the nifi-app.log but they do not populate in the new file specified in the logback.xml I am using Apache nifi 1.3.0.
... View more
02-19-2018
09:08 PM
2 Kudos
I tried following this guide, but it didn't work. https://community.hortonworks.com/articles/79849/nifi-monitoring-controllerstatusreportingtask-with.html I just need to output ControllerStatusReportingTask logs into a separate file in my logs directory. Attached is mylogback.xml logback.xml Any ideas? John
... View more
Labels:
- Labels:
-
Apache NiFi
02-05-2018
10:45 PM
@Matt Clarke Hey Matt, Thanks for the quick feedback! So I can just change the batch size on the listen tcp to a really large number and regaurdless of the actual number of new lines it will package back together a full flowfile? I really need to do TCP to TCP because there is a middle system I will have to send my stream through, that will inspect the stream and TCP is the only way to do so. Any other ideas? John
... View more
02-05-2018
03:29 PM
1 Kudo
I am unable to get TCP nifi to nifi to work. I have a put TCP processor that puts files to a port and host, but when the receiving host receives the TCP via listen TCP it receives a TON of small files that are not the original flowfile. Is listen TCP not putting the packets back together into a flowfile? or am I doing something wrong here? Attached are pics of my configs and setup. Thank you!
... View more
Labels:
- Labels:
-
Apache NiFi
09-05-2017
03:04 AM
2 Kudos
I have a few nifi boxes and they are load balanced using HAproxy. In our architecture we decided not to use NiFi Clustering. I have some data feeds that require In Order Delivery of files (xml). Since their data is load balanced across multiple servers, I cannot guarantee that the files will be delivered in the order that they were received. Any Ideas for how I could achieve this? Is there an attribute I could create? and a processor that maintains order? and waits for files of certain values to be in a queue before delivering them? Maybe a groovy script that could accomplish this (even an example of a groovy script that pays attention to attributes and multiple files would be helpful? John
... View more
Labels:
- Labels:
-
Apache NiFi
08-07-2017
05:04 PM
In apache nifi how can I make the putfile processor use . rename as it writes a file out to the filesystem? or does it do this by default? Where it would write the file as a .filename and then rename it to the actual filename? John
... View more
Labels:
- Labels:
-
Apache NiFi
07-28-2017
09:29 PM
It looks like the file where you are looking for the attribute of "host" is the file that is generated by the "Monitor Activity" processor, not the other file that is being sent into the Monitor activity processor (the one that has the host attribute). To fix this just add an update attribute processor after the "Monitor Activity" processor and create the host attrbiute and pass that into the put email. Hope this help! NiFi FTW!
... View more