Member since
09-04-2019
60
Posts
17
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
630 | 10-30-2023 06:50 AM | |
9130 | 02-27-2023 09:25 AM | |
1617 | 07-07-2022 09:17 AM | |
1498 | 01-26-2022 06:25 AM | |
2218 | 01-25-2022 06:19 AM |
01-24-2022
11:52 AM
1 Kudo
First confirm if the NiFi JVM is running search your logs for the word "UI is available" If it is then I would look at your systems firewall windows defender? If you do not see that log entry I would focus on why NiFi JVM is not coming up
... View more
01-24-2022
11:44 AM
Old post but for awareness: As reported here: UUID Logical type is now included under Apache Parquet 1.12 https://issues.apache.org/jira/browse/PARQUET-1827 Apache NiFi will use apache Parquet 1.12 starting from Apache NiFi 1.14
... View more
01-21-2022
09:46 AM
1 Kudo
Without knowing how the FTP directory is populated, you might have an issue with the ListFTP state and might consider changing its its Listing Strategy, However assuming that is no issue and you want to only perform an action based on any file with a naming structure like this: xxxx-xxxx-xxxx-xxxx-20220121110000-xxxx-xxxx.csv and only if it is older than two hours from "now" I would run the output of your ListFTP to an UpdateAttribute and add these 2 properties: property name fileTime value ${filename:getDelimitedField(5,'-'):trim():toDate('yyyyMMddHHmmss'):toNumber()} property name timeNow value ${now():toNumber():minus(7200000)} Then route that to a RouteOnAttribute and add a new property: property name value 2 hours old ${fileTime:le(${timeNow})} Then you can drag that connection to follow on processing and the unmatched connection to other processing or terminate it. Explanation: filelName ${filename:getDelimitedField(5,'-'):trim():toDate('yyyyMMddHHmmss'):toNumber()} This grabs the time out of your filename and converts it to a Date object so it can be converted to its epoch representation timeNow ${now():toNumber():minus(7200000)} Sets a value 2 hours ago from current time ${fileTime:le(${timeNow})} If attribute fileName is less than or equal to timeNow it means that it is 2 hours old.
... View more
01-20-2022
12:05 PM
1 Kudo
I understand the issue EvaluateXPath does not take in an attribute syntax and fails at validation since it is validating for a valid Xpath path. I wonder if there is a way to use QueryRecord instead.
... View more
01-20-2022
10:58 AM
Have you considered looking at the ListFtp property: Minimum File Age and set it to 2 hours
... View more
01-20-2022
09:43 AM
Deleting that state directory should not be a normal maintenance function. What you initially described is a very odd case. The fact that your node went down 003 and if it was the primary node or the coordinator node, internally there would have been a election to nominate a new cluster member node to perform those functions. In your case node 003 is a member of the cluster but it is not connected. Why it is not connected could be the cause of n reasons typically node is down or it was manually disconnected. When you see that message how many member nodes do you have? I expect the UI to show 2/3 because node 3 is not connected. The solution is to connect it by fixing the issue of why node is down or connect it through the UI
... View more
01-19-2022
10:03 AM
Reading your sample log messages closer I can see that the coordinator received a heartbeat from node 3 "2022-01-19 16:04:52,573 INFO [Process Cluster Protocol Request-30] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 35f2ed1a-ca6f-4cc6-ab4a-6c0774fc9c6d (type=HEARTBEAT, length=2837 bytes) from nifi-hatest-03:9091 in 19 millis" So I also wonder if we have a caching rendering issue, can you see what the UI shows using incognito mode? And finally if this is a DEV environment you can also try and delete your local state directory, that value is set in file "state-management.xml" Deleting state will clear out any local state your processors might depend on if configured as such, so remove with caution. It will also clear out cluster node ID's it local knows of.
... View more
01-19-2022
09:35 AM
This to me sounds like a hostname issue. could you confirm the value on nifi.properites for: nifi.cluster.node.address= nifi.web.https.host= Those values should match for the name of the host. If nothing there stands out check log entries for latest messages contain below strings: "org.apache.nifi.cluster.coordination.node.NodeClusterCoordinator: Status" "org.apache.nifi.controller.StandardFlowService: Setting Flow Controller's Node ID:" "org.apache.nifi.web.server.HostHeaderHandler"
... View more
01-19-2022
09:01 AM
Can I clarify that what you want to do is dealing with XML and XPath? Or based on this "I'm trying to pass an output value of a processor dynamically into another and use it as a new property" Output values in to flowfile attributes depend on the previous processor. If you are not dealing with XML then the processor you most likely need is ExtractText followed by an RouteOnAttribute
... View more
08-06-2020
07:27 PM
Hi @yogesh_shisode awesome that you are exploring NiFi. So just to be clear Apache Zookeeper can be considered an external service to help with state management / NiFi clustering With that said and to make things "flow" better, NiFi allows us to start an embedded zookeeper cluster. To me it seems that is what you are trying to connect to given the IP examples, so you are trying to use NiFi's embedded Zookeeper capability. So let's delve a little into zookeeper, we have zookeeper the service that can be single node or multi node. when in multi node we have a zookeeper ensemble and when we have that we need to maintain a quorum. This answer is very eloquently explained https://stackoverflow.com/questions/25174622/difference-between-ensemble-and-quorum-in-zookeeper And with that said please make sure you follow this guide: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#embedded_zookeeper In it it discusses howe to configure the NiFi to start up as an embedded zookeeper service and the settings needed to accomplish this. For clarity port 2181 is the zookeeper listening port and depending on how many servers you configured to be your zookeeper servers based of this nifi.properties entry: nifi.state.management.embedded.zookeeper.start=false if it is set to true, then Nifi will start a zookeeper service too and will depend on this setting: nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties Which is all explained on the link I gave you https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#embedded_zookeeper AND once you decide who the member nodes of your zookeeper are then all the NiFi's regardless of wether they are zookeeper servers or not should have this property set: nifi.zookeeper.connect.string= ^^^ The same on all NiFi's And using your IP examples if you want 3 servers to be zookeeper servers, then I would expect this setting to be: nifi.zookeeper.connect.string=192.168.0.10:2181,192.168.0.20:2181,192.168.0.30:2181 And on those servers this setting to be: nifi.state.management.embedded.zookeeper.start=true And the additional configurations from the Apache NiFi admin guide linked above.
... View more