Member since
07-30-2019
3434
Posts
1632
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 122 | 01-27-2026 12:46 PM | |
| 537 | 01-13-2026 11:14 AM | |
| 1167 | 01-09-2026 06:58 AM | |
| 973 | 12-17-2025 05:55 AM | |
| 483 | 12-17-2025 05:34 AM |
09-26-2016
11:35 AM
@Felix Duterloo The Read/Write stats on the face of each processor tell you how much content is being read from or written to NiFi's content repository. It is not intended to tell you how much data is written to the next processor or some external system. The purpose is to help dataflow managers understand which processors in their dataflows are disk I/O intensive. The "out" stat tells you how many FlowFiles were routed out of this processor on to one or more output relationships. In the case of a processor like putHDFS, it is typical to auto-terminate the "success" relationship and Loop the "failure" relationship back on to the putHDFS processor itself. Any FlowFile routed to "success" has confirmed delivery to HDFS. FlowFiles routed to failure were unable to be delivered and should produce a bulletin and log messages as to why. If the failure relationship is looped back on your putHDFS, NiFi will try again to deliver the file after the FlowFile penalty has expired. Matt
... View more
09-22-2016
11:20 AM
@mayki wogno When asking new questions unrelated to the current thread, please start a new Community Connection question. This benefits the community at large who may be searching for answers to the same question.
... View more
09-21-2016
05:29 PM
1 Kudo
There is the possibility that the time could differ slightly (ms) between when both now() functions are called in that expression language which could cause the result to push pack to 11:58:59. To avoid this you can simply reduce 43260000 by a few milliseconds (43259990) to ensure that does not happen so 11:59:00 is always returned.
... View more
09-21-2016
04:34 PM
2 Kudos
@Sree Venkata You can do this using a combination of NiFi Expression Language (EL) functions: ${now()
:minus(${now():mod(86400000)})
:minus(43260000) :format('MM-dd-yyyy hh:mm:ss')
} This EL statement takes now subtracts the remainder resulting from dividing now by 86400000 (number of milliseconds in 24 hours) and then subtracts an additional 43260000 (12 hours and 1 minute) from that result and finally formatting the output in the date format you are looking for. I confirmed this EL statement by using it in an UpdateAttribute processor: and if I look at the attributes on a FlowFile that was processed by the above, I see: You can see that the attribute "yesterday" is set to exactly one day earlier from "current time" and 11:59:00. Thanks, Matt
... View more
09-21-2016
03:53 PM
1 Kudo
@mayki wogno Nodes in NiFi cluster do not share data. Each works on the very specific data it has received through some ingest type NiFi processor. As such, each node has its own repositories for storing that specific data (FlowFile content) and the metadata (FlowFile attributes) about that node specific data. As a cluster every node loads and runs the exact same dataflow. One and only one node in the cluster can be the "primary node" at any given time. Some NiFi processors are not cluster friendly and as such should only run on one node in the cluster at any given time. (GetSFTP is a good example) NiFi allows you to configure those processor with a "on primary node" only scheduling strategy. While these processors will still exist on every node in the cluster they will will only run on the primary node. If the primary node designation in the cluster should change at any time, the cluster takes care of stopping the "on primary node" scheduled processors on the original primary node and staring them on the new primary node. When a node goes down, the other nodes in the cluster will not pickup working on the data that was queued on that down node at this time. That Node as Bryan pointed out will pick up where it left off on its queued data once restored to an operational state provide there was no loss/corruption to either the content or FlowFile repositories on that specific node. Thanks, Matt
... View more
09-21-2016
03:39 PM
If the node that goes down happens to be the "primary node", the cluster will automatically elect a new "primary node" from the remaining available nodes and start those "primary node only" processors.
... View more
09-21-2016
11:51 AM
1 Kudo
@Gerd Koenig After a closer look at the jaas file you posted above, I believe you issue is because of a missing " in the following line: principal="nifi@REALM; This line should actually be: principal="nifi@REALM";
Try making the above change and restarting your NiFi. Thanks, Matt
... View more
09-20-2016
01:45 PM
@Gerd Koenig Can you try changing the value you have for "Message Delimiter" from "\n" to an actual new line in your PutKafka processor? You can add a new line by holding the Shift key while hitting enter. The result will appear as below: Thanks, Matt
... View more
09-20-2016
12:35 PM
2 Kudos
@Gerd Koenig The question here is are you running Apache NiFi 0.6 or HDF 1.2? I believe you are using Apache NiFi 0.6 which does not understand PLAINTEXTSASL as the security protocol. The Kafka 0.8 in HDP 2.3.2 and the Kafka 0.9 in HDP 2.3.4 use a custom Hortonworks Kafka client library. Kafka 0.8 in HDP 2.3.2 introduced support for kerberos before it was supported in the community. That support introduced the PLAINTEXTSASL security protocol. later when Apache Kafka 0.9 added kerberos support they used a different security protocol (SASL_PLAINTEXT). In order for HDF 1.2 to work with HDP 2.3.2, the GetKafka processor was modified from the Apache GetKafka to use that modified client library. Hortonworks again modified the client lib in HDP 2.3.4 for Kafka 0.9 so that it was backwards compatible and still supported the PLAINTEXTSASL security protocol. So bottom line here is that HDF 1.2 NiFi can talk kerberos to both HDP 2.3.2 (Kafka 0.8) and HDP 2.3.4 (Kafka 0.9), but Apache NiFi cannot. The NiFi consume and publish Kafka processor available in NiFi 0.7, NiFi 1.0, and HDF 2.0 do not use the Hortonworks custom Kafka client lib and can be used with Kafka 0.9 but not Kafka 0.8. You will need to use the SASL_PLAINTEXT security protocol with these new processors. Thanks, Matt
... View more