Member since
02-16-2017
4
Posts
0
Kudos Received
0
Solutions
02-21-2017
04:11 PM
Thank you @Pierre Villard I will surely have a look.
... View more
02-21-2017
12:30 PM
Hello, We are planning to use NiFi to consume data from Kafka, transform and store the same onto HDFS. We are using Kafka 0.10.0 version hence using consumeKafka. It works perfectly fine when all the properties are set correctly. However we have few questions and it will be great if someone can help with the answers. 1. How NiFi handles the Kafka offset, does it maintain anything by itself or depends upon default topic "__consumer_offsets"? 2. As we set "Offset reset" to "earliest" i.e. to start from the beginning in case there is no initial offset to Kafka or the offset is no longer valid. Apart from the "earliest/ latest/ none" is there any other mechanism to handle the offsets? 3. Can we parallelize NiFi execution? Can it run on multiple servers on a distributed fashion or we need to install it in every single server we want it to run? If it is running only on a single machine how to scale it? Where does NiFi executes its back-end jobs, is it on a local JVM or Jetty Server process or something else? Thanking in advance!
Anshu
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
02-16-2017
03:02 PM
@Matt Clarke Thanks! I will do that. Regret for any inconvenience.
... View more
02-16-2017
11:53 AM
Hello, Anshu here. We have a requirement to anonymize IP address. So we would identify IP address and replace the last part with some arbitrary value. We have tried using the following regex for that (with "Regex Replace" for replacement strategy, "Line-by-Line" as evaluation mode) (1) \\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3} (2) ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$ (3) ((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?) None of this works. Can you help me with the right value of regex we can use and the replacement value as well. Thank you in advance!
... View more