Hi, I have a three node HDF 2.0 unsecured cluster , and i would like to get some log file in another system. I originally intend to test this : SYSLOG ==> KAFKA ==> HDF ==> HDFS But i have read this : https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html https://pierrevillard.com/tag/apache-minifi/ https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/bk_minifi-quick-start/content/index.html And i was wondering maybe I can do this : ( MINIFI:HDF ) ==> KAFKA ==> HDF ==> HDFS My question are: IS MY USE CASE POSSIBLE WITH THE SECOND SCHEMAS ? AND IF IT IS HOW ? I have tried the Getting started with MiNiFi in the first link but when i replace the config.yml with my HDF template ( transform with minifi toolkit and renamed config.yml ). i get this from logs/minifi-app.log : ERROR [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise) I'm missing many things here and need some help. best regards, Toky ... View more

Hello everyone thanks for reading me, i'm very new to nifi as you can see in some of my previous question 🙂 I have an unsecure NIFI 1.0 cluster, i think i have configured the nifi.properties and the zookeeper.properties right but when i'm trying to use a REMOTE PROCESS GROUP , i always end up with this " NiFi Flow does not have any input ports " This is my nifi.properties for one of my three servers # Site to Site properties nifi.remote.input.host=server1 nifi.remote.input.secure=false nifi.remote.input.socket.port=8022 nifi.remote.input.http.enabled=true nifi.remote.input.http.transaction.ttl=30 sec # web properties nifi.web.war.directory=./lib nifi.web.http.host=server1 nifi.web.http.port=8070 nifi.web.https.host= nifi.web.https.port= nifi.web.jetty.working.directory=./work/jetty nifi.web.jetty.threads=200 # cluster node properties (only configure for cluster nodes) # nifi.cluster.is.node=true nifi.cluster.node.address=server1 nifi.cluster.node.protocol.port=8021 nifi.cluster.node.protocol.threads=10 nifi.cluster.node.event.history.size=25 nifi.cluster.node.connection.timeout=5 sec nifi.cluster.node.read.timeout=5 sec nifi.cluster.firewall.file= nifi.cluster.flow.election.max.wait.time=5 mins nifi.cluster.flow.election.max.candidates=3 I have tried all configuration possible of the RPG with all the port here but I can't connect the RPG to a get file processor, I have read this https://community.hortonworks.com/articles/16461/nifi-understanding-how-to-use-process-groups-and-r.html and this https://community.hortonworks.com/questions/47516/nifi-rpg-and-clustering.html but those are for nifi inferiror version , can anyone explain me how to use the RPG without a NCM ? What i want to do is to get some file localy on one of my Nifi node by s2s but i think i'm missing something here Thanks for your time ... View more

Hi everyone, I'm trying to change this kind of timestamp: "2016-07-06T15:04:33.332+00:00" to this "2016/07/06" with apache Nifi, i have tried many things but failed miserably. This is my configuration of the ReplaceText processor Search Value = ${dbbf.timestamp_da} Replacement Value = ${bddf.timestamp_da:toDate("yyyy-MM-dd'T'HH:mm:ss.SSSZ"):format('yyyy/MM/dd')} Character Set = UTF-8 Maximum Buffer Size = 1 MB Replacement Strategy = Literal Replace Evaluation Mode = Entire text If Anyone can help i would really appreciate, Thanks in advance 🙂 ... View more

Hello everyone, I've start working with nifi last week so sorry if my question is an obvious one. I'm currentlty taking data in a Kafka Topic, those data are in Json format. I'm scaning every Json document and depending of the value of some fields, i would like to sort them in a directory with the name of the value. for example if i have : {"timestamp_da":"2016-07-06T15:04:33.332+00:00","type":"question","subject":"nifi"} {"timestamp_da":"2016-07-06T15:04:33.332+00:00","type":"question","subject":"Kafka"} ==> I would like to extract the timestamp value and convert it into "20160706" and then create a directory with it. After that every Json with the same timestamp will be route there. Is it possible ? or I'm I dreaming ? I initially was doing this with spark but i think it's a waste to use spark just for this. If my question wasn't clear enought i can give more details. Thanks in advance ... View more