@Xavier COUDRE There are multiple policies needed for S2S to work. The first policy is a "Global" policy (Found under "Policies" via the hamburger menu in upper right corner of NiFi UI): You must authorize the source NiFi instance(s) (This would be any NiFi instance running a RPG trying to connect to this NiFi). This allows the RPG to communicate with the target to retrieve details such as number of connected nodes, load on those nodes, list of authorized input and output ports and supported connection information (node hostnames, is RAW transfer enabled, what is RAW port...) - In order for any ports to be returned, the source NiFi instance(s) must be authorized in those ports. I believe this is the step you have already completed which we were discussing initially in this answer. - Thanks, Matt - HCC Forum Tip: Try to avoid responding to an existing answer with a new answer. Comment on answer instead. It makes discussion easier to follow, especially when multiple answer are being discussed. ... View more

@Jose Gonzalez You can specify more then one host but it is not required. Once the RPG establish a connection to the target host it will retrieve the S2S details of the target cluster and store that locally. If the host you provided become unavailable at anytime after that initial connection , it will try anyone of the other nodes it learned about previously to get S2S details. Having multiple nodes configured helps when NiFi by giving the source Nifi more then one target node to establish initial connection with. - Your load-balancing issue is completely unrelated to how many nodes URLS you configured in your RPG. Here is an article that covers how load-balancing works with an RPG: https://community.hortonworks.com/content/kbentry/109629/how-to-achieve-better-load-balancing-using-nifis-s.html - Thanks, Matt ... View more

@Olivier Drouin and @Xavier COUDRE - At this time Remote Process Groups are only able to see "remote" input and output port. NiFi only treats input and output ports added to the root canvas of the target NiFi as remote Process groups. So as you can see from the above diagram, your MiNiFi RPG will only be able to see the input port at the root level. You will then need to use an input port in each sub-process group to move the received FlowFiles from the root canvas level down in to Process group level 3. I completely understand the complexity this adds in a multi-nested process group dataflow design. - There is a Jira out there requesting an enhancement to NiFi to separate input/output ports into local and remote: components. https://issues.apache.org/jira/browse/NIFI-2933 - However, my understanding is that there are significant design hurdles for such a change. There have been other design discussion around being able to connect directly from a top level process group to a deep nested process group without needing to create ports at each level. - Hope this helps to clarify things, Matt ... View more

@mayki wogno It is very possible the slow responses are the result of a very high number of http requests coming in to your NiFi nodes. - The main contributor to high number of requests are Remote Process Groups (RPG). It is very common for users to design dataflows that use many RPGs throughout their canvas to redistribute FlowFiles across their cluster. Each RPG is pinging the target http Nifi instance for the current Site-To-Site (S2S) details. Assume you have a 5 node cluster with 20 RPGs all pointing back to same cluster as an example. That means that every RPG on every node is requesting S2S details every 30 seconds. That alone is 100 HTTP request every 30 seconds. There is a improvement implemented through https://issues.apache.org/jira/browse/NIFI-4598 (fixed in NiFi 1.5) to improve how RPGs work in this scenario. - Additionally NiFi 1.2 is hardcoded to allow only 100 concurrent http requests which can lead to temporary unavailable of the http endpoint. This was resolved in an improvement covered via https://issues.apache.org/jira/browse/NIFI-4143 (fixed in NiFi 1.4) which allows users to increase the number of allowed concurrent http requests. - Another suggestion to improve http endpoint performance would be to make sure your RPGs are configured to use the "RAW" transport instead of "HTTP". While S2S details are still retrieved over http, the transfer of FlowFiles would be sent over a dedicated socket port instead of over http port. - Upgrading to Apache NiFi 1.5 will include both of the fixes above. HDF 3.1 is based off Apache NIFi 1.5. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login and click "accept". ... View more

@Davide Isoardi - The UpdateAttribute processor does not read the content of a FlowFile. In order for the above Expression Language statements to work, the incoming FlowFile's must have FlowFile attributes "user" and "datetime" created on them. - Stop the UpdateAttribute processor and allow a few FlowFiles to queue. Then list that queue and verify what attributes currently exist on those listed FlowFiles. - Thanks, Matt ... View more

@laiju cbabu The most important things to understand about NiFi's cluster architecture is the every node in the cluster runs with its own local copy of the flow.xml.gz (this file contains every configuration any user has made the the NiFi Ui (building flows on canvas, adding reporting tasks, adding controller services, etc...). - Because of NiFi's HA control layer, user can login to any node in the an active cluster and make changes within the canvas. The NiFi control layer takes care of making sure those changes are replicated to every node connected to that cluster. - Each node also node runs with its own set of repositories (FlowFile, content, provenance and database). Since NiFi does not currently have a HA data layer, should a NiFi node go down the data currently being processed by that node will not be processed until that node is restarted. It is important that the flowfile and content repositories (essential for data integrity) are protected through using RAID disk setups. It is actually easy to standup an entirely new node that uses these same repos and pickup where the old dead node left off. There is no way to merge the contents of two node's repositories together however. - Thank you, Matt ... View more

@Olivier Drouin Where is this input port "From Minifi" located on your canvas? At top level canvas or with a process group? Input and output ports facilitate the movement of FlowFiles between a parent process group and a child process group (where the input and output ports exist). So Input/output ports created within a process group only allow for receiving or sending of FlowFiles from the parent process group. The top level of your canvas is actually just another process group (root process group). When you create an input or output port here it becomes a remote input/output port capable of transferring Flowfiles over S2S. You will even see that these root level input/output ports are rendered a little different and have more configuration options. Thanks, Matt ... View more

@Rahoul A Here is a typical logback appender entry for the nifi-app.log: <appender name="APP_FILE"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>100MB</maxFileSize> <!-- keep 30 log files worth of history --> <maxHistory>30</maxHistory> <!-- optional setting for keeping 10GB total of log files <totalSizeCap>10GB</totalSizeCap> --> </rollingPolicy> <immediateFlush>true</immediateFlush> <encoder> <pattern>%date %level [%thread] %logger{40} %msg%n</pattern> </encoder> </appender> - Their are couple of lines you want to pay attention to here: <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> The above line defines what type of policy is being used by this appender. Here is will roll based in size and time. <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> In the above line you can see log is designed to roll on the hour based on yyyy-MM-dd_HH. <maxFileSize>100MB</maxFileSize> The above lines states no single log file should be larger then 100 MB, so you must account for scenarios where logs for a single hour may exceed 100 MB. That is what the "%i" is for in the fileNamePattern line above. <maxHistory>30</maxHistory> The above lines dictates how many "hours" of logs to retain. Each hour may contain 1 to many log files because of the max size constraint. - - - So in your case, using above as an example you would set maxHistory to "240". - - - You will also see an additional option property (commented out in above example): <!-- optional setting for keeping 10GB total of log files <totalSizeCap>10GB</totalSizeCap> --> This property is important if you have limited logging disk space, It will start rolling off the oldest logs if you exceed this configured max threshold. - Thank you, Matt ... View more