@Alvin Jin NiFi would have never started if a port was not specified. Let me be clear that you are setting up yoru NiFi to either run over a secure or non-secure port and not both. for a non-secure NiFi set these: nifi.web.http.host= nifi.web.http.port= and for a secured NiFi, set these: nifi.web.https.host= nifi.web.https.port= I am not familiar with Kubernete containers. Is the hostname of the container the same as the public facing hostname? Thank you, Matt ... View more

@dhieru singh Yo may also consider a two phase approach to mergeContent processor. By using two MergeContent processors in series, you will reduce your NiFi's heap usage and the number of FLowfiles needing to be merged in each iteration. For example, you may have first mergeContent merge based on number of FlowFiles (min 10000 and max 15000). Then have the second merge on size (assuming 64 MB to 128 MB for your HDFS, you would set min size to 64 MB and max to 128 MB). Thanks, Matt ... View more

@Alvin Jin With NiFi 1.5, NiFi has become more restrictive with requard to the allowed headers coming from a client. The hostname in the request header is checked against the configured hostname in the nifi.properties file: nifi.web.http(s).host= If they do not match, you will encounter the error you are seeing. So you will need to access the NiFi UI in 1.5 using the same hostname as specified in that property. Thanks, Matt ... View more

@Andrew thomas Once you have a NiFi cluster installed and running, any change you make within the NiFi Ui affects every node in the cluster. There is no way in a NiFi cluster to deploy a flow to just one node only. There is no need to stop any existing running flow within the UI when you are adding/building new flows within the same UI. Any configuration changes that affect any one of NiFi's configuration files (except logback.xml) will require a NiFi restart before those changes will take affect. Thank you, Matt ... View more

@Chris Lundeberg Are you entering "${schema.fingerprint}" or "schema.fingerprint" in the "Correlation Attribute Name" property field of the MergeContent processor? If you are looking to bin files where the value assigned to the "schema.fingerprint" attribute matches, you will want to enter only "schema.fingerprint" in that property. If you want your correlation attribute name to be more unique, you can use the updateAttribute processor before the MergeContent to create something more unique based on both ${schema.fingerprint} and ${tablename}. for example: followed by MergeContent configured similar to below: You understanding of my original explanation was correct. That could also explain missing attributes since all Flwofile without the attribute would end up in same bin and only matching attributes would be retained on merged FlowFiles. As far as one or many duplicate flows. It makes sense to have only a single flow here, but having many identical flows is also ok. Only thing to consider is NiFi processors obtain threads from a thread pull configured in NiFi. The more processors you have, the more processors there are requesting to use a thread from that thread pool. Plus more dataflows just means more processors to manage. The "Max Timer Driven thread count" is set within "Controller settings" found within the hamburger menu icon in the upper right corner of the NiFi UI. You will also find a setting for "Max Event Driven Thread count there",but do not change the value there. There is nothing you will add to the canvas that will use Event Driven Threads unless you specifically configure the processor to use them. This is a deprecated feature that only still exist to avoid breaking backwards compatibility with older flows. Timer Driven thread are much more efficient and will out perform Event Driven threads anyway. The default for Timer Driven threads is only 10. a good staring place here is 2 - 4 times the number of cores you have on a single NiFi host. Assume a 4 node NiFi cluster 4 hosts X (32 cores per node). You would set the Max Timer driven Thread count setting to 64 - 128. Assuming you set it to 32, this would mean there are 32 threads available per host for a cluster total of 128. Monitor top on your systems as you run your dataflow and adjusts as you see fit from there. Thanks, Matt Please take a moment to click "Accept" if you feel I have addressed your questions. ... View more

@Eric Lloyd With the above configuration, it would only take 1 FlowFile to be assigned to a bin before that bin was marked eligible for merging. There is nothing there that force the processor to wait for other FlowFiles to be allocated to a bin before merge, Both minimums are set to 1 FlowFile and 0 Bytes. In order to actually get 100,000 Flowfiles (this is high and may trigger OOM), there would need to be 100,000 Flowfiles all with the same correlation attribute value in the incoming connection queue at the time the processor runs. This is almost certainly not going to be the case. The Max bin age simply sets an exist strategy here. It will merge a bin regardless if minimums have been met if the bin age has reached this value. You may want to set more reasonable values for your mins and also consider using multiple mergeContent processors in series to step up to the final merged number you are looking for. Thanks, Matt ... View more