I think you need to remove the value for "Maximum Number of Entries", in your screenshot is set to 1000 which means it would attempt to merge at 1000 before seeing all 325070 fragments. Just leave it blank. ... View more

Here is template that demonstrates the behavior on a much smaller scale: splitandmergewithdefrag.xml GenerateFlowFile produces a flow file every 10 seconds with 3 lines of text, SplitText splits each line into 3 flow files, MergeContent defrags it back into 1 flow file, and LogAttribute logs the payload to nifi-app.log. ... View more

So just to recap the scenario, ExecuteSQL should be producing 1 flow file with 325k records in Avro, then SplitAvro (assuming Output Size is 1) produces 325k flow files with 1 Avro record each. Each of these split flow files should have: fragment.identifier - a unique id representing the overall batch of 325k flow files, this should be same on all 325k flow files fragment.count - the total number of fragments with in the fragment.identifier, this should be 325k in your case fragment.index - the index for the given flow with in the 325k fragments, so this should be different on each flow file and should be values like 1 - 325k When using Defrag mode you don't need to set Min/Max Number of Entries, it is going to be based off the attributes above, so you can leave those at 1 and blank respectively. Each fragment identifier will equate to 1 bin, so the number of bins only has to be equal to or greater than the number of fragment.identifiers processed concurrently, it may only be 1 in your case. I'm not sure that you need the run duration set so high to 60 mins, the processor can run frequently and just won't do anything until seeing all the records for the fragment. ... View more

Can you provide some more information like what is your flow doing (screenshot/template)? what is your custom processor doing? how do you know it is unresponsive? what version of NiFi are you using? Thanks. ... View more

There fragment.identifier should represent the overall flow file, so if there was an Avro data file with 1,000 records, there would be 1 identifier for that groups those thousand records together, and then there would be indexes 1 to 1000. Is that not the behavior you are seeing? Can you share a template of your flow? ... View more

When a new node joins the cluster it can either inherit the flow and authorizations (users + policies) from the cluster, or it has to have the exact same flow and authorizations. Lets focus on the authorizations since your error messages indicates that is where the problem was... In order to inherit the authorizations the new node needs to have 0 users and 0 policies, this means the new node can not have a value for "Initial Admin" or "Node Identities" or "Legacy Authorized Users". Populating any of those would generate users and policies on that node. If you started up once with any of those populated then you will want to shutdown, clear out those values, and delete users.xml and authorizations.xml. Generally once a cluster has been running for a while you have likely added users or modified policies through the UI, so the inheriting approach described above is the only way you can get a new node to join. If you happen to be adding a new node right after you just started a new cluster, and you haven't modified any of the users or policies through the UI then you should be able to just make sure the authorizers.xml has the exact same values as the other nodes and it should generate the same users and policies and be able to join. ... View more

Matt's answer below is correct.... If you are doing LDAP authentication then none of your end users need certificates, only the NiFi nodes themselves do, NiFi still needs to be running on https. In you config above your Initial Admin is blank, you always want to provide a value for this... If you are using certificates authentication then it is the DN of your cert, if you are using LDAP then it is the DN of an LDAP users, and if you are using Kerberos then it is the principal of your Kerberos user. ... View more

You can use an UpdateAttribute processor and change the 'filename' attribute. ... View more