@mayki wogno Is the same directory the same size of everyone of your zookeeper nodes? If not you may be having an issue on only one of your znodes. You should be able to shutdown the zookeeper node and purge all those files. The pertain files will be re-written from the other znodes in the zookeeper cluster when it rejoins the zookeeper cluster. Zookeeper is storing information about who is your current cluster coordinator, primary node, and any cluster wide state various from various processor in your dataflows. I am assuming you are running the embedded zookeeper here. In that case the zookeeper.properties file should control the auto purge of the snapshots through the following properties: autopurge.purgeInterval=24 autopurge.snapRetainCount=30 The transaction logs should be handle via routine maintenance which you can find here: http://archive.cloudera.com/cdh4/cdh/4/zookeeper/zookeeperAdmin.html#sc_maintenance Thanks, Matt ... View more

@Diego Labrador Anytime you encounter the message "Unable to perform the desired action due to insufficient permissions. Contact the system administrator.", you are having an authorization issue. Authentication issues present different errors. You should inspect your nifi-user.log while trying to access the UI to see what the exact string is being passed to the authorizer. By default with ldap as your configured login identity provider, the full DN for the user who logged in is passed to the authorizer. By the looks of the above you configured on the CN= as your initial admin identity. The string passed to the authorizer will be shown in nifi-user.log and must matcha exactly (Case sensitive and spaces count as valid characters also. Thanks, Mat ... View more

@Sanaz Janbakhsh HDF 2.0 is based off Apache NiFi 1.0 which no longer has an NCM (NCM based clusters only exists in HDF 1.x or Apache NiFi 0.x versions). HDF 2.0 is a zero master cluster which requires Zookeeper (min 3 ZK nodes for Quorum) for cluster coordinator and primary node designations and for storing your cluster wide state. - Is physical servers recommended for HDF other than VM? I do recommend physical servers over VM for NiFi. Depending on the dataflow(s) you design (which processor and controller service components you use), The load put on your server can go form very light to very heavy. - How many minimum nodes needs to be deployed for the clustering? There is no minimum number of hosts in a NiFi cluster. You can actually even stand up a 1 node cluster (Pointless and actual will perform poorer the a standalone NiFi because of additional cluster overhead that is added). I suggest starting with a 3 node cluster to spread out your load and provide coverage if a node is lost. You can add additional node to an existing Nifi cluster later with minimal effort. - What is the minimum hardware requirements per each node? Not knowing exactly what you plan on doing in your dataflow with regards to your 35,000 FlowFiles per minute, it is difficult to make any CPU suggestions. Generally speaking it is good practice to setup a POC and see how it scales. The fact that you are working with a large number of very small files, NiFi JVM heap usage could potentially be high, so making sure you have enough memory on each node to give NiFi at least 8 GB of heap to start with. You will need additional memory for the OS and any other services running on these host other then NiFi. Thanks, Matt ... View more

@Simon Jespersen The "Host Key File" property is used to specify the file containing your trusted hosts (commonly named known_hosts and found by default in the .ssh directory). It is not the key you are using to connect with. This property works in conjunction with the "Strict Host Key Checking" property when it is set to "true" You are getting a key does not exist because this property ins un related to "Private Key Path" so NiFi is looking in its base default directory for this file. I believe what you are trying to do has nothing to do with host key checking. You want to configure the "Private Key Path" as the full path to your private key including the key name itself. Thank you, Matt ... View more

@Praveen Singh NiFi offers a variety of processors that support executing commands or scripts locally to the NiFi installation. In order to execute a script remotely to NiFi you would still need to have a local script that would be called by the processor that could then go perform that task. Thank you, Matt ... View more