@Vinod Chhabria There is nothing that would prevent that connection from being allowed. Try dragging the connection again. If it still does not highlight green try the following: Try making a copy of your replaceTEXT processor and see if you can connect to the copy. Right click on replace TEXT and select "copy" then right click on empty graph space and click "paste". Matt ... View more

@Sunile Manjee You are absolutely correct, HDF does not require that all the HDF services are installed. All the services are installed through RPMs. When you run through the Ambari Wizard you will be asked to select the services you wish to "Deploy". By default they are all checked except for log search which is currently Technical Preview (TP). Simply uncheck all the services you do not want to install. If you uncheck something that is a dependency of NiFi, the wizard will let you know. Thanks, Matt ... View more

@Sunile Manjee As far as best practices goes, we do not recommend installing ZK on the same servers/nodes as NiFi. NiFi dataflows can be very CPU, disk, and/or memory intensive. Any of these can interfere with ZK --> NiFi comms/performance. This can result in NiFi nodes dropping from cluster, new NiFi cluster coordinators being assigned, and/or new primary nodes being elected frequently. While it does work, i would keep away from co-location in production for sure. Thank, Matt ... View more

@Sunile Manjee There is no reason you can't use another ZK (Including the one provided in HDP). While there is currently no support for installing NiFi within an HDP Ambari stack, you can point your NiFi installation via its config at the ZK quorum in your HDP stack. If you install NiFi via the HDF Ambari stack, it does have a dependency that forces the installation of ZK in the HDF stack and configures your NiFi service to use it. You can however alter the NiFi configs to use your other ZK. If you install HDF NiFi via command line and not with Ambari, you can configure it to use your HDP ZK quorum out the gate. Thanks, Matt ... View more

@Sunile Manjee The "nifi.cluster.is.node" parameter is used to specify whether the NiFi installation is a standalone installation (false) or  a node in a NiFi cluster (true).  When set to true, things like ZK are required as "cluster" wide state management will not take affect and is stored in ZK. With true NiFi also requires a ZK for the NiFi Cluster (even if you have only 1 node) The NiFi node will send heartbeats to ZK and a primary node and cluster coordinator will be elected. By setting to false, NiFi does not need a ZK for all those things above. [11:24 AM] Matthew Clarke: State management is only local as well. You will get better performance out of a Standalone NIFi (false) then you will out of a 1 node cluster (true with only one node) because you reduce the overhead by not having the ZK piece. Thanks, Matt ... View more

@Ronak Jangir The HDFS client does not currently support the LzoCodec and the core-site.xml file you are using includes it. It should work after you remove “com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec” from the “io.compression.codecs” property in the “core-site.xml” file you have referenced in your putHDFS processor. Thanks, Matt ... View more

@Santiago Ciciliani Do you have any idea how many log lines per FlowFile? A suggested dataflow may look like this: The SplitText processor is used to breakup your incoming log files in to many smaller FlowFiles that can more easily be handled by the RouteText processor without running out of heap memory. This is done by setting the line split count property. Depending on how much heap you have configured for your NiFi and size size of each log line really determines how many logs line you can have per split FlowFile. The RouteText processor evaluates the entire FlowFiles content and routes groups of logs lines to a "dt" relationship: The UpdateAttribute processor (Optional) will create a "dt" attribute from the "RouteText.Group" attribute. YOu can use thsi attribute later to define the Hive partition table: The MergeContent processor (Optional) is used to combine FlowFiles with matching values (dates) in the "RouteText.Group" attribute back in to a single FlowFile. Don't forget to set the number of entries and max bin age properties to maximize this processors usage. Route the "Merged" relationship from this processor to your Hive based processor. Thanks, Matt ... View more