Hi @rajdip chaudhuri Have you considered NiFi? you have out of the box processors to list/fetch files and to write to HDFS. You can also use a NiFi cluster if you want to distribute the load on several nodes. ... View more

Hi @Jayendra Patil Setting the optimal value of max thread count depends on your use cases and what processors you are using (CPU intensive like convert processor or IO intensive like the put/get processors). I've seen better usage of my hardware by having thread count around 2x number of cores. I've seen some cluster with 3x number of cores. I think you can go beyond 50 in your case and monitor the behavior. The best thing to do is to proceed in an incremental manner. I hope this helps. Abdelkrim ... View more

Hi @Karl Fredrickson If you have Knox you can use it to encapsulate Kerberos authentication and use username/password. Thanks ... View more

Hi @David Manukian This seems to be a virtualbox network configuration issue. Check your port forwarding and add a rule for 8080. ... View more

Hi @Eric Lloyd I am not sure I understand your use case. NiFi tails a local file. From your question, it looks like you are trying to tail the same fail when master switch. Is your file visible to both nodes (such as NAS storage) ? TailFile saves it's state to avoid duplicating data from one file. There's two option to store the state : local and remote. Have you set "state location" to remote ? As per the doc : Specifies where the state is located either local or cluster so that state can be stored appropriately in order to ensure that all data is consumed without duplicating data upon restart of NiFi ... View more

Hi @spdvnz Your input and output CSV schema for the LookupRecord should be different. In the output schema you should add a field 'Company' that the processor will populate. Take a look at this example where I added the field city in the output schema : https://medium.com/@abdelkrim.hadjidj/data-flow-enrichment-with-apache-nifi-d221f1dde419 ... View more

Hi @Cesar Rodrigues The cleaniest way should be to use ConvertRecord processor with a CSVReader (using Delimiter as pipe) and JSonSetRecordWriter. This directly convert your CSV into JSON without passing by attributes. Using Record processors also gives you better performance. Thanks ... View more

Thanks @David Doran If you found that this answer addressed your question, please take a moment to click "Accept" below. ... View more