About Wynner

Wynner · ‎06-14-2017

@mayki wogno Was does the configuration of the GetFTP processor look like?

Wynner · ‎06-14-2017

@Kiem Nguyen The configuration is to have the same number of concurrent tasks and partitions. So, with 4 partitions on the topics, you want 4 concurrent tasks. Since you have a 3 node cluster, configure your PublishKafka and Consume_Kafka processors with 2 concurrent tasks and you should be good. For an ideal situation, it would be better if they matched exactly. So, if possible, I would configure the Kafka topics with 6 partitions, or some multiple of three.

Wynner · ‎06-13-2017

@Kiem Nguyen On the Consumer_Kafka_0_10 processors, configure them with 2 concurrent tasks, and see if that resolves the issue.

Wynner · ‎06-09-2017

@Kiem Nguyen How many partitions are on the Kafka topic?

Wynner · ‎06-05-2017

@rian amrodin If I understand your question, you want to route files larger than 4MB down a different path from the files smaller than 4MB. In the RouteOnAttribute processor, add a property like: Files greater than 4MB will be routed out the gt4mb realtionship and the files that are smaller than 4MB will go out the unmatched relationship.

Wynner · ‎05-24-2017

@Tinkle Mahendru Have you tried using the MergeContent processor?

Wynner · ‎05-22-2017

@Thierry Vernhet Try putting a larger range between the minimum and maximum group size, like 25 MB and 50 MB. How much memory have you allocated to the NiFi JVM? The default is 512MB, it is set in the bootsrrap.conf file. # JVM memory settings java.arg.2=-Xms512m, change to 2g or 4g if you have the memory available on your system java.arg.3=-Xmx512m, change to 2g or 4g if you have the memory available on your system Since you are dealing with 1.3 GB in the MergeContent processor, make sure to at least allocate double that for the NiFi JVM, because the MergeContent processor uses the JVM memory to build it merged flow files. In addition I would set the number of Concurrent Tasks to 3.

Wynner · ‎05-22-2017

@Thierry Vernhet How are the properties set in the MergeContent processor?

Wynner · ‎05-19-2017

@Prakash Ravi There are only three processors which require a Distributed map Cache server: DetectDuplicate, FetchDistributedMapCache and PutDistributedMapCache. The rest will use zookeeper were applicable. To clear the state of a processor, just do the following steps Right click on a processor, select View state from the menu Then just click Clear state and the files will be listed again.

Wynner · ‎05-19-2017

@Thierry Vernhet So, you are saying it works with Batch Size set, but it does not matter what you set the value to?

Online	Offline
Last Visited	‎10-31-2025 01:29 PM

Member Since	‎07-30-2019 10:40 AM
Last Visited	‎10-31-2025 01:29 PM
Posts	944
Kudos received	197

Cloudera Community

Re: Authentication issue while connecting Nifi to ...

Re: Execute Stream Command - NIFI

Re: How to initialize a processor via NIFI API

Re: HDF cluster in PROD managed by Ambari, did a c...

Re: consuming from kafka topics using NiFi consume...

Re: [NIFI 1.3.0] getFTP : Read timed out

Re: Why consumerKafka_0_10 Processor receives flow...

Re: Why consumerKafka_0_10 Processor receives flow...

Re: Why consumerKafka_0_10 Processor receives flow...

Re: how to filter file based on size file..??

Re: combine two files in nifi

Re: OutofMemory with GetmongoDB

Re: OutofMemory with GetmongoDB

Re: Clarifications on state management within NiFi...

Re: OutofMemory with GetmongoDB