Hi @krishna123   Please try changing the port, you won't face the same issue. ... View more

@hegdemahendra ,   You need to set the time characteristic of the stream for it to work. For example, try setting it to processing time, as shown below: DataStream<String> matechedStream = patternStream .inProcessingTime() .process(new PatternProcessFunction<String, String>() { @Override public void processMatch(Map<String, List<String>> map, Context context, Collector<String> collector) throws Exception { collector.collect(map.get("start").toString()); } } );   Cheers, André ... View more

@MattWho - We have separated out ebs volumes of each repos and also 3 ebs volumes each for content and provenance repos. Now looks like issue is pretty much resolved ! Thanks for all your suggestions, helped a lot   Thanks Mahendra ... View more

@hegdemahendra  1. Do you see any logging related to the content_repository.  Perhaps something related to NiFi not allowing writes to the content repository waiting on archive clean-up? 2. Is any outbound connection from the handleHTTPRequest processor red at the time of the pause?  This indicates backpressure is being applied which would stop the source processor from being scheduled until back pressure ends. 3. How large is your Timer Driven thread pool?  This is the pool of threads from which the scheduled components can use.  If it is set to 10 and and all are currently in use by components, the HandleHTTPRequest processor , while scheduled, will be waiting for a free thread from that pool before it can execute.  Adjusting the "Max Timer Driven Thread pool" requires careful consideration of average CPU load average across on every node in your NiFi cluster since same value is applied to each node separately.  General starting pool size should be 2 to 4 times the number of cores on a single node.  Form there you monitor CPU load average across all nodes and use the one with the highest CPU load average to determine if you can add more threads to that pool.  If you have a single node that is always has a much higher CPU load average, you should take a closer look at that server.  Does it have other service running on it tat are not running on other nodes?  Does it unproportionately consistently have more FlowFiles then any other node (This typically is a result of dataflow design and not handling FlowFile load balancing redistribution optimally.)? 4. How many concurrent tasks on your HandleHttpRequest processor.  The concurrent tasks are responsible for obtaining threads (1 per concurrent task if available) to read data from the Container queue and create the FlowFiles.  Perhaps the request come in so fast that there are not enough available threads to keep the container queue from filling and thus blocking new requests.  Hope the above helps you get to the root of your issue. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more

@hegdemahendra Thank you for your reply.. It will be a continuous migration that will run for few months... Once all the historic data is migrated, it will be a CDC. The no of tables could be close to 20-30 tables.. I dont know yet about the volume but it will be huge...  Based on your response, the solution looks like the combined approach for multiple tables and CDC. Could you please share little more details about this approach? ... View more