@Green_ Thank you so much ! It worked as expected. ... View more

@hegdemahendra  This is not something I have tried before, but... When you execute the nifi.sh script to start it bootstrap the NiFi process via the configuration in the bootstrap.conf NiFi configuration file.  It is during the bootstrap process that NiFi starts the main child process that loads NiFi.  Perhaps you can add additional java.args to handle your pre NiFi needs?   Or maybe modify the the nifi.sh script itself so that is executes your requirements before calling the rest of the NiFi startup process.  If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

Hi @krishna123   Please try changing the port, you won't face the same issue. ... View more

@hegdemahendra ,   You need to set the time characteristic of the stream for it to work. For example, try setting it to processing time, as shown below: DataStream<String> matechedStream = patternStream .inProcessingTime() .process(new PatternProcessFunction<String, String>() { @Override public void processMatch(Map<String, List<String>> map, Context context, Collector<String> collector) throws Exception { collector.collect(map.get("start").toString()); } } );   Cheers, André ... View more

@MattWho - We have separated out ebs volumes of each repos and also 3 ebs volumes each for content and provenance repos. Now looks like issue is pretty much resolved ! Thanks for all your suggestions, helped a lot   Thanks Mahendra ... View more

@hegdemahendra  1. Do you see any logging related to the content_repository.  Perhaps something related to NiFi not allowing writes to the content repository waiting on archive clean-up? 2. Is any outbound connection from the handleHTTPRequest processor red at the time of the pause?  This indicates backpressure is being applied which would stop the source processor from being scheduled until back pressure ends. 3. How large is your Timer Driven thread pool?  This is the pool of threads from which the scheduled components can use.  If it is set to 10 and and all are currently in use by components, the HandleHTTPRequest processor , while scheduled, will be waiting for a free thread from that pool before it can execute.  Adjusting the "Max Timer Driven Thread pool" requires careful consideration of average CPU load average across on every node in your NiFi cluster since same value is applied to each node separately.  General starting pool size should be 2 to 4 times the number of cores on a single node.  Form there you monitor CPU load average across all nodes and use the one with the highest CPU load average to determine if you can add more threads to that pool.  If you have a single node that is always has a much higher CPU load average, you should take a closer look at that server.  Does it have other service running on it tat are not running on other nodes?  Does it unproportionately consistently have more FlowFiles then any other node (This typically is a result of dataflow design and not handling FlowFile load balancing redistribution optimally.)? 4. How many concurrent tasks on your HandleHttpRequest processor.  The concurrent tasks are responsible for obtaining threads (1 per concurrent task if available) to read data from the Container queue and create the FlowFiles.  Perhaps the request come in so fast that there are not enough available threads to keep the container queue from filling and thus blocking new requests.  Hope the above helps you get to the root of your issue. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt ... View more