About sonnychee

MattWho · ‎01-25-2021

@vchhipa If you are forced to "terminate" the processor after requesting it to stop, this can mean that you have a stuck or very long running process thread. The "terminate" does not actually kill the thread, it disassociates that JVM thread from the processor and he current FlowFile that thread is associated with. The terminated thread continues to run until it completes (NiFi does request thread to quit/exit when terminating it, but success of that varies since not all process support that ability). Any terminated threads still active will be represented in the processor by a small number in parentheses (1) displayed in its upper right corner. The previously associated FlowFile is left on the inbound connection and will be handled based on queue priority when the consuming processor is started again. If the "terminated" thread should eventually complete, any output/return from that thread including logging is just sent to null. To figure out what is going on when you have a seemingly hung thread is to get a series of NiFi thread dumps: ./nifi.sh dump <dump-filename-nodenum-01.txt> Getting at least 3 dumps at an interval of 2-5 minutes apart is usually good. What you are looking for is a thread associated with your processor class (invokeHTTP in this case) where the same thread number exists in every thread dump collected. Then you will want to luck that the thread stack to see if all are identical. If thread output changes between thread dump outputs, it indicates that thread is not hung but rather just long running. If thread dump output does not change, you'll want to dig in to that output to see what it is waiting on. Hope this helps, Matt

Shu_ashu · ‎06-20-2018

@Sonny Chee As a temporary work around to fix this issue please look into this link.

mallikarjun_r_t · ‎12-06-2018

Is there a way (without using execute script) to modify the processor_group variable in processor?

lba · ‎09-18-2018

@Kiran Nittala, @Sonny Chee, What exactly would I need to name the dynamic property to have its content sent as the HTTP status message? Thanks, Lars

MattWho · ‎05-15-2018

@Sonny Chee It is likely the flow is becoming out of sync as a result of one of the following scenarios: - 1. A change replication request was made to all nodes in the cluster. One or more nodes failed to process that request in the configured allowable nifi.cluster.node.connection.timeout and/or nifi.cluster.node.read.timeout. By default these values are set very low. Recommend setting these at a minimum of 30 secs. Additional reasons why a replication request may fail can range from network issues (high latency or packet loss) to the size of the replication request (snippets) (pasting/copying large selection of canvas components or instantiating a large template. Each node must complete adding every component, connection, controller service, etc in that snippet before responding to the replication request) - 2. A node was in a disconnected state. Every cluster has a cluster coordinator which every node sends heartbeats to. If a node is disconnected from the cluster (through user manual action, or because request failure above), that node's UI cab still be accessed. It will display the work "disconnected" on status bar, but will still allow a user who is connected directly to that node's UI to make changes. So if an attempt is made later (after making some change) to reconnect this node to the cluster, it will fail because flows do not match between cluster and this node anymore. User's who are connected to anyone of the other nodes still in the cluster will see a missing node in the status bar (for example: 1/2). From these node's UIs changes will not be allowed because node is missing. - It is best to look at nifi-app.log or user the cluster UI in NiFi to see why a node was disconnected in the first place. - Users should be educated on how NiFi works as described above and should avoid making changes to canvas when "disconnected" is displayed on NiFi UI status bar above canvas. - NiFi allows changes to be made on disconnected nodes by design. The intent here is to allow users to disconnect a node to perform troubleshooting of a misbehaving node or to perform some temporary one off testing that they do not want to perform across entire cluster. IN such cases, those changes must be backed out before being able to rejoin cluster. Another option is to rename/remove flow.xml.gz on that disconnected node. On restart, in the absence of a flow.xml.gz file, the connecting node will inherit the flow.xml.gz from the cluster. - Thanks, Matt

MattWho · ‎05-15-2018

@Sonny Chee It is likely the flow is becoming out of sync as a result of one of the following scenarios: - 1. A change replication request was made to all nodes in the cluster. One or more nodes failed to process that request in the configured allowable nifi.cluster.node.connection.timeout and/or nifi.cluster.node.read.timeout. By default these values are set very low. Recommend setting these at a minimum of 30 secs. Additional reasons why a replication request may fail can range from network issues (high latency or packet loss) to the size of the replication request (snippets) (pasting/copying large selection of canvas components or instantiating a large template. Each node must complete adding every component, connection, controller service, etc in that snippet before responding to the replication request) - 2. A node was in a disconnected state. Every cluster has a cluster coordinator which every node sends heartbeats to. If a node is disconnected from the cluster (through user manual action, or because request failure above), that node's UI cab still be accessed. It will display the work "disconnected" on status bar, but will still allow a user who is connected directly to that node's UI to make changes. So if an attempt is made later (after making some change) to reconnect this node to the cluster, it will fail because flows do not match between cluster and this node anymore. User's who are connected to anyone of the other nodes still in the cluster will see a missing node in the status bar (for example: 1/2). From these node's UIs changes will not be allowed because node is missing. - It is best to look at nifi-app.log or user the cluster UI in NiFi to see why a node was disconnected in the first place. - Users should be educated on how NiFi works as described above and should avoid making changes to canvas when "disconnected" is displayed on NiFi UI status bar above canvas. - NiFi allows changes to be made on disconnected nodes by design. The intent here is to allow users to disconnect a node to perform troubleshooting of a misbehaving node or to perform some temporary one off testing that they do not want to perform across entire cluster. IN such cases, those changes must be backed out before being able to rejoin cluster. Another option is to rename/remove flow.xml.gz on that disconnected node. On restart, in the absence of a flow.xml.gz file, the connecting node will inherit the flow.xml.gz from the cluster. - Thanks, Matt

sonnychee · ‎04-02-2018

Thank you for the clarification.

Shu_ashu · ‎03-11-2018

@Sonny Chee If you are trying to replace the value then you can do that by using Replace Text processor instead of ExtractText processor. Extract text processor extracts the contents of flowfile for the matching regex and keep the extracted contents as the attributes of the flowfile. Example:- If you are trying to find all matches for ${key_name}\s*:(\s*.+\n) in the flowfile and then replace the matches with ${key_value) Then replacetext processor configs:-

MattWho · ‎03-06-2018

@Sonny Chee I am assuming you are referring to the H2 Binary DB file nifi-flow-audit.h2.db? In that case, properties coded as sensitive are not persisted to the H2 database. NiFi writes literally " ******** " to the H2 DB for these sensitive property values. Thanks, Matt

Online	Offline
Last Visited	‎11-30-2018 09:19 PM

Date Registered	‎03-06-2018 02:33 AM
Last Visited	‎11-30-2018 09:19 PM
Total Messages Posted	18
Total Kudos Received	1

Re: Flowfiles stuck in queue to InvokeHttp...

Re: NiFi SelectHiveQL - flowfile attributes not co...

Re: NiFi: Is it possible to access Processor Group...

Re: Nifi: Setting Status Message for HandleHttpRes...

Re: Flow.xml.gz appears to be getting out of sync

Re: Flow.xml.gz appears to be getting out of sync

Re: Nifi JVM Memory Utilization Continually Rising...

Re: Interpolating attribute into regular expressio...

Re: NIFI Sensitive Property doesn't appear to work...