About MattWho

MattWho · ‎12-11-2018

@Puneet Nagpal - Running a NiFi Expression Language evaluator against the returned value from an existing Attribute can become dangerous/complicated. - Since an EL expression starts with "${" the evaluator would interpret that string as the start of an EL statement. That means that a value that contains that string would be interpreted as an EL statement and throw an exception. Another example may be where the full EL statement structure exists ${<some string>} which is not intended to be an EL statement. If the evaluator ran against that and the <some string> did not match it would return null and value would get changed. - So using the updateAttribute to set a your new filename is going to be best option. filename --> enrollments_${now():format('yyyy_MM')}.csv - Thanks. Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

MattWho · ‎12-10-2018

@SP - A couple of things that can be done to troubleshoot this. 1. Did you inspect both the nifi-app.log and nifi-bootstrap.log for any ERROR or WARN messages during NiFi start? 2. After Staring NiFi and waiting at least 5 minutes, take a series of 3 thread dumps (3-5 minutes apart) that can be inspected for any threads that may not be progressing (same thread ID in same state in every thread dump). NiFi will validate the components as it loads the flow from the flow.xml.gz. If you are using any type of mounted NAS and that mount is in a bad state and your new flow is trying to access that mount it may cause the validation thread to hang waiting on response from that NAS. 3. Perform a diff between working flow.xml.gz and non working flow.xml.gz to see what kind of components are i use in new flow and how they have been configured. - Thank you, Matt

MattWho · ‎12-10-2018

*** Community Forum Tip: Try to avoid starting a new answer in response to an existing answer. Instead use comments to respond to existing answers. There is no guaranteed order to different answer which can make it hard following a discussion

MattWho · ‎12-07-2018

@Nicolas Osorio . You are correct. I wrote this article some time ago. The use of the less secure aes128 is not going to be accepted by Newer versions of browsers and NiFi. Switching to a more secure aes256 will resolve issue.

MattWho · ‎12-05-2018

@Robert Papel I would suggest running your GetSFTP processor while leaving the RouteOnAttribute processor stopped log enough to get a some FlowFiles queued in the connection between those processors. Then right click on that queued connection and select "list queue" from menu that appears. From the list queue UI select a few different FlowFiles and look at the values assigned to the various attributes. Does filename show as being "<some value>.csv" or just "<some value>"? If filename does not include extension, perhaps try using the IdentifyMimeType processor to see if it properly sets a "mime.type" attribute on each FlowFile which could then be used by your RouteOnAttribute processor. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

MattWho · ‎12-03-2018

@Gillu Varghese A few questions: 1. Are you sure all 136 files are reaching the MergeContent processor's inbound connection within 5 minutes? The bin age starts when very first FlowFile is added to a bin. At 5 minutes from that point the bin will be merged even if not all 136 have arrived. 2. Is your NiFi a cluster or standalone instance of NiFi? If cluster, are all 136 FlowFiles on same NiFi node? Each node in a cluster can only merge FlowFiles residing on same Node. There is a new load balanced connection feature in NiFi 1.8 that can help here if this is the case. https://blogs.apache.org/nifi/entry/load-balancing-across-the-cluster - Try setting your max bin age to a much higher value and see what results you see. - Thank you, Matt

MattWho · ‎11-27-2018

@Gillu Varghese - I am assuming you mean GetFile instead of PutFile? - How is your GetFile scheduled to run? Is it always going to get exactly 136 FlowFiles? As far as scheduling ingest, are all 136 files expected to arrive at same time each day? You can schedule the GetFile using a Cron that runs daily. - As far as MergeContent configuration goes, you would configure both the minimum and maximum number if Entries to 136. Each entry is a FlowFile. I would also set your Max Bin age to some value that allows sufficient time from ingest of FlowFiles 1 through 136. The Max bin age serves as your exit strategy should not all 136 FLowFiles get ingested in the desired max bin age time frame. The merged flowfile produced will have a new FlowFile attributes generated on it named "merge.count" that reports the number of FlowFiles in the bundle. After your mergeContent processor I would add a RouteOnAttribute processor that is configured to verify the the value assigned to the "merge.content" FlowFile attribute is actually the expected 136 (for example '${merge.count:equals('136')}' ). If it is not then the bundle can be routed down a different path for error handling. - - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

MattWho · ‎11-27-2018

@alfredo reza Upon successful authentication (via openID in your case), a JWT token is issued for the successfully authenticated user and returned to the client (your browser in this case). A matching server side token is stored locally on the node that serviced the login request. The login UI then redirects to the NiFi UI. In subsequent requests to same browser address (https://nifi.dev-bristol.nec-ccoc.com) that issued token is provided to verify user identity. In your case, it would appear that follow on requests are not being routed to same NiFi cluster node which issued the original token. Since tokens are only valid for the node which issued them, the attempt to use same token to authenticate in to other nodes will fail. The failure will trigger a redirect to the login page again. - The load-balancer should be configured with sticky sessions to avoid these subsequent requests from going to random nodes in the NiFi cluster. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

MattWho · ‎11-26-2018

@Jose Paul - The Defragment strategy will bin FlowFiles based on the fragment.identifier attributes set on the FLowFiles by the split processor in your flow. - A bin will only be merged when utilizing this strategy if one of the following occurs: 1. all the fragments for a give fragment.identifier are currently in the bin. 2. max bin age has been reached. Max bin age property works like exit strategy in the event a bin never contains all required pieces. such as 1 or more fragments never arriving at processor. It will force a merge or failure (depending on strategy in use) 3. Not enough free bins. For example you have 5 bins yet your incoming connections contains more then 5 unique fragment identifier values. This will force the oldest bin to merge or route to failure to free a bin for the next fragment identifier. Strategy for bin number should be number of unique values +1. You always want to have at least 1 free bin. - Thank you, Matt

MattWho · ‎11-26-2018

@Mikel Emaldi - The particular error points at a very specific processor still on the canvas that is trying to process a FlowFile. Search your canvas for the component uuid in the error log message. In above error that uuid is 3ad5bd27-0167-1000-0000-0000256eb90d - NiFi will not allow you to delete a connection that still contains FlowFiles. So the particular dataflow that you have deleted was not the dataflow containing this particular processor because it would have had an inbound connection with this FlowFile still present on it. - Thank you, Matt - If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.

Online	Offline
Last Visited	‎12-23-2025 12:36 AM

Member Since	‎07-30-2019 10:41 AM
Last Visited	‎12-23-2025 12:36 AM
Posts	3,406
Kudos received	1618

Cloudera Community

Re: Error importing NiFi workflow template from ve...

Re: Error importing NiFi workflow template from ve...

Re: How to elevate a default nifi user to admin - ...

Re: NiFi EnvokeHTTP - putting current date on HTTP...

Re: Invoking Nifi rest api in Data Flow

Re: Evaluate attribute value containing expression...

Re: NiFi UI is not loading template after login.

Re: NiFi UI is not loading template after login.

Re: How to create user generated keys for securing...

Re: NiFi: Routing on File Types, e.g. csv, tsv, xl...

Re: mergecontent processor in nifi

Re: mergecontent processor in nifi

Re: NIFI Unable to validate the access token. Acce...

Re: nifi: how to handle with mergeContent ...

Re: Deleted NiFi workflow still running