1973
Posts
1225
Kudos Received
124
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1984 | 04-03-2024 06:39 AM | |
| 3134 | 01-12-2024 08:19 AM | |
| 1707 | 12-07-2023 01:49 PM | |
| 2485 | 08-02-2023 07:30 AM | |
| 3481 | 03-29-2023 01:22 PM |
02-10-2021
09:03 AM
3 Kudos
@Jarinek The process really depends on what update you are trying to make. 1. You can not remove a connection that has queued FlowFiles in it, but you can redirect it to a different target processor with queued data. 2. You can not redirect a connection if the processor it is currently attached to still has a running thread. Stopping a processor does not kill threads, it simply tells the processor to not execute again at the configured run schedule. Existing threads will continue to run until they complete. Until all threads exit, the processor is still in a state of "stopping" even though UI reflect red square for "stopped". 3. You cannot modify a processor if is still has running threads (see note about "stopping" processors above) 4. If you stop the component that is on the receiving side of a connection, any FlowFiles queued on that connection, not tied to any active thread still running on target processor component, will not be processed and remain queued on the connection. You can manual empty a queue through a rest-api call (means data loss), but that is not necessary if you are not deleting the connection. Attempts to perform configuration changes when components still have active threads or are in a running state will result in an exception being thrown and the change not happening. Attempts remove connections that have queued FlowFiles will throw an exception and block removal. Now if all you are trying to do is modify some configuration on a processor, all you need to do is stop the processor, check that it has no active threads, make the config change, and then start the processor again. Not sure wha you are asking with "update the flow ignoring any data in failure or error connection queues". NiFi does not ignore queued FlowFiles. It also not wise to leave connection with queued FlowFiles just sitting around your dataflows. Those old queued FlowFile will prevent removal or content claims that contain that FlowFiles data. Since a content claim can contain the data from 1 to many FlowFiles, this can result in your content repository filling up. NiFi can only remove content claims which have no FlowFiles pointing to them anymore. Here are some useful links: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nipyapi.html https://github.com/Chaffelson/nipyapi http://nifi.apache.org/docs/nifi-docs/rest-api/index.html https://community.cloudera.com/t5/Community-Articles/Update-NiFi-Connection-Destination-via-REST-API/ta-p/244211 https://community.cloudera.com/t5/Community-Articles/Change-NiFi-Flow-Using-Rest-API-Part-1/ta-p/244631 Hope this helps, Matt
... View more
02-09-2021
10:41 AM
Here is a cool NiFi websocket app https://www.datainmotion.dev/2020/12/ingesting-websocket-data-for-live-stock.html Hosting web apps in NiFi https://www.datainmotion.dev/2020/11/flank-smart-weather-websocket.html https://www.datainmotion.dev/2020/12/ingesting-websocket-data-for-live-stock.html
... View more
02-08-2021
07:49 AM
1 Kudo
Grok is Grok. Find Grok expressions that work for you and use a grok tester https://stackoverflow.com/questions/38462630/logstash-grok-filter-key-value-pairs http://grokconstructor.appspot.com/do/match#result https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-record-serialization-services-nar/1.5.0/org.apache.nifi.grok.GrokReader/additionalDetails.html
... View more
01-28-2021
07:34 AM
QueryRecord is the way to go. You can compare with a sql SELECT * FROM FLOWFILE WHERE timestamp > ${event_time} and value = ${id}
... View more
01-26-2021
11:40 AM
Automating Starting Services in Apache NiFi and Applying Parameters Automate all the things! You can call these commands interactively or script all of them with awesome DevOps tools. @Andre Araujo and @dchaffey can tell you more about that. Enable All NiFi Services on the Canvas By running this three times, I get any stubborn ones or ones that needed something previously running. This could be put into a loop; check the status before trying again. nifi pg-list
nifi pg-status
nifi pg-get-services The NiFi CLI has interactive help available and also some good documentation: NiFi CLI Toolkit Guide /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root
/opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root
/opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root We could then start a process group if we wanted: nifi pg-start -u http://edge2ai-1.dim.local:8080 -pgid 2c1860b3-7f21-36f4-a0b8-b415c652fc62 List all process groups /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-list -u http://edge2ai-1.dim.local:8080 List Parameters /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi list-param-contexts -u http://edge2ai-1.dim.local:8080 -verbose Set parameters to set parameter context for a process group; you can loop to do all. pgid => parameter group id pcid => parameter context id I need to put this in a shell or Python script: /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-set-param-context -u http://edge2ai-1.dim.local:8080 -verbose -pgid 2c1860b3-7f21-36f4-a0b8-b415c652fc62 -pcid 39f0f296-0177-1000-ffff-ffffdccb6d90 Example setupnifi.sh (Github Link) You could also use the NiFi REST API or Dan's awesome Python API NiPyApi: A Python Client SDK for Apache NiFi References DevOps: Working with Parameter Contexts NiFi Toolkit CLI No More Spaghetti Flows Report on this Apache NiFi Everything Apache Nifi Cloudera Data Platform - Using Apache NiFi REST API in the Public Cloud Using NiFi CLI to Restore NiFi Flows From Backups Automating the Building, Migration, Backup, Restore and Testing of Streaming Applications Apache NiFi Toolkit Guide An overview of Apache NiFi and Toolkit CLI deployments Automate workflow deployment in Apache NiFi with the NiFi Registry DevOps for Apache NiFi 1.7 and More
... View more
Labels:
01-26-2021
08:51 AM
https://nifi.apache.org/docs/nifi-docs/components/nifi-docs/components/org.apache.nifi/nifi-gcp-nar/1.9.0/org.apache.nifi.processors.gcp.bigquery.PutBigQueryBatch/index.html With this processor, you can do Batch loads flow files content to a Google BigQuery table.
... View more
01-21-2021
11:44 AM
https://nipyapi.readthedocs.io/en/latest/ Can build flows with Python Code. Or you can write a custom nifi processor https://www.nifi.dev/2019/03/custom-processors.html
... View more
01-21-2021
07:08 AM
Hi timothy, will you please help me with this question
... View more
01-15-2021
06:32 AM
1 Kudo
Install Cloudera Data Platform Public Cloud Azure edition and there is a one button click to run a NiFi cluster as a datahub. Does all setup, configuration, optimization and security. https://docs.cloudera.com/cdf-datahub/7.2.1/nifi-azure-ingest/topics/cdf-datahub-fm-adls-ingest-overview.html
... View more
01-13-2021
07:10 AM
Here are a few examples of moving Flume flows to NiFi. https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache.html
... View more