1973
Posts
1225
Kudos Received
124
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
820 | 04-03-2024 06:39 AM | |
1602 | 01-12-2024 08:19 AM | |
792 | 12-07-2023 01:49 PM | |
1364 | 08-02-2023 07:30 AM | |
1991 | 03-29-2023 01:22 PM |
02-22-2021
06:51 AM
Is this one node of NiFi or a cluster? remember NiFi is case sensitive and windows could have case issues on file names. you need to put a minimum file age and size on there so it doesn't try to process something before it's done being written to
... View more
02-22-2021
06:49 AM
Command path: C:\Temp\activate.bat get rid of the rest
... View more
02-22-2021
06:46 AM
Don't use Map Reduce anymore. It's time to move to Spark or Flink.
... View more
02-22-2021
06:44 AM
1 Kudo
We don't support Mosquitto. That is an open source Eclipse project https://mosquitto.org/ You can log to topics https://mosquitto.org/man/mosquitto-conf-5.html https://www.datainmotion.dev/2019/12/iot-series-minifi-agent-on-raspberry-pi.html https://community.cloudera.com/t5/Community-Articles/MQTT-with-Apache-NiFi/ta-p/248016
... View more
02-22-2021
06:41 AM
Flume was deprecated. https://www.datainmotion.dev/2019/08/migrating-apache-flume-flows-to-apache.html
... View more
02-18-2021
08:01 AM
The name of the server must be set, it can't be localhost. NiFi and Hive are on other machines.
... View more
02-17-2021
08:47 AM
https://www.datainmotion.dev/2020/10/running-flink-sql-against-kafka-using.html https://www.datainmotion.dev/2020/08/deleting-schemas-from-cloudera-schema.html https://www.cloudera.com/tutorials/schema-registry-in-trucking-iot/3.html
... View more
02-11-2021
01:19 AM
Thanks Spann for useful links and information about python-schema-registry-client library. So now I can see one of possible way to connect to kerberos Schema registry URL through python is requests_kerberos library. Thanks and Regards
... View more
02-10-2021
09:03 AM
3 Kudos
@Jarinek The process really depends on what update you are trying to make. 1. You can not remove a connection that has queued FlowFiles in it, but you can redirect it to a different target processor with queued data. 2. You can not redirect a connection if the processor it is currently attached to still has a running thread. Stopping a processor does not kill threads, it simply tells the processor to not execute again at the configured run schedule. Existing threads will continue to run until they complete. Until all threads exit, the processor is still in a state of "stopping" even though UI reflect red square for "stopped". 3. You cannot modify a processor if is still has running threads (see note about "stopping" processors above) 4. If you stop the component that is on the receiving side of a connection, any FlowFiles queued on that connection, not tied to any active thread still running on target processor component, will not be processed and remain queued on the connection. You can manual empty a queue through a rest-api call (means data loss), but that is not necessary if you are not deleting the connection. Attempts to perform configuration changes when components still have active threads or are in a running state will result in an exception being thrown and the change not happening. Attempts remove connections that have queued FlowFiles will throw an exception and block removal. Now if all you are trying to do is modify some configuration on a processor, all you need to do is stop the processor, check that it has no active threads, make the config change, and then start the processor again. Not sure wha you are asking with "update the flow ignoring any data in failure or error connection queues". NiFi does not ignore queued FlowFiles. It also not wise to leave connection with queued FlowFiles just sitting around your dataflows. Those old queued FlowFile will prevent removal or content claims that contain that FlowFiles data. Since a content claim can contain the data from 1 to many FlowFiles, this can result in your content repository filling up. NiFi can only remove content claims which have no FlowFiles pointing to them anymore. Here are some useful links: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nipyapi.html https://github.com/Chaffelson/nipyapi http://nifi.apache.org/docs/nifi-docs/rest-api/index.html https://community.cloudera.com/t5/Community-Articles/Update-NiFi-Connection-Destination-via-REST-API/ta-p/244211 https://community.cloudera.com/t5/Community-Articles/Change-NiFi-Flow-Using-Rest-API-Part-1/ta-p/244631 Hope this helps, Matt
... View more
02-09-2021
10:41 AM
Here is a cool NiFi websocket app https://www.datainmotion.dev/2020/12/ingesting-websocket-data-for-live-stock.html Hosting web apps in NiFi https://www.datainmotion.dev/2020/11/flank-smart-weather-websocket.html https://www.datainmotion.dev/2020/12/ingesting-websocket-data-for-live-stock.html
... View more