Member since
07-21-2020
29
Posts
5
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
413 | 03-21-2022 05:11 AM | |
2622 | 04-22-2021 12:21 PM | |
971 | 02-08-2021 11:44 PM |
08-06-2022
10:28 AM
1 Kudo
@Jarinek Please check my inline answer to your queries: 1) is it possible to alter compression.type on a kafka topic while having applications on the topic running? >> Yes, we would recommend stopping the producer and then alert compression type as whenever the Kafka broker gets a compressed batch of messages from a producer. It will decompress the data to validate if the compression type is different at the topic level and producer level and if compression is the same from both sides, then it will write directly to the topic/leader partition. 2) What should be expected hard drive saving factor for text records like json (2KB size) and compression.type=gzip. >> You should use and test snappy or lz4 for best speed/compression ratio 3) Does it apply to KafkaMirrorMaker only? >> Yes, as per the screenshot it will apply for the MM only. 4) Are producer and consumer applications somehow affected, when the global parameter changes. >> If you are changing it from the Kafka side, then it should not affect the applications. If you found this response assisted with your query, please take a moment to log in and click on KUDOS 🙂 & ”Accept as Solution" below this post. Thank you.
... View more
03-21-2022
05:11 AM
I have eventually solved the issue. The Writter property "Schema Access Strategy" was misconfigured. It must be set to "Use 'Schema Text' Property" to really apply the given schema .
... View more
03-03-2022
04:27 PM
@Jarinek , This is total volume, in bytes, received from the network by all the brokers across the cluster. Replication traffic is counted by this counter as well. Cheers, André -- Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
... View more
02-22-2022
11:23 AM
@Jarinek Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. If you are still experiencing the issue, can you provide the information @steven-matison has requested?
... View more
12-02-2021
09:11 PM
@Jarinek, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
10-21-2021
10:44 AM
@Jarinek, Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
05-24-2021
05:47 AM
Hi @Jarinek, Did the reply from @mridley resolve your issue? If so, please mark the reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
04-15-2021
06:28 AM
check disk space http://apache-nifi-users-list.2361937.n4.nabble.com/Clarifications-on-getting-flowfiles-using-FlowFileFilters-td7333.html
... View more
02-10-2021
09:03 AM
3 Kudos
@Jarinek The process really depends on what update you are trying to make. 1. You can not remove a connection that has queued FlowFiles in it, but you can redirect it to a different target processor with queued data. 2. You can not redirect a connection if the processor it is currently attached to still has a running thread. Stopping a processor does not kill threads, it simply tells the processor to not execute again at the configured run schedule. Existing threads will continue to run until they complete. Until all threads exit, the processor is still in a state of "stopping" even though UI reflect red square for "stopped". 3. You cannot modify a processor if is still has running threads (see note about "stopping" processors above) 4. If you stop the component that is on the receiving side of a connection, any FlowFiles queued on that connection, not tied to any active thread still running on target processor component, will not be processed and remain queued on the connection. You can manual empty a queue through a rest-api call (means data loss), but that is not necessary if you are not deleting the connection. Attempts to perform configuration changes when components still have active threads or are in a running state will result in an exception being thrown and the change not happening. Attempts remove connections that have queued FlowFiles will throw an exception and block removal. Now if all you are trying to do is modify some configuration on a processor, all you need to do is stop the processor, check that it has no active threads, make the config change, and then start the processor again. Not sure wha you are asking with " update the flow ignoring any data in failure or error connection queues". NiFi does not ignore queued FlowFiles. It also not wise to leave connection with queued FlowFiles just sitting around your dataflows. Those old queued FlowFile will prevent removal or content claims that contain that FlowFiles data. Since a content claim can contain the data from 1 to many FlowFiles, this can result in your content repository filling up. NiFi can only remove content claims which have no FlowFiles pointing to them anymore. Here are some useful links: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nipyapi.html https://github.com/Chaffelson/nipyapi http://nifi.apache.org/docs/nifi-docs/rest-api/index.html https://community.cloudera.com/t5/Community-Articles/Update-NiFi-Connection-Destination-via-REST-API/ta-p/244211 https://community.cloudera.com/t5/Community-Articles/Change-NiFi-Flow-Using-Rest-API-Part-1/ta-p/244631 Hope this helps, Matt
... View more
02-08-2021
11:53 PM
Hi @Jarinek I think you have to set -pv in front of the new value: nifi set-param -u http://localhost:8088 -pcid 85b8b5bf-0177-1000-3881-8ddd8d31fed8 -pn sample.URL -pv newvalue This one worked on my local NiFi. Remark: Haven't seen that you found it yourself. Great!
... View more
12-08-2020
09:27 AM
Jaro, nipiapi Windows automated testing not attempted. Are there any particular things you want to achieve using nipiapi+Windows combination? -Akash
... View more
09-24-2020
12:59 PM
Hi, @MattWho we actually already implemented prototypes of such custom processors and the basic principles are clear. By processor logic, I did not mean the structure of the processor like relationships. The structure is static in our case. By the processor logic, I refer, the code which is called in "onTrigger" in "StreamCallback.process". In my case, the logic can be parametrized by some configuration data (These configuration data is provided by a service or database tables). The configuration data is static during the whole processor run and must be provided during instantiation of the business logic object. You might see it as lookup, which must not happen during the processing "onTrigger", but "onScheduled" (sorry I wrote in "init" previously, that was not right)
Best regards Jaro
... View more
- Tags:
- apache nifi
09-15-2020
09:51 AM
1 Kudo
Actually, both replies can be considered as valid. I confirmed that one, which better fits to my use case.
... View more
09-08-2020
12:26 AM
1 Kudo
It sounds like your testing solution is exceeding the inbound capabilities of the flow tuning (nifi config, processor/queue config) Correct assessment. It has showed that the pipeline was not properly sized for the amount of data, which lead to a back-pressure in the ingest component
... View more
07-21-2020
09:19 AM
1 Kudo
The easiest way to grab monitoring data is via the NiFi REST API. Also everything in the NiFi UI is done through REST calls which you can call programmatically. Please read the NiFi docs they are linked directly from your running NiFi application or on the web. They are very thorough and have all the information you could want: https://nifi.apache.org/docs/nifi-docs/. If you are not running NiFi 1.11.4, I recommend you please upgrade. This is supported by Cloudera on multiple platforms. NiFi Rest API https://nifi.apache.org/docs/nifi-docs/rest-api/ There's also an awesome Python wrapper for that REST API: https://pypi.org/project/nipyapi/ Also in NiFi flow programming, every time you produce data to Kafka you get metadata back in FlowFile Attributes. You can push those attributes directly to a kafka topic if you want. So after your PublishKafkaRecord_2_0 1.11.4 so for success read the attributes on # of record and other data then AttributesToJson and push to another topic. you may want a mergerecord in there to aggregate a few of those together. If you are interested in Kafka metrics/record counts/monitoring then you must use Cloudera Streams Messaging Manager, it provides a full Web UI, Monitoring Tool, Alerts, REST API and everything you need for monitoring every producer, consumer, broker, cluster, topic, message, offset and Kafka component. The best way to get NiFi stats is to use the NiFi Reporting Tasks, I like the SQL Reporting task. SQL Reporting Tasks are very powerful and use standard SELECT * FROM JVM_METRICS style reporting, see my article: https://www.datainmotion.dev/2020/04/sql-reporting-task-for-cloudera-flow.html Monitoring Articles https://www.datainmotion.dev/2019/04/monitoring-number-of-of-flow-files.html https://www.datainmotion.dev/2019/03/apache-nifi-operations-and-monitoring.html Other Resources https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_9.html https://www.datainmotion.dev/2019/08/using-cloudera-streams-messaging.html https://dev.to/tspannhw/apache-nifi-and-nifi-registry-administration-3c92 https://dev.to/tspannhw/using-nifi-cli-to-restore-nifi-flows-from-backups-18p9 https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html https://www.datainmotion.dev/p/links.html https://www.tutorialspoint.com/apache_nifi/apache_nifi_monitoring.htm https://community.cloudera.com/t5/Community-Articles/Building-a-Custom-Apache-NiFi-Operations-Dashboard-Part-1/ta-p/249060 https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-metrics-reporting-nar/1.11.4/org.apache.nifi.metrics.reporting.task.MetricsReportingTask/ https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-scripting-nar/1.11.4/org.apache.nifi.reporting.script.ScriptedReportingTask/index.html
... View more