@Jarinek    Please check my inline answer to your queries:    1) is it possible to alter compression.type on a kafka topic while having applications on the topic running? >> Yes, we would recommend stopping the producer and then alert compression type as whenever the Kafka broker gets a compressed batch of messages from a producer. It will decompress the data to validate if the compression type is different at the topic level and producer level and if compression is the same from both sides, then it will write directly to the topic/leader partition.    2) What should be expected hard drive saving factor for text records like json (2KB size) and compression.type=gzip. >> You should use and test snappy or lz4 for best speed/compression ratio   3) Does it apply to KafkaMirrorMaker only? >> Yes, as per the screenshot it will apply for the MM only.   4) Are producer and consumer applications somehow affected, when the global parameter changes. >> If you are changing it from the Kafka side, then it should not affect the applications.   If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂  & ”Accept as Solution" below this post.   Thank you. ... View more

I have eventually solved the issue. The Writter property "Schema Access Strategy" was misconfigured. It must be set to "Use 'Schema Text' Property" to really apply the given schema . ... View more

@Jarinek ,   This is total volume, in bytes, received from the network by all the brokers across the cluster. Replication traffic is counted by this counter as well.   Cheers, André   -- Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button. ... View more

Stupid me, the port was already occupied ... View more

@Jarinek  The process really depends on what update you are trying to make. 1. You can not remove a connection that has queued FlowFiles in it, but you can redirect it to a different target processor with queued data. 2. You can not redirect a connection if the processor it is currently attached to still has a running thread.  Stopping a processor does not kill threads, it simply tells the processor to not execute again at the configured run schedule.  Existing threads will continue to run until they complete.  Until all threads exit, the processor is still in a state of "stopping" even though UI reflect red square for "stopped". 3. You cannot modify a processor if is still has running threads (see note about "stopping" processors above) 4. If you stop the component that is on the receiving side of a connection, any FlowFiles queued on that connection, not tied to any active thread still running on target processor component, will not be processed and remain queued on the connection.  You can manual empty a queue through a rest-api call (means data loss), but that is not necessary if you are not deleting the connection. Attempts to perform configuration changes when components still have active threads or are in a running state will result in an exception being thrown and the change not happening. Attempts remove connections that have queued FlowFiles will throw an exception and block removal. Now if all you are trying to do is modify some configuration on a processor, all you need to do is stop the processor, check that it has no active threads, make the config change, and then start the processor again. Not sure wha you are asking with " update the flow ignoring any data in failure or error connection queues".  NiFi does not ignore queued FlowFiles.  It also not wise to leave connection with queued FlowFiles just sitting around your dataflows.   Those old queued FlowFile will prevent removal or content claims that contain that FlowFiles data.  Since a content claim can contain the data from 1 to many FlowFiles, this can result in your content repository filling up.  NiFi can only remove content claims which have no FlowFiles pointing to them anymore. Here are some useful links: https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nipyapi.html https://github.com/Chaffelson/nipyapi http://nifi.apache.org/docs/nifi-docs/rest-api/index.html https://community.cloudera.com/t5/Community-Articles/Update-NiFi-Connection-Destination-via-REST-API/ta-p/244211 https://community.cloudera.com/t5/Community-Articles/Change-NiFi-Flow-Using-Rest-API-Part-1/ta-p/244631 Hope this helps, Matt ... View more

Hi @Jarinek  I think you have to set -pv in front of the new value: nifi set-param -u http://localhost:8088 -pcid 85b8b5bf-0177-1000-3881-8ddd8d31fed8 -pn sample.URL -pv newvalue This one worked on my local NiFi. Remark: Haven't seen that you found it yourself. Great! ... View more

Jaro, nipiapi  Windows automated testing not attempted.  Are there any particular things you want to achieve using nipiapi+Windows combination?   -Akash ... View more

The easiest way to grab monitoring data is via the NiFi REST API.  Also everything in the NiFi UI is done through REST calls which you can call programmatically.   Please read the NiFi docs they are linked directly from your running NiFi application or on the web.   They are very thorough and have all the information you could want:   https://nifi.apache.org/docs/nifi-docs/.   If you are not running NiFi 1.11.4, I recommend you please upgrade.   This is supported by Cloudera on multiple platforms.   NiFi Rest API https://nifi.apache.org/docs/nifi-docs/rest-api/   There's also an awesome Python wrapper for that REST API:  https://pypi.org/project/nipyapi/   Also in NiFi flow programming, every time you produce data to Kafka you get metadata back in FlowFile Attributes.   You can push those attributes directly to a kafka topic if you want.   So after your PublishKafkaRecord_2_0 1.11.4 so for success read the attributes on # of record and other data then AttributesToJson and push to another topic.   you may want a mergerecord in there to aggregate a few of those together.   If you are interested in Kafka metrics/record counts/monitoring then you must use Cloudera Streams Messaging Manager, it provides a full Web UI, Monitoring Tool, Alerts, REST API and everything you need for monitoring every producer, consumer, broker, cluster, topic, message, offset and Kafka component.   The best way to get NiFi stats is to use the NiFi Reporting Tasks, I like the SQL Reporting task.   SQL Reporting Tasks are very powerful and use standard SELECT * FROM JVM_METRICS style reporting, see my article: https://www.datainmotion.dev/2020/04/sql-reporting-task-for-cloudera-flow.html   Monitoring Articles   https://www.datainmotion.dev/2019/04/monitoring-number-of-of-flow-files.html https://www.datainmotion.dev/2019/03/apache-nifi-operations-and-monitoring.html   Other Resources https://www.datainmotion.dev/2019/10/migrating-apache-flume-flows-to-apache_9.html https://www.datainmotion.dev/2019/08/using-cloudera-streams-messaging.html https://dev.to/tspannhw/apache-nifi-and-nifi-registry-administration-3c92 https://dev.to/tspannhw/using-nifi-cli-to-restore-nifi-flows-from-backups-18p9 https://nifi.apache.org/docs/nifi-docs/html/toolkit-guide.html https://www.datainmotion.dev/p/links.html https://www.tutorialspoint.com/apache_nifi/apache_nifi_monitoring.htm https://community.cloudera.com/t5/Community-Articles/Building-a-Custom-Apache-NiFi-Operations-Dashboard-Part-1/ta-p/249060 https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-metrics-reporting-nar/1.11.4/org.apache.nifi.metrics.reporting.task.MetricsReportingTask/ https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-scripting-nar/1.11.4/org.apache.nifi.reporting.script.ScriptedReportingTask/index.html   ... View more