File replication in nifi cluster, prometheus custom metric

Question

hello,

please help with below questions:

1. I have nifi cluster with 2 nodes and running ExecuteGroovyScript processor to create text formatted file generated from flow data content and save it on local filesystem but it is saving file only on one nifi node, is there any way to replicate this file on all nodes at a time?

2. Above file formatting by ExecuteGroovyScript processor as text in Prometheus format and using such flow http_request-> fetchfile-> http_response processors to create custom Prometheus healthcheckmetric web page, it worked for single node but with 2 nodes all data balanced randomly between 2 nodes so I have different content in 2 web pages ( http://node1/healthcheck/ and http://node2/healthcheck/ ) so monitoring not working properly, could you help how to add custom Prometheus metric so it will be available like regular nifi prometheus web page /metrics/ with same content on both nodes? may be other solutions?

MattWho · Accepted Answer

@Vapper

In a NiFi cluster each node executes its own copy of the flow.xml.gz, has its own set of repositories, and executes against only the FlowFiles on that specific node. So you need to consider this in your dataflow designs.

Perhaps the dataflow that executes the groovy Script on all nodes could be designed to write all the produced files to a single destination server. Then in your http_request-> fetchfile-> http_response dataflow, instead of "fetchFile" use fetchSFTP to retrieve all those files and mergeContent to merge in to one file before the http_response.

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.