question PrometheusReportingTask with same port and multiple nifi instances on same machine in Support Questions

PrometheusReportingTask with same port and multiple nifi instances on same machine

kkau — Mon, 10 May 2021 12:44:32 GMT

I am using Nifi 1.12.1 version and configuring Prometheus for monitoring. I have multiple nifi instances running on the same machine. But I am not able to get metrics for all on the same port 9092 and getting below error in nifi log.

From 3 nifi instances, only one nifi is able to start Prometheus reporting task properly others are saying Address already in use as displayed in error below.

Can't we change the Prometheus metrics endpoint from nifi? For example for each nifi instance, we should be able to use a different metric endpoint in below way.

nifi-1 - 9092/metrics (default metric end point)
nifi-2 >>> 9092/nifi_2_metrics
nifi-3 >>> 9092/nifi_3_metrics

2021-05-10 09:55:39,957 ERROR [Timer-Driven Process Thread-4] o.a.n.c.s.StandardProcessScheduler Failed to invoke the On-Scheduled Lifecycle methods of [PrometheusReportingTask[id=47b5aa61-0179-1000-ce9d-d59160596cdb], java.lang.reflect.InvocationTargetException, 30 sec] due to {}; administratively yielding this ReportingTask and will attempt to schedule it again after {} java.lang.reflect.InvocationTargetException: null at jdk.internal.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52) at org.apache.nifi.controller.scheduling.StandardProcessScheduler$2.run(StandardProcessScheduler.java:222) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.nifi.processor.exception.ProcessException: Failed to start Jetty server at org.apache.nifi.reporting.prometheus.PrometheusReportingTask.onScheduled(PrometheusReportingTask.java:152) ... 15 common frames omitted Caused by: java.io.IOException: Failed to bind to /0.0.0.0:9092 at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346) at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:307) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.server.Server.doStart(Server.java:385) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.apache.nifi.reporting.prometheus.PrometheusServer.<init>(PrometheusServer.java:90) at org.apache.nifi.reporting.prometheus.PrometheusReportingTask.onScheduled(PrometheusReportingTask.java:115) ... 15 common frames omitted Caused by: java.net.BindException: Address already in use at java.base/sun.nio.ch.Net.bind0(Native Method) at java.base/sun.nio.ch.Net.bind(Net.java:455) at java.base/sun.nio.ch.Net.bind(Net.java:447) at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227) at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80) at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:342) ... 23 common frames omitted

Re: PrometheusReportingTask with same port and multiple nifi instances on same machine

TimothySpann — Mon, 10 May 2021 16:42:34 GMT

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.11.1/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/

You will need to open a JIRA.

You should not run 3 nifis on the same machine unless it's in K8. NiFi needs a lot of RAM and cores.

Re: PrometheusReportingTask with same port and multiple nifi instances on same machine

MattWho — Mon, 10 May 2021 19:36:37 GMT

@kkau

The ERROR you see is because the PrometheusRecordSink controller services when started binds to the configured port as a listener which allows Prometheus server to connect to that port for the purpose fo scraping data from NiFi.

On a single host, only one service can bind to a port. So any subsequent component on that server that attempts to bind to that same port will fail with "address already in use".

I don't know much of anything about Prometheus server, but if you can configure Prometheus to scrape from a unique port on each NiFi instance then you should be able use the variable registry to define a unique port number per host. By using NiFi Expression Language (EL) you would still have matching flow.xmls on all your NiFi nodes. You would simply have a unique variable registry file configured for each instance in that instance's nifi.properties file.

So for your nodes you would have three files:
<path to file>/node1-variables
<path to file>/node2-variables
<path to file>/node3-variables

In inside each of those files you would have the same property but each with unique value
PrometheusPort=9092 (written to node1-variables file)
PrometheusPort=9093 (written to node2-variables file)
PrometheusPort=9094 (written to node3-variables file)

Then when you start this reporting task, each node binds to a different port number. Then you have your Prometheus server scrape from all three unique endpoints.

But I do agree with @TimothySpann that having multiple NiFi hosts run on the same server is not a recommended setup.

If you found that any of the provided solutions helped with your query, please take a moment to login and click accept on the solution(s).

Thanks,
Matt

Re: PrometheusReportingTask with same port and multiple nifi instances on same machine

kkau — Wed, 12 May 2021 12:18:37 GMT

@MattWho

@TimothySpann

I tried the solution given by you that is to use variable registry and to use the variable defined there with EL in PrometheusReportingTask property as ${PrometheusPort}. But it is not evaluating it as an expression it shows the error that it's not an Integer and I got NumberFormatException in logs.

I think nifi-1.12.1 version just shows in the UI that this property support EL but actually it is not supporting.

I checked the source code as well in the git repo there as well, No evaluation is done for the Port property Expression ${}. Code is also expecting a hard-coded value.

Git source code for PrometheusReportingTask class.

Error in logs

2021-05-12 17:00:27,122 ERROR [Timer-Driven Process Thread-2] o.a.n.c.s.StandardProcessScheduler Failed to invoke the On-Scheduled Lifecycle methods of [PrometheusReportingTask[id=47b53f71-0179-1000-c8a2-06939a325530], java.lang.reflect.InvocationTargetException, 30 sec] due to {}; administratively yielding this ReportingTask and will attempt to schedule it again after {} java.lang.reflect.InvocationTargetException: null at jdk.internal.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:142) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:130) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:75) at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:52) at org.apache.nifi.controller.scheduling.StandardProcessScheduler$2.run(StandardProcessScheduler.java:222) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.nifi.processor.exception.ProcessException: Failed to start Jetty server at org.apache.nifi.reporting.prometheus.PrometheusReportingTask.onScheduled(PrometheusReportingTask.java:152) ... 15 common frames omitted Caused by: java.lang.NumberFormatException: For input string: "${PrometheusPort}" at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.base/java.lang.Integer.parseInt(Integer.java:638) at java.base/java.lang.Integer.parseInt(Integer.java:770) at org.apache.nifi.reporting.prometheus.PrometheusReportingTask.onScheduled(PrometheusReportingTask.java:115) ... 15 common frames omitted

Re: PrometheusReportingTask with same port and multiple nifi instances on same machine

MattWho — Wed, 12 May 2021 19:02:15 GMT

@kkau

Sorry to hear that the embedded documentation was not accurate for the port property.
I filed an apache jira to address that issue:
https://issues.apache.org/jira/browse/NIFI-8536

Best path forward is following NIFi recommended best practice where each NiFi node in your cluster is on a unique host. This would allow each node to bind this reporting task to the same port.

Having multiple hosts on same server will present issue/challenges with any component (processor, controller service, reporting tasks, etc) that create a listener port. While some may you may be able to work around with Variable Registry suggestion I shared, others like this use case will not.

Thanks,
Matt

Re: PrometheusReportingTask with same port and multiple nifi instances on same machine

kkau — Wed, 12 May 2021 19:26:28 GMT

Thanks @MattWho

Sure, I will consider your suggestion to not running multiple nifi on the same machine.

I tried the variable registry approach as well but the problem is the same with that as well that we can not use the EL parameter.