Nifi - Listen Syslog - getting failed @OnScheduled

[#1]

We are trying to capture the syslog in Nifi and it shows the following error on ListenSyslog processor. Set up the UDP port 514 on the processor as well. Any pointers?

In nifi-app.log,

ERROR [StandardProcessScheduler Thread-3] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.

java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.

at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1483)

at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:103)

at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1302)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:206)

at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1466)

... 9 common frames omitted

Caused by: java.lang.reflect.InvocationTargetException: null

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:137)

at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:125)

at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotations(ReflectionUtils.java:70)

at org.apache.nifi.util.ReflectionUtils.invokeMethodsWithAnnotation(ReflectionUtils.java:47)

at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1306)

at org.apache.nifi.controller.StandardProcessorNode$1$1.call(StandardProcessorNode.java:1302)

... 6 common frames omitted

Caused by: java.net.SocketException: Permission denied

at sun.nio.ch.Net.bind0(Native Method)

at sun.nio.ch.Net.bind(Net.java:433)

at sun.nio.ch.DatagramChannelImpl.bind(DatagramChannelImpl.java:691)

at sun.nio.ch.DatagramSocketAdaptor.bind(DatagramSocketAdaptor.java:91)

at org.apache.nifi.processor.util.listen.dispatcher.DatagramChannelDispatcher.open(DatagramChannelDispatcher.java:99)

at org.apache.nifi.processors.standard.ListenSyslog.onScheduled(ListenSyslog.java:322)

... 16 common frames omitted

[#2 · Accepted solution]

Port 514 is a privileged port. This means it can only be accessed by a superuser.

Since there are security implications to running Nifi as root, it is typically run as the nifi user.

There are a couple of options:

1. run the syslog listener on a port > 1024, e.g. port 1514 instead of 514.
2. use iptables to forward port 514 external to a non-privileged internal port, and have the syslog listener listen to that port.
3. use authbind to allow the Nifi user permissions to bind to port 514.

[#3]

@Alex Woolford

I did the option 1 and the error is no longer there. But still the data is not getting loaded in the syslog loader.

What could be wrong and is there any basic troubleshooting i can do?

[#4]

You might start by using the `logger` command to send some sample syslog messages. Don't forget to add the `--port 1514` argument. Try running that on the Nifi host, and then on a host that's external to your Nifi cluster. If it works from a Nifi host but not from outside Nifi, you might need to tweak iptables or a firewall rule.

You might try using tcpdump to monitor network traffic for port 1514.

I'd also recommend running a `tail -f /var/log/nifi/nifi-app.log` on the Nifi host(s) while you're running the syslog listener to see if there are any interesting messages.

[#5]

@Alex Woolford Used option2 and set the Nifi to listen on that port. Works fine. Thanks!