Cron scheduling strategy with GetSplunk fails on successive calls

[#1]

I have multiple GetSplunk processors running using a Cron driven scheduling strategy. The Cron expression looks like '0 30 13 * * ?'. They all successfully execute the query the first time it's run. But, the next day it errors out with a 401 error from Splunk. The error from nifi-app.log is as below

WARN [Timer-Driven Process Thread-7] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding GetSplunk[id=01581009-026c-114b-5e2e-401ebea6427d] due to uncaught Exception: com.splunk.HttpException: HTTP 401 -- call not properly authenticated
2016-12-21 13:30:00,300 WARN [Timer-Driven Process Thread-2] o.a.n.c.t.ContinuallyRunProcessorTask
com.splunk.HttpException: HTTP 401 -- call not properly authenticated
	at com.splunk.HttpException.create(HttpException.java:84) ~[na:na]
	at com.splunk.HttpService.send(HttpService.java:452) ~[na:na]
	at com.splunk.Service.send(Service.java:1293) ~[na:na]
	at com.splunk.HttpService.get(HttpService.java:165) ~[na:na]
	at com.splunk.Service.export(Service.java:222) ~[na:na]
	at com.splunk.Service.export(Service.java:237) ~[na:na]
	at org.apache.nifi.processors.splunk.GetSplunk.onTrigger(GetSplunk.java:461) ~[na:na]
	at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064) ~[nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:165) [nifi-framework-core-1.1.0-SNAPSHOT.jar:1.1.0-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_101]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_101]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]

After doing some research, it seems this might be an issue due to multiple threads. Have anyone of you bumped into this? Help appreciated.

[#2 · Accepted solution]

@Raghav Ramakrishann Probably related to https://issues.apache.org/jira/browse/NIFI-3349

If someone restarts your Splunk Search Head, Nifi will not re-authenticate.

[#3]

Does anybody have a workaround for this issue? Restarting the nifi processor manually is not feasible for a flow that should run "lights out".

[#4]

@Ed Prout

One way around the issue, is to monitor the success relationship out of the GetSplunk processor using the MonitorActivity processor. If data does not pass through in a set time period, the the MonitorActivity processor generates an "inactive" flow file and this can be used as a trigger for an ExecuteScript processor which would run a curl script to restart the processor. Not an elegant solution, but it should work.