Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

Nifi ExecuteStreamCommand Process error with IllegalStateException: Partition is Cloed

Rising Star

Hi Team ,

to pull the data from servers , we are running ssh commands using ExecuteStreamCommand processor . looks like some ssh commands/flowfiles are blocked in this processor due to no response from server(it is acceptable and common in server) . if it is the case ,this processor has to terminate the flowfile , but it is not and after sometime , ExecuteStreamCommand processor is hanging up .

Queue is showing some no of flow files ,but could not able to clear the queue in Nifi UI . looks ExecuteStreamCommand processor is trying or locked in it . what will be the solution to overcome this situation even if it is unable to process or run the ssh commads . Error we got in Nifi-app.log : 2017-03-27 04:03:56,125 ERROR [Timer-Driven Process Thread-82] o.a.n.p.standard.ExecuteStreamCommand ExecuteStreamCommand[id=5bda45d9-6601-14f0-3a44-294b3f5d994e] Transferring flow file StandardFlowFileRecord[uuid=633b08a2-f84d-4dc7-9a8c-175c5bc13b2e,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1490600595096-9, container=default, section=9], offset=0, length=0],offset=0,,size=0] to output stream. Executable command ssh ended in an error: 2017-03-27 04:03:56,125 WARN [Timer-Driven Process Thread-84] o.a.n.p.standard.ExecuteStreamCommand ExecuteStreamCommand[id=5bda45d9-6601-14f0-3a44-294b3f5d994e] Processor Administratively Yielded for 1 sec due to processing failure 2017-03-27 04:03:56,125 WARN [Timer-Driven Process Thread-84] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding ExecuteStreamCommand[id=5bda45d9-6601-14f0-3a44-294b3f5d994e] due to uncaught Exception: java.lang.IllegalStateException: Partition is closed 2017-03-27 04:03:56,126 WARN [Timer-Driven Process Thread-84] o.a.n.c.t.ContinuallyRunProcessorTask java.lang.IllegalStateException: Partition is closed at org.wali.MinimalLockingWriteAheadLog$Partition.update( ~[nifi-write-ahead-log-] at org.wali.MinimalLockingWriteAheadLog.update( ~[nifi-write-ahead-log-] at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository( ~[nifi-framework-core-] at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository( ~[nifi-framework-core-] at org.apache.nifi.controller.repository.StandardProcessSession.commit( ~[nifi-framework-core-] at org.apache.nifi.controller.repository.StandardProcessSession.commit( ~[nifi-framework-core-] at org.apache.nifi.processor.AbstractProcessor.onTrigger( ~[nifi-api-] at org.apache.nifi.controller.StandardProcessorNode.onTrigger( ~[nifi-framework-core-]



How have you set up your shell script to handle closing on job timeout? I'm not aware of ExecuteStreamCommand having a hidden timeout property, though you could add something like a 'max wait' timeout if you wanted to do a PR.