nifi disabled controller-services and ConsumeJMS failed

[#1]

When I use Nifi connect to IBM MQ, nifi disabled controller-services and ConsumeJMS failed.

I got the following error:

ConsumeJMS - null ConsumeJMS - null failed to process session due to java.lang.IllegalStateException: Cannot invoke method public abstract javax.jms.ConnectionFactory org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition.getConnectionFactory() on Controller Service with identifier 57fa0ec2-015a-1000-1377-c1374f9dd1c1 because the Controller Service is disabled: java.lang.IllegalStateException: Cannot invoke method public abstract javax.jms.ConnectionFactory org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition.getConnectionFactory() on Controller Service with identifier 57fa0ec2-015a-1000-1377-c1374f9dd1c1 because the Controller Service is disabled

nifi Error log:

2017-02-20 05:32:21,023 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS - null ConsumeJMS - null failed to process session due to java.lang.IllegalStateException: Cannot invoke method public abstract javax.jms.ConnectionFactory org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition.getConnectionFactory() on Controller Service with identifier 57fa0ec2-015a-1000-1377-c1374f9dd1c1 because the Controller Service is disabled: java.lang.IllegalStateException: Cannot invoke method public abstract javax.jms.ConnectionFactory org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition.getConnectionFactory() on Controller Service with identifier 57fa0ec2-015a-1000-1377-c1374f9dd1c1 because the Controller Service is disabled 2017-02-20 05:32:21,025 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.jms.processors.ConsumeJMS java.lang.IllegalStateException: Cannot invoke method public abstract javax.jms.ConnectionFactory org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition.getConnectionFactory() on Controller Service with identifier 57fa0ec2-015a-1000-1377-c1374f9dd1c1 because the Controller Service is disabled at org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:172) ~[nifi-framework-core-1.1.1.jar:1.1.1] at com.sun.proxy.$Proxy139.getConnectionFactory(Unknown Source) ~[na:na] at org.apache.nifi.jms.processors.AbstractJMSProcessor.buildTargetResource(AbstractJMSProcessor.java:190) ~[na:na] at org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:134) ~[na:na] at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:57) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.1.jar:1.1.1] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) ~[nifi-framework-core-1.1.1.jar:1.1.1] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.1.jar:1.1.1] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.1.jar:1.1.1] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.1.jar:1.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_111] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_111] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

JMSConnectionFactoryProvider settings:

ConsumeJMS settings

Any suggestion on how to solve this issue is appreciated.

Thanks a lot

Andy

[#2]

Andy

Sorry to hear you're having some issues.

It appears this is a configuration issue. I just dug up some notes from when I was implementing it and it appears you have issues with ConnectionFactory configuration as well as some additional properties required by IBM that would need to be supplied as dynamic properties (see 'IBM specific properties as Dynamic Properties' below). Please see below and let me know. Would love ti help you put this matter to rest.

Processor (required properties):

- Destination Name: <topic or queue name> (e.g., queue://MYQUEUE)

- Destination Type: QUEUE or TOPIC

- Session Cache size: 1 (read its doc for more details)

- Connection Factory service: <the name of the controller service for CF> (e.g., IBMMQ)

Controller Service (e.g., IBMMQ)

- MQ Connection Factory Implementation: com.ibm.mq.jms.MQConnectionFactory

- MQ Client Libraries path: <path to client JARs from UBM>

- Broker URI: <uri to MQ broker> (e.g., foo.bar:1234)

Then you would have to configure IBM specific properties as Dynamic Properties:

- channel: <name of the channel> (e.g., SYSTEM.ADMIN.SVRCONN)

- queueManager: <name of queue managere> (e.g., FOO)

- transportType: <identifier of the transport type> (make sure you put '1' as a value which states that TCP/IP will be used)

[#3]

@ozhurakousky For the confiugration of IBM MQ, I followed the instruction at https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.sce.doc/q123070_.htm

It should should have configured the parameters of the Nifi, The only parameter that it skips is the channel. any other configuration that I missed?

[#4]

@ozhurakousky Thank you very much for your help, I am fine tuning the settings and keep testing. It's in the right direction now, although I still have other issues.

[#5]

Also, keep in mind, you your provided connection factory impl is wrong since you lower-cased the last part while it supposed to be the usual camel-case - https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.javadoc.doc/WMQJMSClasses/com...

Also, the url beginning with 'http'. . . I do not believe that is correct. At least I've not seen a configuration like that. Anyway, please reach out to your MQ administrator to get proper values for the required properties.

Also, if you be so kind please include the error logs relevant to the controller service (CS). What you are attaching above is related to inability of processor to start which is understandable since it can not start due to the fact that dependent CS is disabled. The real question why the dependent CS is disabled, I mean we do know already that there is configuration issue, but he logs may shed some lights as to what those issues are.

[#6]

@ozhurakousky Thank you very much for your advice. It's very helpful for me.

After I modified the connection factory to com.ibm.mq.jms.MQQueueConnectionFactory and set the Destination Name to queue://NifiMQ/Q1, I didn't see the controller-services issue. I got another error:

2017-02-20 09:33:28,807 WARN [Timer-Driven Process Thread-8] o.a.n.c.t.ContinuallyRunProcessorTask org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception occurred in the Java(tm) MQI. The Java(tm) MQI has thrown an exception describing the problem. See the linked exception for further information.; nested exception is com.ibm.mq.jmqi.JmqiException: CC=2;RC=2495;AMQ8568: The native JNI library 'mqjbnd64' was not found. For a client installation this is expected. [3=mqjbnd64]

For development, I installed IBM MQ locally on my computer, and set the MQ Client Libraries path to C:/Program Files/IBM/WebSphere MQ/java/lib

[#7]

Andy, the error you see is due to the fact that the connection attempt is made over native transport protocol using JNI and the required native libraries are not present on the classpath. To avoid it you should rely on TCP transport type. NOTE: the dynamic property mentioned above 'transportType". It's values should be set to 1 (which denotes TCP).

Keep in mind that the other two properties described above (channel and queueManager) are also required

Also, the document you've provided describes JNDI configuration which we do not currently support.

[#8]

Just in case you are not familiar with dynamic properties look for a '+' sign in the top right corner of the configuration window for CS. Click on the '+' and enter the name of the property (i.e., transportType) and then value of '1'.

[#9]

@ozhurakousky Thank you very much for spending time to helping me. Do you have any information on how to setup IBM mq for Nifi? e.g. how could I set up the SYSTEM.ADMIN.SVRCONN for channel or is it build-in already? Do I need to create a transmission queue for the channel?

[#10]

@ozhurakouskyThanks for the advice and your time. I had setup the processor as the following:

Controller Service

and

Processor

For the IBM MQ side

1) I found the channel SYSTEM.ADMIN.SVRCONN , do I need to create a sender channel or it's already built in?

2) I already have a listener for Nifi (localhost:1419)

3) I create a local queue Q1 (do I have to set it's Usage to transmission?)

4) I got the following error:

ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] failed to process due to org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NifiMQ' with connection mode 'Client' and host name 'localhost(1419)'.

Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2540' ('MQRC_UNKNOWN_CHANNEL_NAME').; rolling back session: org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NifiMQ' with connection mode 'Client' and host name 'localhost(1419)'.

Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2540' ('MQRC_UNKNOWN_CHANNEL_NAME').

Is there Any other configuration that I missed?

Thanks a lot. @ozhurakousky I am really appreciated for your help and your time.