question Re: nifi disabled controller-services and ConsumeJMS failed in Support Questions

nifi disabled controller-services and ConsumeJMS failed

toandyliang — Mon, 19 Aug 2019 11:00:33 GMT

When I use Nifi connect to IBM MQ, nifi disabled controller-services and ConsumeJMS failed.

I got the following error:

ConsumeJMS - null ConsumeJMS - null failed to process session due to java.lang.IllegalStateException: Cannot invoke method public abstract javax.jms.ConnectionFactory org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition.getConnectionFactory() on Controller Service with identifier 57fa0ec2-015a-1000-1377-c1374f9dd1c1 because the Controller Service is disabled: java.lang.IllegalStateException: Cannot invoke method public abstract javax.jms.ConnectionFactory org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition.getConnectionFactory() on Controller Service with identifier 57fa0ec2-015a-1000-1377-c1374f9dd1c1 because the Controller Service is disabled

nifi Error log:

2017-02-20 05:32:21,023 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.jms.processors.ConsumeJMS ConsumeJMS - null ConsumeJMS - null failed to process session due to java.lang.IllegalStateException: Cannot invoke method public abstract javax.jms.ConnectionFactory org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition.getConnectionFactory() on Controller Service with identifier 57fa0ec2-015a-1000-1377-c1374f9dd1c1 because the Controller Service is disabled: java.lang.IllegalStateException: Cannot invoke method public abstract javax.jms.ConnectionFactory org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition.getConnectionFactory() on Controller Service with identifier 57fa0ec2-015a-1000-1377-c1374f9dd1c1 because the Controller Service is disabled 2017-02-20 05:32:21,025 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.jms.processors.ConsumeJMS java.lang.IllegalStateException: Cannot invoke method public abstract javax.jms.ConnectionFactory org.apache.nifi.jms.cf.JMSConnectionFactoryProviderDefinition.getConnectionFactory() on Controller Service with identifier 57fa0ec2-015a-1000-1377-c1374f9dd1c1 because the Controller Service is disabled at org.apache.nifi.controller.service.StandardControllerServiceProvider$1.invoke(StandardControllerServiceProvider.java:172) ~[nifi-framework-core-1.1.1.jar:1.1.1] at com.sun.proxy.$Proxy139.getConnectionFactory(Unknown Source) ~[na:na] at org.apache.nifi.jms.processors.AbstractJMSProcessor.buildTargetResource(AbstractJMSProcessor.java:190) ~[na:na] at org.apache.nifi.jms.processors.AbstractJMSProcessor.onTrigger(AbstractJMSProcessor.java:134) ~[na:na] at org.apache.nifi.jms.processors.ConsumeJMS.onTrigger(ConsumeJMS.java:57) ~[na:na] at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) ~[nifi-api-1.1.1.jar:1.1.1] at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1099) ~[nifi-framework-core-1.1.1.jar:1.1.1] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136) [nifi-framework-core-1.1.1.jar:1.1.1] at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) [nifi-framework-core-1.1.1.jar:1.1.1] at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) [nifi-framework-core-1.1.1.jar:1.1.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_111] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_111] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_111] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]

JMSConnectionFactoryProvider settings:

ConsumeJMS settings

Any suggestion on how to solve this issue is appreciated.

Thanks a lot

Andy

Re: nifi disabled controller-services and ConsumeJMS failed

ozhurakousky — Mon, 20 Feb 2017 21:46:22 GMT

Andy

Sorry to hear you're having some issues.

It appears this is a configuration issue. I just dug up some notes from when I was implementing it and it appears you have issues with ConnectionFactory configuration as well as some additional properties required by IBM that would need to be supplied as dynamic properties (see 'IBM specific properties as Dynamic Properties' below). Please see below and let me know. Would love ti help you put this matter to rest.

Processor (required properties):

- Destination Name: <topic or queue name> (e.g., queue://MYQUEUE)

- Destination Type: QUEUE or TOPIC

- Session Cache size: 1 (read its doc for more details)

- Connection Factory service: <the name of the controller service for CF> (e.g., IBMMQ)

Controller Service (e.g., IBMMQ)

- MQ Connection Factory Implementation: com.ibm.mq.jms.MQConnectionFactory

- MQ Client Libraries path: <path to client JARs from UBM>

- Broker URI: <uri to MQ broker> (e.g., foo.bar:1234)

Then you would have to configure IBM specific properties as Dynamic Properties:

- channel: <name of the channel> (e.g., SYSTEM.ADMIN.SVRCONN)

- queueManager: <name of queue managere> (e.g., FOO)

- transportType: <identifier of the transport type> (make sure you put '1' as a value which states that TCP/IP will be used)

Re: nifi disabled controller-services and ConsumeJMS failed

ozhurakousky — Mon, 20 Feb 2017 21:58:59 GMT

Also, keep in mind, you your provided connection factory impl is wrong since you lower-cased the last part while it supposed to be the usual camel-case - https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.javadoc.doc/WMQJMSClasses/com/ibm/mq/jms/MQQueueConnectionFactory.html

Also, the url beginning with 'http'. . . I do not believe that is correct. At least I've not seen a configuration like that. Anyway, please reach out to your MQ administrator to get proper values for the required properties.

Also, if you be so kind please include the error logs relevant to the controller service (CS). What you are attaching above is related to inability of processor to start which is understandable since it can not start due to the fact that dependent CS is disabled. The real question why the dependent CS is disabled, I mean we do know already that there is configuration issue, but he logs may shed some lights as to what those issues are.

Re: nifi disabled controller-services and ConsumeJMS failed

toandyliang — Mon, 20 Feb 2017 22:50:28 GMT

@ozhurakousky Thank you very much for your advice. It's very helpful for me.

After I modified the connection factory to com.ibm.mq.jms.MQQueueConnectionFactory and set the Destination Name to queue://NifiMQ/Q1, I didn't see the controller-services issue. I got another error:

2017-02-20 09:33:28,807 WARN [Timer-Driven Process Thread-8] o.a.n.c.t.ContinuallyRunProcessorTask org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSFMQ6312: An exception occurred in the Java(tm) MQI. The Java(tm) MQI has thrown an exception describing the problem. See the linked exception for further information.; nested exception is com.ibm.mq.jmqi.JmqiException: CC=2;RC=2495;AMQ8568: The native JNI library 'mqjbnd64' was not found. For a client installation this is expected. [3=mqjbnd64]

For development, I installed IBM MQ locally on my computer, and set the MQ Client Libraries path to C:/Program Files/IBM/WebSphere MQ/java/lib

Re: nifi disabled controller-services and ConsumeJMS failed

toandyliang — Tue, 21 Feb 2017 00:03:09 GMT

@ozhurakousky For the confiugration of IBM MQ, I followed the instruction at https://www.ibm.com/support/knowledgecenter/SSFKSJ_9.0.0/com.ibm.mq.sce.doc/q123070_.htm

It should should have configured the parameters of the Nifi, The only parameter that it skips is the channel. any other configuration that I missed?

Re: nifi disabled controller-services and ConsumeJMS failed

toandyliang — Tue, 21 Feb 2017 00:50:32 GMT

@ozhurakousky Thank you very much for your help, I am fine tuning the settings and keep testing. It's in the right direction now, although I still have other issues.

Re: nifi disabled controller-services and ConsumeJMS failed

ozhurakousky — Tue, 21 Feb 2017 01:07:24 GMT

Andy, the error you see is due to the fact that the connection attempt is made over native transport protocol using JNI and the required native libraries are not present on the classpath. To avoid it you should rely on TCP transport type. NOTE: the dynamic property mentioned above 'transportType". It's values should be set to 1 (which denotes TCP).

Keep in mind that the other two properties described above (channel and queueManager) are also required

Also, the document you've provided describes JNDI configuration which we do not currently support.

Re: nifi disabled controller-services and ConsumeJMS failed

ozhurakousky — Tue, 21 Feb 2017 01:11:27 GMT

Just in case you are not familiar with dynamic properties look for a '+' sign in the top right corner of the configuration window for CS. Click on the '+' and enter the name of the property (i.e., transportType) and then value of '1'.

Re: nifi disabled controller-services and ConsumeJMS failed

toandyliang — Tue, 21 Feb 2017 23:21:24 GMT

@ozhurakousky Thank you very much for spending time to helping me. Do you have any information on how to setup IBM mq for Nifi? e.g. how could I set up the SYSTEM.ADMIN.SVRCONN for channel or is it build-in already? Do I need to create a transmission queue for the channel?

Re: nifi disabled controller-services and ConsumeJMS failed

toandyliang — Mon, 19 Aug 2019 11:00:20 GMT

@ozhurakouskyThanks for the advice and your time. I had setup the processor as the following:

Controller Service

and

Processor

For the IBM MQ side

1) I found the channel SYSTEM.ADMIN.SVRCONN , do I need to create a sender channel or it's already built in?

2) I already have a listener for Nifi (localhost:1419)

3) I create a local queue Q1 (do I have to set it's Usage to transmission?)

4) I got the following error:

ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] ConsumeJMS - JMSConsumer[destination:null; pub-sub:false;] failed to process due to org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NifiMQ' with connection mode 'Client' and host name 'localhost(1419)'.

Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2540' ('MQRC_UNKNOWN_CHANNEL_NAME').; rolling back session: org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'NifiMQ' with connection mode 'Client' and host name 'localhost(1419)'.

Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.; nested exception is com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2540' ('MQRC_UNKNOWN_CHANNEL_NAME').

Is there Any other configuration that I missed?

Thanks a lot. @ozhurakousky I am really appreciated for your help and your time.

Re: nifi disabled controller-services and ConsumeJMS failed

ozhurakousky — Wed, 22 Feb 2017 01:12:41 GMT

What error are you getting now? Could you please post

Also, it appears form your configuration that your queue name is 'Q1' but in NiFi config you're using 'queue://NiFiMQ' instead of 'queue://Q1'

I am not MQ administrator and there is nothing specific to NiFi anyone on the MQ side needs to do. Do you have a plain Java program that is able to send/receive messages from MQ instance you have created. That would be the easiest way to debug. The information such as queue, channel, queueManager, transportType and many other is provided by MQ administrators and have no relevance to NiFi.

Re: nifi disabled controller-services and ConsumeJMS failed

ozhurakousky — Wed, 22 Feb 2017 01:33:45 GMT

Ok, based on your additional info that was just posted you are having a connection issue to localhost:1419

Have you explicitly change it from the default port of 1414? I know you have a config window that shows some kind of listener on on 1419, but I am not sure what that is.

In any event, you're definitely having networking issue so, let's try this and see if you can connect to either 1419 and 1414 via telnet. From the terminal window execute the following:

> telnet localhost 1419

Repeat the same for 1414

Let me know

Re: nifi disabled controller-services and ConsumeJMS failed

toandyliang — Thu, 23 Feb 2017 05:14:05 GMT

@ozhurakousky

Hi Oleg,

I had re-installed IBM MQ and setting up the queue manager, Channel, listener, and queue.

I will do more testing on Nifi and post the result. Thanks a lot

Andy

Re: nifi disabled controller-services and ConsumeJMS failed

ozhurakousky — Thu, 23 Feb 2017 05:50:36 GMT

Yes, you can see there are two properties in ConsumeJMS processor User Name and Password. So you can provide them there.

Re: nifi disabled controller-services and ConsumeJMS failed

toandyliang — Fri, 24 Feb 2017 02:40:30 GMT

@ozhurakousky

Hi Oleg,

I finally got the connection between Nifi and IBM MQ working now. It's some configuration issue on MQ.

Thank you very much for your help and time. I am very appreciated.

Andy

Re: nifi disabled controller-services and ConsumeJMS failed

ozhurakousky — Fri, 24 Feb 2017 02:44:57 GMT

Andy, that is a great news!

If you don't mind, would you please post your working configuration here (masking user/password or course), so others can benefit from your experience.

Cheers

Oleg

Re: nifi disabled controller-services and ConsumeJMS failed

toandyliang — Fri, 24 Feb 2017 02:49:34 GMT

@ozhurakousky of course, I will post what I did on the IBM MQ here later today.

Re: nifi disabled controller-services and ConsumeJMS failed

toandyliang — Sat, 25 Feb 2017 06:29:22 GMT

@ozhurakousky

Here is the steps to setup IBM MQ for the Nifi connection

1) Create queue manager myMQManager using MQ Explorer

2) create Channel and queue using the following commands

runmqsc myMQManager

def chl(myMQChannel) chltype(svrconn) replace

def chl(myMQChannel) chltype(clntconn) conname('localhost(1420)') QMNAME('myMQManager') replace

def ql(myMQqueue)

2) Grant necessary authorization for a user using MQ Explorer

2.1 Queue Manager Authorities Setting

right click your queue manager-->Object Authorities-->Add Role Based Authorities -->give the user access permission

2.2 Queue Authorities Setting

right click your queue-->Object Authorities-->Manage Authority Records-->Specific Profiles--> your queue--> Accumulated authorities-->setup your user Access Permission for the queue

3) After the configuration and setup your Nifi ConsumeJMS processor, you should be able connect to IBM MQ