Member since
11-16-2016
40
Posts
7
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2811 | 12-19-2016 03:17 PM |
01-23-2017
09:07 PM
1 Kudo
Using HDF version 2.0.1 I’m seeing an issue where a custom java processor within the NiFi
flow becomes unresponsive or, rather, the messages being sent automatically
routed to the ‘failure’ without any logged reason for the failure.
This instability seems to occur randomly after an unspecified amount of idle
time. If I refresh the processor by toggling on and off then the “failed”
documents are successfully cycled through the custom processor.
One of the problems I’m having is that I cannot use the org.apache.nifi.logging.ComponentLog
or related ProcessorLog which provides logging with in the application.
Either logging within a custom application has been deprecated or I’m
just not able to find a supported jar for such a purpose.
Do either of you have any suggestions for debugging /tips or any
information you may have which would help discern what might be the cause of
these disruptions in the flowfiles?
... View more
Labels:
- Labels:
-
Apache NiFi
12-21-2016
07:45 PM
I see getting customized processors is a bit more involved as described here: https://community.hortonworks.com/articles/4318/build-custom-nifi-processor.html
... View more
12-21-2016
07:33 PM
Thank you for the quick response. I'm needing to go the roll my own route, since the I get an error that "targetClient" is not allowed for 'XMSC_DESTINATION_NAME'. When setting up a processor that uses custom java code what script engine to you specify assuming you are using on of the script processors? Also, to you know how to inject the flowfile stream contents into the java code, since I need that instead of the line ""Hello MQSTR world via MQQueue"? import javax.jms.Queue;
import javax.jms.QueueConnection;
import javax.jms.QueueConnectionFactory;
import javax.jms.QueueSender;
import javax.jms.QueueSession;
import javax.jms.Session;
import javax.jms.TextMessage;
import com.ibm.mq.jms.MQQueue;
import com.ibm.mq.jms.MQQueueConnectionFactory;
import com.ibm.msg.client.wmq.compat.jms.internal.JMSC;
public void testMqstrViaApiTextMessage() throws Exception {
QueueConnectionFactory connectionFactory = new MQQueueConnectionFactory();
((MQQueueConnectionFactory) connectionFactory).setHostName("mfdevlcics.mayo.edu");
((MQQueueConnectionFactory) connectionFactory).setPort(3667);
((MQQueueConnectionFactory) connectionFactory).setChannel("MCF.EDT.Q10I.01");
((MQQueueConnectionFactory) connectionFactory).setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
QueueConnection connection = connectionFactory.createQueueConnection();
connection.start();
QueueSession session = connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue("IMA.EDT.NL.007");
// Force MQSTR format
((MQQueue) queue).setTargetClient(JMSC.MQJMS_CLIENT_NONJMS_MQ);
QueueSender messageProducer = session.createSender(queue);
TextMessage textMessage = session.createTextMessage("Hello MQSTR world via MQQueue");
messageProducer.send(textMessage);
session.close();
connection.close();
}
... View more
12-21-2016
03:40 PM
1 Kudo
I’m seeing an issue where the
output sent to the IBM MQ (ESB) from our NiFi configuration sends a format
(MQHRF2) not recognized by the ESB consumer. This is very
similar to the forum issue posted here: http://forum.spring.io/forum/spring-projects/integration/jms/31307-invalid-message-received-in-remote-mq-what-will-be-the-possible-cause Unfortuntately I don't have the capability to get this correctly configured from the MQ Administration
side. There doesn't appear to be a property to control this from the connection factory settings for the "com.ibm.mq,jms.MQQueueConnectionFactory" that I can find. However, it appears the "com.ibm.mq.jms.MQQueue" counterpart will provide this capability via the "targetClient" with value "1". So I'm hoping to be able to instantiate the MQQueue with a property or similar w/in PublishJMS or via an context parameter similar to the behavior of "SS Context Service" I believe. Any light you can shed on this would be greatly appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
12-19-2016
03:17 PM
1 Kudo
Thank you for your reply. The problem was related to the "Support Fragmented Transactions" setting in the PutSQL. This needed to be changed to 'false' in my setting due to filtering logic upstream (i.e. ${fragment.index:plus(1):equals(${executesql.row.count})} which only allowed the last transaction to pass a the routeattribute.
... View more
12-14-2016
10:18 PM
1 Kudo
A message will sit indefinetly on the inbound queue prior to a PutSQL processor. I can see a single thread processor start and the bytes indicator, however, nothing hits Read/Write or Out counter in processor panel nor does the successful, failure, or retry queues ever get populated. I see scheduler agent related messages that seem to indicate a thread stopping or related. I have other flows using this same combination of processors working successfully reading from the same MySQL datasource. Is there another way to determine what might be the problem? 2016-12-14 16:01:49,915 INFO [StandardProcessScheduler Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PutSQL[id=dd773e31-6c2d-1e1c-31b6-4e5051563488] to run
2016-12-14 16:01:51,169 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@375b74aa // Another save pending = false
2016-12-14 16:02:01,114 INFO [StandardProcessScheduler Thread-5] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutSQL[id=dd773e31-6c2d-1e1c-31b6-4e5051563488] to run with 1 threads
2016-12-14 16:02:02,101 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@375b74aa // Another save pending = false
... View more
Labels:
- Labels:
-
Apache NiFi
12-09-2016
02:25 PM
For those who may be at a back level HDF version as we are a good workaround is to use the SplitContent instead as it utilizes many of the attributes Matt has documented above for the SplitJson processor.
... View more
12-06-2016
10:13 PM
@Matt Burgess You are correct about my being back level for this support. Can anyone suggest a workaround, such as another processor attribute that could do something similar?
... View more
12-06-2016
09:55 PM
Ah, I see we're at NiFi version 0.5.1.1.1.2.1-34, so that explains why I am not seeing these attributes.
... View more
12-06-2016
09:52 PM
Thanks for the prompt reply. I've also tried to address this question in item: https://community.hortonworks.com/questions/68745/nifi-iteration-of-queue-entries-between-processors.html I'm trying to get similar behavior as can be seen below for 'executesql.row.count' by using a user-defined property (property - iteration : value - ${'fragment.index'} <with and without single quote, set prior to 'SplitJson' or set post 'SplitJson') I'm only able to get 'No value set' or 'Empty value set' no matter what I try. The splitjson is very straight forward and successfully builds many json array objects using the expression below. All I'm attempting to due is keep track of the queue position, so I can act on the last row to post the last transaction date. The attribute 'queue position' would also be of interest, but it also contains no data. Thanks, ~Sean
... View more
- « Previous
-
- 1
- 2
- Next »