Member since
11-16-2016
40
Posts
7
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1806 | 12-19-2016 03:17 PM |
08-18-2017
06:48 PM
I guess what I was hoping for from the community is a sense of what is acceptable or expected regarding bulletin messages and their relative handling. I suspect there is no global means to suppress these messages, in which case, I will feel justified in telling our administrators of the NiFi cluster another means of monitoring should be considered.
... View more
08-16-2017
08:12 PM
A processor which is handling a json flow file via EvaluateJsonPath is posting a lot of ERROR such as the following: StandardContentClaim [resourceClaim=StandardResourceClaim[id=1502912107775-8018, container=default, section=850], offset=587561, length=41229],offset=0,name=1314908846183834,size=41229] did not have valid JSON content.
We're being told that the amount of messages are undesirable since there will be a ticket generated as a result of this level of message. Since I'm handling the failure relationship via another flow I am hoping to prevent this error from bubbling up to other parent processors or similar. I suppose a better option would be to validate the JSON payload via ExecuteScript and introduce a try/exception block: var StreamCallback = Java.type("org.apache.nifi.processor.io.StreamCallback");
var IOUtils = Java.type("org.apache.commons.io.IOUtils");
var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
var flowFile = session.get();
var text;
if(flowFile != null) {
// Create a new StreamCallback, passing in a function to define the interface method
flowFile = session.write(flowFile,
new StreamCallback(function(inputStream, outputStream) {
text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
outputStream.write(JSON.stringify(text).getBytes(StandardCharsets.UTF_8))
}));
session.transfer(flowFile, REL_SUCCESS);
} Any ideas would be appreciated. Thanks, ~Sean
... View more
- Tags:
- error
- nifi-processor
Labels:
- Labels:
-
Apache NiFi
04-28-2017
03:21 PM
It occurred to me that it might be better to just convert the entire flowfile contents to the stringify version rather than isolating the problematic blob first. I attempted this by using the following in the ExecuteScript: var StreamCallback = Java.type("org.apache.nifi.processor.io.StreamCallback");
var IOUtils = Java.type("org.apache.commons.io.IOUtils");
var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
var flowFile = session.get();
if(flowFile != null) {
// Create a new StreamCallback, passing in a function to define the interface method
flowFile = session.write(flowFile,
new StreamCallback(function(inputStream, outputStream) {
var text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
outputStream.write(JSON.stringify(text).getBytes(StandardCharsets.UTF_8))
}));
session.transfer(flowFile, REL_SUCCESS);
}
However, the resulting content has been modified where the existing json structure is now masked by the use of escape characters before double quotes for instance. Is there an easy way to convert this object back into a json array? Thanks.
... View more
04-27-2017
09:29 PM
Thank you for the reply Bryan. I'm stripping the data not needing to be converted prior to the ExecuteScript via two SplitContext processors that utilize hardcoded hexadecimal strings to isolate the string (called attributed 'default_payload') needing to be converted via the ExecuteStript: var flowFile = session.get();
if (flowFile !== null) {
var StreamCallback = Java.type("org.apache.nifi.processor.io.StreamCallback");
var IOUtils = Java.type("org.apache.commons.io.IOUtils");
var StandardCharsets = Java.type("java.nio.charset.StandardCharsets");
flowFile = session.write(flowFile, new StreamCallback(function(inputStream, outputStream) {
var contentObj = flowFile.getAttribute("default_payload");
outputStream.write(JSON.stringify(contentObj ).getBytes(StandardCharsets.UTF_8));
}));
session.transfer(flowFile, REL_SUCCESS);
} If I understand what your saying then the ExecuteScript would need to handled both the splitting and conversion at the same processor as to not write over attributes which would keep it synchronized in the flow? I had tried to use a parent UpdateAttribute to introduce a unique value to the flowfile(s) prior to the split/convertion and other JSON processing, but I am thinking this is to loose of a linkage for this type of processing.
... View more
04-27-2017
07:09 PM
I'm utilizing "SplitAvro" in prior to "ConvertAvroToJSON" to allow blob objects too large to convert via varchar in an SQL stream to be able to be ingested into a flowfile. I use several "SplitContent" to isolate the object needed to be handled specially via a custom "ExecuteScript" which, in turn, utilizes a "JSON.stringify" to get a string which can be handled by any subsequent JSON related processors. I'm wondering if I can utilize a MergeContent or something similar to re-merge the customized string back to the sibling flowfile which contained the other JSON structure? Is the processor dynamic enough to pair back to a separate flowfile using uuid or something similar and the 'Defragment' Merge Strategy. Or could I create a dynamic variable in the JSON structure flowfile which could map back to the corresponding customized string via some type of iterative process? Thank you for any hints or suggestions you can provide. ~Sean
... View more
Labels:
- Labels:
-
Apache NiFi
04-20-2017
04:20 PM
I suppose I should look outside of NiFi for solutions as there doesn't seem to be similar experiences or use cases as I have attempted.
... View more
04-20-2017
02:27 PM
If loading the entire table it the only option then would not providing a downstream processor allow the table to drain without clogging memory or other resources? Which processor would give me the most control on the number of rows/records returned with each iteration? With GenerateTableFetch it appears to be handled with the Partion Size parameter. With the QueryDatabaseTable processor I've experimented with using the Columns to Return
and Max Rows per File parameters without much success. The result of executing the QueryDatabaseTable processor always results in bring the NiFi cluster to a halt requiring one or more restarts to recover, so I'm reluctant to just arbitrarily try things. Any suggestions would greatly be appreciated.
... View more
04-19-2017
04:23 PM
Due to a limitation to the size (and type of data?) that can be cast within the ExecuteSQL processor (when casting a blob to a varchar I have issues if the field size exceeds 32K) I'm looking to utilize the GenerateTableFetch or QueryDatabaseTable processors provided with NiFi to ingest entire tables. QueryDatabaseTable seems promising, but I have run into two issues that has hindered using this approach. First of all the existing table size is way to large to ingest (probably terabytes or more) and I'm only interested in getting the records that are most recent, anyhow. Using a Max Initial Value on the date field seems to be a suitable key to keep track of the state. Is there anyway to inject a starting value other than the implicit key state achieve by ingesting the entire table? I was hoping to possible prime the query by initially hard coding a value for max_value_column in the `initial.maxvalue.{max_value_column}` parameter, but to no avail. Secondly, from what I've read in the forum it seems GenerateTableFetch would be the best option of there are joins required from more than one table. If this is so, could you provide an example of how one might go about implementing this? Thanks, ~Sean
... View more
Labels:
- Labels:
-
Apache NiFi
01-30-2017
08:30 PM
The issue was with the customized bundle utilizing a NiFi 0.5 api where the NiFi running the processor was at 1.0.0.
... View more
01-26-2017
08:02 PM
I realize my question is too vague to assist based on what I've provided. I'm not sure the "thread-dump.txt" is even capturing anything related to the problem. I know the processor is entering the exception part of the custom code, but I cannot get the inherent getLogger() to produce the error logs. (I've tried a standard System.out.println as well, but I'm not sure where the console output would be written to) I've seen examples where the logger is instantiated via "final ComponentLog logger = getLogger()" and other examples that suggest the getLogger() is inherented from the "AbstractProcessor" thus no instantiation is required. What is further confusing that if I try the above I see an no such method errors on the 'org.apache.nifi.logger.ProcessorLog' which I thought was deprecated. Our NiFi is running on HDF Version 2.0.1. Thanks of any assistance you can provide.
... View more
01-24-2017
05:18 PM
Hello, My custom java processor becomes unresponsive after being idle for a while resulting in or causing subsequent flows to fail. I used the ./nifi.sh dump thread-dump.txt to capture the problem and I've attached the resulting file. thread-dump.txt It appears that the dump shows predominately TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject, but I'm not sure how to interpret this issue. This is a continuation of the question asked earlier (https://community.hortonworks.com/questions/79331/debugging-custom-nifi-unresponsive-flows.html#answer-79337), but since I answered as satisfied I'm not sure that it will be monitored. Thanks, ~Sean
... View more
Labels:
- Labels:
-
Apache NiFi
01-23-2017
09:31 PM
Thank you for the prompt reply. I'll your suggestions a try.
... View more
01-23-2017
09:07 PM
1 Kudo
Using HDF version 2.0.1 I’m seeing an issue where a custom java processor within the NiFi
flow becomes unresponsive or, rather, the messages being sent automatically
routed to the ‘failure’ without any logged reason for the failure.
This instability seems to occur randomly after an unspecified amount of idle
time. If I refresh the processor by toggling on and off then the “failed”
documents are successfully cycled through the custom processor.
One of the problems I’m having is that I cannot use the org.apache.nifi.logging.ComponentLog
or related ProcessorLog which provides logging with in the application.
Either logging within a custom application has been deprecated or I’m
just not able to find a supported jar for such a purpose.
Do either of you have any suggestions for debugging /tips or any
information you may have which would help discern what might be the cause of
these disruptions in the flowfiles?
... View more
Labels:
- Labels:
-
Apache NiFi
12-21-2016
07:45 PM
I see getting customized processors is a bit more involved as described here: https://community.hortonworks.com/articles/4318/build-custom-nifi-processor.html
... View more
12-21-2016
07:33 PM
Thank you for the quick response. I'm needing to go the roll my own route, since the I get an error that "targetClient" is not allowed for 'XMSC_DESTINATION_NAME'. When setting up a processor that uses custom java code what script engine to you specify assuming you are using on of the script processors? Also, to you know how to inject the flowfile stream contents into the java code, since I need that instead of the line ""Hello MQSTR world via MQQueue"? import javax.jms.Queue;
import javax.jms.QueueConnection;
import javax.jms.QueueConnectionFactory;
import javax.jms.QueueSender;
import javax.jms.QueueSession;
import javax.jms.Session;
import javax.jms.TextMessage;
import com.ibm.mq.jms.MQQueue;
import com.ibm.mq.jms.MQQueueConnectionFactory;
import com.ibm.msg.client.wmq.compat.jms.internal.JMSC;
public void testMqstrViaApiTextMessage() throws Exception {
QueueConnectionFactory connectionFactory = new MQQueueConnectionFactory();
((MQQueueConnectionFactory) connectionFactory).setHostName("mfdevlcics.mayo.edu");
((MQQueueConnectionFactory) connectionFactory).setPort(3667);
((MQQueueConnectionFactory) connectionFactory).setChannel("MCF.EDT.Q10I.01");
((MQQueueConnectionFactory) connectionFactory).setTransportType(JMSC.MQJMS_TP_CLIENT_MQ_TCPIP);
QueueConnection connection = connectionFactory.createQueueConnection();
connection.start();
QueueSession session = connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = session.createQueue("IMA.EDT.NL.007");
// Force MQSTR format
((MQQueue) queue).setTargetClient(JMSC.MQJMS_CLIENT_NONJMS_MQ);
QueueSender messageProducer = session.createSender(queue);
TextMessage textMessage = session.createTextMessage("Hello MQSTR world via MQQueue");
messageProducer.send(textMessage);
session.close();
connection.close();
}
... View more
12-21-2016
03:40 PM
1 Kudo
I’m seeing an issue where the
output sent to the IBM MQ (ESB) from our NiFi configuration sends a format
(MQHRF2) not recognized by the ESB consumer. This is very
similar to the forum issue posted here: http://forum.spring.io/forum/spring-projects/integration/jms/31307-invalid-message-received-in-remote-mq-what-will-be-the-possible-cause Unfortuntately I don't have the capability to get this correctly configured from the MQ Administration
side. There doesn't appear to be a property to control this from the connection factory settings for the "com.ibm.mq,jms.MQQueueConnectionFactory" that I can find. However, it appears the "com.ibm.mq.jms.MQQueue" counterpart will provide this capability via the "targetClient" with value "1". So I'm hoping to be able to instantiate the MQQueue with a property or similar w/in PublishJMS or via an context parameter similar to the behavior of "SS Context Service" I believe. Any light you can shed on this would be greatly appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
12-19-2016
03:17 PM
1 Kudo
Thank you for your reply. The problem was related to the "Support Fragmented Transactions" setting in the PutSQL. This needed to be changed to 'false' in my setting due to filtering logic upstream (i.e. ${fragment.index:plus(1):equals(${executesql.row.count})} which only allowed the last transaction to pass a the routeattribute.
... View more
12-14-2016
10:18 PM
1 Kudo
A message will sit indefinetly on the inbound queue prior to a PutSQL processor. I can see a single thread processor start and the bytes indicator, however, nothing hits Read/Write or Out counter in processor panel nor does the successful, failure, or retry queues ever get populated. I see scheduler agent related messages that seem to indicate a thread stopping or related. I have other flows using this same combination of processors working successfully reading from the same MySQL datasource. Is there another way to determine what might be the problem? 2016-12-14 16:01:49,915 INFO [StandardProcessScheduler Thread-8] o.a.n.c.s.TimerDrivenSchedulingAgent Stopped scheduling PutSQL[id=dd773e31-6c2d-1e1c-31b6-4e5051563488] to run
2016-12-14 16:01:51,169 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@375b74aa // Another save pending = false
2016-12-14 16:02:01,114 INFO [StandardProcessScheduler Thread-5] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled PutSQL[id=dd773e31-6c2d-1e1c-31b6-4e5051563488] to run with 1 threads
2016-12-14 16:02:02,101 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@375b74aa // Another save pending = false
... View more
Labels:
- Labels:
-
Apache NiFi
12-09-2016
02:25 PM
For those who may be at a back level HDF version as we are a good workaround is to use the SplitContent instead as it utilizes many of the attributes Matt has documented above for the SplitJson processor.
... View more
12-06-2016
10:13 PM
@Matt Burgess You are correct about my being back level for this support. Can anyone suggest a workaround, such as another processor attribute that could do something similar?
... View more
12-06-2016
09:55 PM
Ah, I see we're at NiFi version 0.5.1.1.1.2.1-34, so that explains why I am not seeing these attributes.
... View more
12-06-2016
09:52 PM
Thanks for the prompt reply. I've also tried to address this question in item: https://community.hortonworks.com/questions/68745/nifi-iteration-of-queue-entries-between-processors.html I'm trying to get similar behavior as can be seen below for 'executesql.row.count' by using a user-defined property (property - iteration : value - ${'fragment.index'} <with and without single quote, set prior to 'SplitJson' or set post 'SplitJson') I'm only able to get 'No value set' or 'Empty value set' no matter what I try. The splitjson is very straight forward and successfully builds many json array objects using the expression below. All I'm attempting to due is keep track of the queue position, so I can act on the last row to post the last transaction date. The attribute 'queue position' would also be of interest, but it also contains no data. Thanks, ~Sean
... View more
12-06-2016
09:10 PM
Some processor's written attributes are readily available within the FIowFile attributes downstream. For example 'executesql.row.count' is populated after ExecuteSQL. I'm not seeing the same behavior with many of the other attributes such as SplitJson. Are we expected to use Groovy script or some other customer process to extract these values? A simple example would be appreciated.
... View more
Labels:
- Labels:
-
Apache NiFi
11-30-2016
03:20 PM
I'm seeing behavior in my environment were the work between processors is being handled by more than one node/thread resulting in multiples of the payload being created. For instance, the ExecuteSQL processor uses three nodes (I'm not sure I'm using the correct terminology here, but the number shows in the upper right of the processor box when active) which each generates a separate copy of the results set. Another potential issue outside of being inefficient is that the results aren't always exactly the same, but I use xpath to evaluate the results downstream. I'm assuming there are configuration properties that will globally influence this behavior, but I'm also interested in more per processor tuning options. Can you offer any suggestion regarding how to control this behavior?
... View more
Labels:
- Labels:
-
Apache NiFi
11-16-2016
11:04 PM
Just a quick clarification for others following this post the "SELECT" statement should use 'cast' and not 'case' as I have it above.
... View more
11-16-2016
10:30 PM
Thank you very much. That worked great.
... View more
11-16-2016
10:16 PM
When using an SQL query with a "BLOB" content the following error is thrown: ExecuteSQL Unable to execute SQL select query SELECT BLOBTBL.BLOB_CONTENTS FROM BLOB_DECOMP BLOBTBL fetch first 10 rows only with UR due to org.apache.nifi.processor.exception.ProcessException: com.ibm.db2.jcc.am.jo: [jcc][t4][1092][11638][3.57.110] Invalid data conversion: Wrong result column type for requested conversion. ERRORCODE=-4461, SQLSTATE=42815; routing to failure: org.apache.nifi.processor.exception.ProcessException: com.ibm.db2.jcc.am.jo: [jcc][t4][1092][11638][3.57.110] Invalid data conversion: Wrong result column type for requested conversion. ERRORCODE=-4461, SQLSTATE=42815 Tried the work around of using a 'cast' such as SELECT case(BLOBTBL.BLOB_CONTENTS as varchar(2000)) FROM BLOB_DECOMP BLOBTBL fetch first 10 rows only with UR But then the following is thrown: "failed to process session due to org.apache.avro.SchemaParseException: Illegal initial character: 1: org.apache.avro.SchemaParseException: Illegal initial character: 1"
... View more
Labels:
- Labels:
-
Apache NiFi