how do we create table in particular database? its going to default db. if we give db.table name its giving error saying, dot wont be accepted in table name..  ... View more

I'm new to NiFi, and I'm not sure your data flow has same condition as mine, but I have a same issue of being occurred the same exception that you mentioned. I'm using Oracle 11g XE, there was no invalid query nor invalid data. In addition, I had another problem with the Oracle session of PutSQL been locked when I let a lot of flowfile flow to PutSQL processor, e.g., 5,000 flowfile in 0.5 sec. I have spent all day long to fix this problem today modifying almost every single properties of all processors connected to the flow, and even of DBCP controller service... and finally found the cause.   In processor PutSQL, there is a property named ' Support Fragmented Transactions'. I don't know pretty much about this and need to know how it works, but when I have set it false, the problem was solved. And it took some time more than before.   I'm not an expert of NiFi, but I hope this might be helpful for you.   ... View more

Hello @JoeWitt ,   Thanks for your feedback. Actually, my flowFile is created by a syslog processor. I see no error in the Nifi log file regarding processing, and by the way I think I collect all my data correctly.   Stéphane ... View more

I am working on NIFI-4456 which will allow the JSON reader/writer to support the "one JSON per line" format as well as the "JSON array" format for input and output, so you will be able to read in one JSON per line and output a JSON array, using ConvertRecord (or any other record-aware processor). In the meantime, you can use the following crude script in an ExecuteGroovyScript processor to process your entire file (avoiding the Split/Merge pattern), it should get you what you want: def flowFile = session.get() if(!flowFile) return flowFile = session.write(flowFile, {inStream, outStream -> outStream.write('['.bytes) inStream.eachLine { line, i -> if(i > 1) outStream.write(','.bytes) outStream.write(line.bytes) } outStream.write(']'.bytes) } as StreamCallback) session.transfer(flowFile, REL_SUCCESS) The script just adds array brackets around the whole doc, and separates the lines by a comma. I did the crude version because it doesn't need to load the entire input content into memory. If you need more control over the JSON objects, you could iterate over the lines (still with eachLine), use JsonSlurper to deserialize each string into a JSON object, then add each object to an array, then use JsonOutput to serialize the whole thing back to a string. However that involves having the entire content in memory and could get unwieldy for large input flow files. ... View more

I would greatly appreciate if I could get some pointers to my request (this thread Feb 28, comment to @jwitt) for how to document in architectural plans that we can skip manual file integration checks when moving files over sftp and NiFi into HDP. I'd love to avoid it these checks, but need to refer to solid docs/arguments showing file integrity it's already guaranteed by underlaying technology (e.g. SFTP vs FTP). Any hints on where to go with this? Thanks. ... View more

I just verified on a latest build that setting that property does not result in any validation errors. It is possible there was an issue that has been resolved since that release. You might want to try HDF 3.1. ... View more

How far is the work on surfacing data provenance data in error handling? I have just discussed this option internally, rolling our own error handling process group using the data provenance rest api for looking up relevant data to convey in error logs and messages, but if on the near horizon as a built-in option, that sounds great. ... View more

Hi @mayki wogno The first error message was also written by the same error with the second error message. The processor reported the error twice, because it logged an error message when the ListHDFS processor caught the exception, then re-throw it, and NiFi framework caught the exception and logged another error message. When NiFi framework catches an exception thrown by a processor, it yields the processor for the amount of time specified by 'Yield Duration'. Once the processor successfully accesses core-site.xml and hdfs-site.xml, both error messages will be cleared. ... View more

Thanks for sharing your knowledge, I will try your tips. This specific GC issue is happening only when I assign multiple threads to the processors and try to speed up the flow, that otherwise runs at roughly 10MB/s in single thread. I originally designed the flow to use flowfile attributes cause I was tempted to make the computation happen in memory. I thought that it would have been faster with respect to reading the flowfile content in each processor, and consequently parsing it to get specific fields. Do you suggest trying to implement a version that works, let's say, "on disk" on flowfile content instead of attributes? ... View more

@Riccardo Iacomini Thank you for the great post! This is very helpful. Here I am wondering how you batch things together like having many csv rows instead of one csv row. Because if we want to batch csv row into multiple rows, we use MergeContent processor, but you also mention that MergeContent is costly. So how batch processing will work on Nifi?? ... View more

Thanks you,I understand. It seems the document need to be updated. ... View more

Yep what you describe with UpdateAttribute/MergeContent sounds perfectly fine. What you'll want there precisely will depend on how many relationships you have out of RouteText. As for concurrent tasks I'd say it would be 1 for GetFile 1 for SplitFile 2...4 or 5 or so on RouteText. No need to go too high generally. 1 for MergeContent 1 to 2 for PutHDFS You don't have to stress too much on those numbers out of the gate. You can run it with minimal threads first, find any bottlenecks and increase if necessary. ... View more

Thanks for your reply. I get it. ... View more

yeah thanks.. append works.. \\n (double backslash) doesnt.. I was doing this while writing count to file.. it now works.. The count is: ${executesql.row.count:append('\n')} ... View more

@Raghu Gurrala Hi Raghu Can you please help me in understanding how to load the data into OpenTSDB using Nifi alone? ... View more

@ccasano I don't see any issues in having Isilon to store the workflow repositories. Isilon is scalable storage solution and based on my experience, Isilon can be a good solution based on ... View more

Thanks, Joe. I'm aware of Java 8 dropping those settings, wherever they decided to have permgen be now, good choice there with defaults in bootstrap. I now realized that I got thrown off by off-heap vs non-heap, was thinking of the off-heap memory for this reason. I realize there's no off-heap memory used in the product. ... View more

Yes, Joe, I had something like snappy in my mind as a good middle ground between size and performance. As a minimum, a compression level property should be exposed to the operator to balance an existing compression protocol between speeed/cpu load and network traffic volume. ... View more

I'm surprised it wasn't @bbende@hortonworks.com who wrote this article 😉 ... View more

It's actual title is "Hadoop and Kerberos: The Madness Beyond the Gate" —there's an HP Lovecraft theme of "forbidden knowledge which will drive you insane" which is less a joke and more commentary. it's actually rendered on gitbook If you are working with Kerberos, get a copy of the O'Reilly Hadoop Security book too. My little e-book was written to cover the bits that was left out: to extend rather than replace. Finally, being open source: contributions are welcome ... View more