Member since
09-03-2020
6
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
5898 | 09-10-2020 12:54 PM |
09-10-2020
12:54 PM
Maybe I have found a solution.. I'm gonna use the ExecuteSQL to do a "select insert" query.. The query will perform the joins and load the data into a table. Then the QueryDatabaseTable will read from the new table.. That way I'll be able to use the "Max Rows Per Floe File" property.
... View more
09-08-2020
07:45 AM
The process is triggered by a GetSFTP that retrieves a date file.. Then the query uses the date for filtering.. Thanks for your contribution.. As soon as I make progress I'll update you guys.
... View more
09-08-2020
07:17 AM
My point is, the error occurs when the ExecuteSQL is running. The cause is: ExecuteSQL needs to create a huge flow file that the edge machine doest not have enough processing power to create.. Your solutions looks good, but it would split the flow file after it was produced by the ExecuteSQL, your solution takes action in a moment when the error has already occurred.
... View more
09-04-2020
06:56 AM
I don't know if these solutions works for me. What I really wanna do is make ExecuteSQL work as (for example) SelectHiveQL, which means, ExecuteSQL get only one incoming flow file and sends forward multiple flows file that can be merged. My real problem is that the ExecuteSQL query sometimes needs to produce a flowfile whose size is too large for the edge machine to process, what ends in error.. so, I need to split the flow file to decrease the pressure on the edge machine... I've been through the same situation in Hive queries, but I solved using the "Max Rows Per Flow File" property and then merging the flowfiles.
... View more
09-03-2020
01:21 PM
There is no such a option at ExecuteSQL.. Perhaps you're using a newer version of NiFi: Mine is 1.5.0.3.1.2.0-7
... View more
09-03-2020
07:31 AM
Hello. I'm using an ExecuteSQL Processor to extract data from Oracle DB. The query has multiple joins and returns a large number of fields. The problem is that ExecuteSQL Processor returns a single flow file (avro format) that is huge. I want to split the flow file (based in a number of rows, for example) and then merge them at the proper moment. I have read about QueryDatabaseTable and GenerateFetchTable which are processors that can split the output flow files but looks like these processors aren't able to perform joins.. Does anyone knows a workaround?
... View more
Labels:
- Labels:
-
Apache NiFi