Member since
03-10-2017
155
Posts
79
Kudos Received
32
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
312 | 08-12-2024 08:42 AM | |
810 | 05-30-2024 04:11 AM | |
1255 | 05-29-2024 06:58 AM | |
747 | 05-16-2024 05:05 AM | |
606 | 04-23-2024 01:46 AM |
01-19-2022
08:10 AM
Although 2 nodes showing connected , we should also see the time stamps under column started At and LastHearbeat . also I am expecting to see coordinator and primary under status column. it is not clear from attached snap, as in from which node you are trying to access the UI ? if you open UI from each node 1,2, and 3 what you see? Is 3rd is showing as disconnected ? have you checked the process id's of nifi on 3rd node? How zookeeper is used ? External user or embedded zookeeper ?
... View more
01-19-2022
04:58 AM
1 Kudo
Can you please provide more details on this "delay" and why delay is needed ? FYI: Processors can be configured to all the time 0 sec or after a certain interval example after every 60sec or at fix time with cron driven. So if there is chance when files may not be present on ftp to be able to Fetch by NiFI : List/Fetch/GetSFTP processors , you can schedule them specific time. I can help you more once you provide more details on your use case. Thank You.
... View more
01-14-2022
05:45 AM
1 Kudo
Thank You, If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank You.
... View more
01-14-2022
05:26 AM
2 Kudos
Thanks for additional details , You can use QueryDatabaseTable or GenerateTableFetch with ExecuteSQL both will be fine, as said you need to limit number of records per flow file . When you say "When I try to move table with more then 1 GB data, my nifi GUI crashes"does that mean 1 flow file size is 1GB ? in NiFi generate by QueryDatabaseTable ? Is this happening when loading into NiFi by QueryDatabaseTable or after SplitAvro ? or At PutDabaseRecord ? Ideal way is to have small flow file size at the loading from DB by limiting setting in QueryDatabaseTable or GenerateTableFetch which ever you use so memory footprint will be less and records will be streamed into multiple flow files. Thank You.
... View more
01-14-2022
05:13 AM
In order to read parquet data from s3 bucket flow would look like : ListS3 -> FetchS3 -> ConvertRecord with parquet reader . So you are facing issue with ListS3 ? Can you please provide more details , I mean if you have configured with all the required details and upon start processor is throwing any error ? Processor show invalid and need some additional information? snapshot , error stack trace would help . You can also open Support case at support Thank You
... View more
01-14-2022
05:00 AM
1 Kudo
What I understand : As soon as you start the processor ExecuteSQL to load the data from oracle to NiFI , after few min NiFi UI became unviable but when you check on host , NiFi process id shows running. if this the case then This is a behaviour of resource constraints in NiFi with respect to memory and not the right way to read/load data from database. Two things needs to be considered here . 1. Need to make sure NiFi os configured with right heap settings 2. Load the data in incremental manner. NiFi can run various Sql loads with 8-16 GB of heap, so this much heap is required Use GenerateTableFetch before ExecuteSQL and limit number of records in each flow file with settings "Max Rows Per Flow File" & "Output Batch Size" & Fetch Size. With some test run iterations you need to determine what would be right value for above setting which does not create high heap usage. Thank You.
... View more
01-13-2022
04:03 AM
1 Kudo
To answer your question "is there a way to read a parameter from environment inside a parameter context?" No. The way you want to manage password at runtime is not possible.
... View more
01-13-2022
03:05 AM
1 Kudo
It is not allowed to set processor password property configuration through environment variable or variable registry. Only through parameter context values for passwords can be defined at processor group level in nifi.
... View more
06-01-2021
08:42 AM
2 Kudos
I have used CSV file withCountText and it worked fine for me, Upon success it will writes an attribute name "text.line.count" with number of lines as value, attaching sample csv file which I used to test and it has 6 lines.
... View more
- « Previous
- Next »