Member since
02-01-2022
281
Posts
103
Kudos Received
60
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1120 | 05-15-2025 05:45 AM | |
| 4947 | 06-12-2024 06:43 AM | |
| 7917 | 04-12-2024 06:05 AM | |
| 5814 | 12-07-2023 04:50 AM | |
| 3202 | 12-05-2023 06:22 AM |
07-26-2022
08:18 AM
Thank you @steven-matison This is really helpful.. /\ And If you allow me, I have a different question regarding Nifi CaptureChangeMySQL.. the username in this processor, should be the one with highest privileges/ root user ? because it is not mentioned anywhere in the help section (https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-cdc-mysql-nar/1.5.0/org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL/ ) And when i provide a user credentials who has GRANT ALL PRIVILEGES ON *.* then it is able to access CDC bin logs But getting permission denied when trying to access with a user who has only read permissions on *.* Since the processor has nothing to do with write, i thought only read permissions were enough, but looks like its not the case.. Can you please throw some light in this regard..
... View more
07-26-2022
12:31 AM
My problem was solved after using a different email id. As far as gmail settings are concerned: access to less secure apps must be turned on. IMAP access must be turned on in settings of gmail.
... View more
07-20-2022
08:23 AM
I think you will also need to do the transfer/commit in the each list logic
... View more
07-20-2022
05:14 AM
@AbhishekSingh I believe you can accomplish this with some very complicated Expression Language chaining. It is possible to have many Expression Languages combined into one expression. Reference: https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html I do not have time to prove this out, but in theory, you would combine lesson learned in your other post (expression language to change the database.table with Replace, ifElse and maybe Equals to get your desired outcome.
... View more
07-20-2022
05:02 AM
@AbhishekSingh It looks like the issue with the Replace Expression Language was just the ` around the database and table name. That works for me as follows: ${query:replace('`nrpuserorgdb2306`.`status`', '`nrpreportdb`.`user_org_status`')} Here is my flow to test: Notice the use of a simple flow to test the concept. I also set query in GenerateFlowfile and define a separate query2 in UpdateAttribute. I always work in ways that prove functionality, then when the concept is working, take the lesson learned into real flows. Some additional Screen Shots: Template and Flow Definition File on my GitHub: https://github.com/cldr-steven-matison/NiFi-Templates
... View more
06-29-2022
06:20 AM
@Tryfan You mention this file comes in daily. You also mention that this file arrives through a load-balancer so you don't know which node will receive it. This means you can't configure your source processor for "primary node" only execution as you have done in your shared sample flow with the ListFile. As Primary Node only, the elected primary node will be the only node that executes that processor. So if the source file lands on any other node, it would not get listed. You could handle this flow in the following manor: GetFile ---> (7 success relationships) PostHTTP or InvokeHTTP (7 of these with one configured for each node in your cluster cluster) ListenHTTP --> UpdateAttribute --> PutFile So in this flow, no matter which node receives your source file, the GetFile will consume it. It will then get cloned 6 times (7 copies then exist) with one copy of the FlowFile getting routed to 7 unique PostHttp processors. Each of these targets the ListenHTTP processor listening on each node in your cluster. That ListenHTTP processor will receive all 7 copies (one copy per node) of the original source file. Then use the UpdateAttribute to set your username and location info before the putFile which place each copy in the desired location on the source node. If you add or remove nodes from your cluster, you would need to modify this flow accordingly which is a major downside to such a design. Thus the best solution is still one where the source file is placed somewhere all nodes can retrieve it from so it scales automatically. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
06-10-2022
12:17 AM
2 Kudos
Follow the steps to install, lack _psycopg psycopg2 package. Py files The solution 1:wget https://github.com/psycopg/psycopg2/archive/refs/tags/2_8_5.tar.gz 2:tar -xf 2_8_5.tar.gz 3:python setup.py build python setup.py install 4:cp psycopg2-2.8.5-py3.7-linux-x86_64.egg/psycopg2 /usr/share/python3
... View more
06-01-2022
10:59 AM
1 Kudo
@steven-matison thank you so much.
... View more
05-24-2022
06:02 AM
1 Kudo
@FediMannoubi Below is a basic approach to solve. Assuming both postgres tables are populated with rows per your example, your nifi flow would need to get the CSV (various ways to do that), once the contents of the csv are in a flowfile (i use GenerateFlowFile processor), you can use a RecordReader based processor to read the csv. This will allow you to write SQL against the flowfile with QueryRecord to get a single value. For example: SELECT city_name FROM FLOWFILE Next, in your flow you will need to get the city_name value into an attribute, i use EvaluateJsonPath. After that a ExecuteSQL processor and associated DBCP Connection pool to postgres. Then in ExecuteSQL your query is SELECT city_id FROM CITY WHERE city_name=${city_name} At the end of this flow you will have the city_name from csv, and city_id from postgres. You can now combine or use the further downstream to suit your needs. INSERT is done similarly, once you have the data in flowfiles, or attributes, using the same ExecuteSQL you write an insert instead. My test flow looks like this, but forgive the end, as I did not actually have a postgres database setup. You can find this sample flow [here]. I hope this gets you pointed in the right direction for reading csv and querying data from database.
... View more
- « Previous
- Next »